Our latest machine learning paper has been published. link to view
MODELING NON-LINGUISTIC CONTEXTUAL SIGNALS IN LSTM LANGUAGE MODELS VIA DOMAIN ADAPTATION
Authors:
MODELING NON-LINGUISTIC CONTEXTUAL SIGNALS IN LSTM LANGUAGE MODELS VIA DOMAIN ADAPTATION
Abstract:
When it comes to speech recognition for voice search, it would be
advantageous to take into account application information associated
with speech queries. However, in practice, the vast majority
of queries typically lack such annotations, posing a challenge to
train domain-specific language models (LMs). To obtain robust domain
LMs, typically a LM which has been pre-trained on general
data will be adapted to specific domains. We propose four adaptation
schemes to improve the domain performance of long shortterm
memory (LSTM) language models, by incorporating application
based contextual signals of voice search queries. Most adaptation
strategies are shown to be effective, giving up to 21% relative
reduction in perplexity relative to a fine-tuned baseline on a heldout
domain specific development set. Initial experiments using a
state-of-the-art Italian ASR system show a 3.1% relative reduction
in WER on top of an unadapted 5-gram LM. In addition, human
evaluations show significant improvements on sub-domains from using
application signals. Our first three schemes focus on improving
domain perplexity, while the fourth scheme provides a possible solution
to simultaneously reduce domain perplexity while attenuating
catastrophic forgetting, which is known to be a common problem
in the adaptation of neural networks. We present a thorough exploration
on incorporating application signals, which could be easily
generalized for generic contextual information.
advantageous to take into account application information associated
with speech queries. However, in practice, the vast majority
of queries typically lack such annotations, posing a challenge to
train domain-specific language models (LMs). To obtain robust domain
LMs, typically a LM which has been pre-trained on general
data will be adapted to specific domains. We propose four adaptation
schemes to improve the domain performance of long shortterm
memory (LSTM) language models, by incorporating application
based contextual signals of voice search queries. Most adaptation
strategies are shown to be effective, giving up to 21% relative
reduction in perplexity relative to a fine-tuned baseline on a heldout
domain specific development set. Initial experiments using a
state-of-the-art Italian ASR system show a 3.1% relative reduction
in WER on top of an unadapted 5-gram LM. In addition, human
evaluations show significant improvements on sub-domains from using
application signals. Our first three schemes focus on improving
domain perplexity, while the fourth scheme provides a possible solution
to simultaneously reduce domain perplexity while attenuating
catastrophic forgetting, which is known to be a common problem
in the adaptation of neural networks. We present a thorough exploration
on incorporating application signals, which could be easily
generalized for generic contextual information.
Authors:
Comments
Post a Comment