Machine Learning: Token Positional Encoding

Positional Encoding (PE)

• PE is a technique used to preserve the order of sequence in a model
• The formula for PE is:
  • PE(pos, 2i) = sin(2i) / 10000^(d_model/2)
  • PE(pos, 2i + 1) = cos(2i) / 10000^(d_model/2)
• The sine function is applied to tokens at even indices, and the cosine function is applied to tokens at odd indices
• d_model represents the output dimension of the model

Variance Adaptor

• The Variance Adaptor is a component that adds variance information to the phoneme hidden sequence
• It consists of three predictors: Duration, Energy, and Pitch
• The Variance Adaptor is used to provide better and more accurate information for predicting variant speech in TTS
• The three predictors analyze phoneme duration, pitch, and energy, which are essential for conveying emotions and prosody

Duration Predictor

• The Duration Predictor is a component that analyzes phoneme duration
• It addresses the challenge of emotional voice conversion and aims to transfer emotional prosody while preserving linguistic content and speaker identity
• The Duration Predictor introduces the Emotional Speech Dataset (ESD) for multilingual and multi-speaker applications in speech synthesis and voice conversion
• The ESD consists of 350 parallel utterances, each with an average duration of 2.9 seconds, delivered by 10 native English and 10 native Mandarin speakers

Speech-to-Speech Translation

• The paper proposes a TTS model based on FastSpeech 2, which integrates source language information
• The model uses a corpus called LibriS2S, which consists of audio pairs of the same sentences in two languages
• The paper collects data from three sources: OpenSLR Dataset, Self-Recorded Data, and YouTube Audiobook Clippings
• The model uses Tacotron2 for Mel spectrogram generation, and HiFiGAN and WaveGlow to synthesize speech from the generated Mel spectrogram

PolyVoice

• PolyVoice is a language-based model for speech-to-speech translation using a decoder-only language modeling approach
• It consists of two main components: Speech-to-Unit Translation (S2UT) and Unit-to-Speech (U2S)
• The S2UT component converts source language speech into discrete units through self-supervised training
• The U2S component synthesizes the translated speech while preserving the original speaker's style by processing these semantic units and generating codec units that embed the source speaker's style

Hindi-English Speech Conversion

• The study uses advanced models such as Bark, mBART, and fine-tuned XLSR Wav2Vec2 for Hindi-English speech conversion
• The researchers utilized 19 hours of audio from the Common Voice 13 dataset to finetune the XLSR Wav2Vec2 for speech recognition
• The mBART model was employed for translation, and Bark, a transformer-based model, was used for text-to-audio synthesis