Deepfakes and Audio Processing

Audio Deepfake Detection: A Survey

• Audio deepfake detection is an emerging topic, and despite promising performance, it remains an open problem.
• The survey aims to provide a systematic overview of developments in audio deepfake detection, including competitions, datasets, features, classifiers, and evaluation.

Types of Deepfake Audio

• There are five kinds of deepfake audio:
  • Text-to-speech (TTS): aims to synthesize natural speech from text using machine learning models.
  • Voice conversion (VC): aims to change the timbre and prosody of a speaker's speech to another speaker.
  • Emotion fake: aims to change the emotion of the speech while keeping other information intact.
  • Scene fake: aims to change the acoustic scene of an original utterance with another scene.
  • Partially fake: aims to distinguish between partially fake and real audio.

Competitions and Datasets

• Competitions and datasets include:
  • ASVspoof 2015, 2017, 2019, and 2021: evaluation of audio deepfake detection systems.
  • ADD 2022: includes three tasks: low-quality fake audio detection, partially fake audio detection, and audio fake game.
  • ASVspoof challenges: LA (logical access) task, PA (physical access) task, and speech deepfake detection task.
  • ADD 2022 challenges: LF (low-quality fake) task, PF (partially fake) task, and FG (audio fake game) task.

Features and Classifiers

• Discriminative features for audio deepfake detection include:
  • CQCC (constant-Q cepstral coefficients)
  • LFCC (linear frequency cepstral coefficients)
  • Raw audio features
  • Wav2vec2.0
• Representative classifiers include:
  • GMM (Gaussian mixture model)
  • LCNN (light convolutional neural network)
  • RawNet2
  • ResNet + Openmax

Challenges and Future Directions

• Remaining challenges in audio deepfake detection include:
  • Lack of large-scale datasets in the wild
  • Poor generalization of existing detection methods to unknown fake attacks
  • Interpretability of detection results
• Future research should focus on addressing these challenges and developing more effective detection methods.### Partially Fake Utterances
• Partially fake utterances are generated by manipulating the original utterances with genuine or synthesized audio clips.
• The speaker of the original utterance and fake clips is the same person.
• The synthesized audio clips, while keeping the speaker identity unchanged, are used to generate partially fake utterances.

Competitions

• A series of competitions have played a key role in accelerating the development of audio deepfake detection.
• The ASVspoof and ADD challenges have been designed to protect ASV systems or human listeners from spoofing or deceiving.

Benchmark Datasets

• Many early studies designed spoofed datasets to develop spoofing countermeasures for ASV systems.
• The ASVspoof 2015 involves logical access to detect spoofed audio from the perspective of protecting ASV systems.
• The ADD 2022 challenge includes three subchallenges: audio fake game (FG), manipulation region location (RL), and deepfake algorithm recognition (AR).

Characteristics of Representative Datasets

• The characteristics of representative datasets on audio deepfake detection include:
  • Language (English, Chinese, etc.)
  • Goal (Detection, Game fake, Forensics, etc.)
  • Fake types (VC, TTS, partially fake, etc.)
  • Condition (Clean, Noisy, etc.)
  • Format (FLAC, WAV, etc.)
  • Sampling rate (SR, Hz)
  • Average length of utterances (SL, s)
  • Number of hours
  • Number of real/fake utterances
  • Number of real/fake speakers

Evaluation Metrics

• Equal Error Rate (EER) is used as the evaluation metric for audio deepfake detection tasks.
• EER is defined as the error rate at the threshold θEER, where the two detection error rates are equal.
• The final ranking is in terms of the weighted EER (WEER), which is defined as the weighted average of EER of the two rounds.

Discriminative Features

• The feature extraction module is a key component of the pipeline detector.
• The goal of feature extraction is to learn discriminative features that capture audio fake artifacts from speech signals.
• Features are divided into four categories: short-term spectral features, long-term spectral features, prosodic features, and deep features.
• Short-term spectral features are computed mainly by applying the short-time Fourier transform (STFT) on a speech signal.
• Magnitude-based features are directly derived from the magnitude spectrum, while phase-based features are derived from the phase spectrum.
• Long-term spectral features capture long-range information from speech signals.
• Prosodic features span over longer segments, such as phones, syllables, words, and utterances.
• Deep features are extracted via deep neural network-based models.

Short-term Spectral Features

• Short-term spectral features are mainly composed of short-term magnitude and phase-based features.
• Magnitude-based features include:
  • Magnitude spectrum
  • Log magnitude spectrum (LMS)
  • Power spectrum
  • Log power spectrum (LPS)
  • Cepstrum (Cep)
  • Filter bank-based cepstral coefficients (FBCC)
  • All-pole modeling-based cepstral coefficients (APCC)
  • Subband spectral (SS) features
• Phase-based features include:
  • Instantaneous frequency (IF) spectrum
  • Group delay (GD) spectrum### Phase Features
• Phase Spectrum does not have stable patterns for fake audio detection due to phase warping
• Post-processing methods are used to generate short-term phase-based features including:
• Group Delay (GD) based features: GD, Modified Group Delay (MGD), MGD cepstral coefficients (MGDCC), and All-Pole Group Delay (APGD)
• Cosine-Phase (CosPhase) features
• Instantaneous Frequency (IF)
• Baseband Phase Difference (BPD)
• Relative Phase Shift (RPS)
• Pitch Synchronous Phase (PSP)

Long-term Spectral Features

• Short-term spectral features are not good at capturing temporal characteristics of speech feature trajectories
• Long-term spectral features are used to capture long-range information from speech signals
• Four types of long-term spectral features:
• STFT based features: Modulation features, Shifted Delta Coefficients (SDC), Frequency Domain Linear Prediction (FDLP), and Local Binary Pattern (LBP) features
• CQT based features: CQT spectrum, CQT cepstral coefficients (CQCC), extended CQCC (eCQCC), and inverted CQCC (ICQCC)
• HT based features: Mean Hilbert Envelope Coefficients (MHEC)
• WT based features: Mel Wavelet Packet Coefficients (MWPC), Cochlear Filter Cepstral Coefficients (CFCC), and CFCC plus Instantaneous Frequency (CFCCIF)

Prosodic Features

• Prosody refers to non-segmental information of speech signals, including:
• Syllable stress
• Intonation patterns
• Speaking rate
• Rhythm
• Important prosodic parameters:
• Fundamental frequency (F0)
• Duration (e.g. phone duration, pause statistics)
• Energy distribution
• F0 is also known as pitch, and its pattern is different between synthetic speech and natural speech

Deep Features

• Learnable spectral features:
• Partially learnable spectral features: extracted by training a filterbank matrix with a spectrogram
• Fully learnable spectral features: learned directly from raw waveforms
• Supervised embedding features:
• Spoof embeddings
• Emotion embeddings
• Speaker embeddings
• Pronunciation embeddings
• Self-supervised embedding features: learned from self-supervised speech models using unannotated speech data

Classification Algorithms

• Traditional classification algorithms
• Deep learning classification algorithms
• The backend classifier is important for audio deepfake detection, aiming to learn high-level feature representation and model excellent discrimination capabilities

Test your knowledge on deepfakes, audio deepfake detection, and speech processing technologies like text-to-speech models and voice conversion. Learn about the main purposes and categories of features used for detecting fake attacks.