Synchronization in Internet Video Streaming

Synchronization in Internet Video Streaming

• Synchronization is crucial in media streaming to avoid a bad quality of experience, which can occur when audio and video are not in sync.
• Humans typically perceive audio-video synchronization errors if the difference in time ("skew") is higher than about 80 ms.

Audio and Video Synchronization

• To achieve synchronization, different media streams (audio and video) need to be synced using time references for specific points in media, such as video frames and audio frames.
• All content and time references need to be embedded into a single data flow, which requires data multiplexing.

Audio and Video Multiplexing

• MPEG standards provide a mechanism to achieve multiplexing with proper synchronization.
• Multiplexing is typically achieved through Time Division Multiplexing (TDM), where audio, video, and data are inserted as chunks in a single stream.
• These chunks are named "packets" in the MPEG standard.

Stream Types

• There are two types of streams: Elementary Stream (ES) and Program Stream (PS) / Transport Stream (TS).
• Elementary Stream (ES) is a sequence of encoded bits (video) or NALUs (since AVC) and sequence of encoded frames (audio).
• Program Stream (PS) is aimed at error-free applications, such as files on reliable storage, while Transport Stream (TS) is adapted from PS and aimed at error-prone applications, such as digital TV.

Packetized Elementary Stream

• Packetized Elementary Stream (PES) is a packetized version of ES, where each packet contains a header, timestamp, and data bits.
• PES packets can be further divided into groups called "packs".

PES Stream IDs

• PES streams can support up to 32 audio streams, 16 video streams, and other streams for information and control, such as private, proprietary, or program guide information.

Transport Streams

• Transport Streams (TS) have fixed-length (188 bytes) packets with headers derived from PES streams.
• These packets contain a transport header and stuffing bits that will be discarded.
• TS headers contain a master clock, typically 27 MHz, which is used for synchronization.

Synchronization

• TS headers contain a master clock (PCR) and two clock values: PTS (Presentation Time Stamp) and DTS (Decoding Time Stamp).
• PTS is the time at which decompressed data should be played back, while DTS is the time at which compressed data should be decoded.

Container Formats

• Container formats specify a binary syntax suitable to contain media data encoded with various codecs.
• Examples of container formats include MP4, AVI, MKV, MOV, VOB, 3GP, and ASF.
• These formats typically include additional metadata, such as seek support and indexing.

Mixed-Media Formats

• Mixed-media formats can contain audio, video, and metadata, such as MP4, AVI, and MKV.
• MP4 is a popular container format derived from MPEG-4 Part 12 (ISO Base Media File Format) and is widely used for online video streaming.

Matroska Multimedia Container

• Matroska is an open standard, open format, royalty-free container format (.mkv).
• It is based on Extensible Binary Meta Language (EBML) and is suitable for streaming, with support for out-of-the-box playback on Windows 10 and WebM.

Learn about the importance of synchronization in media streaming to avoid poor user experience, and how audio-video synchronization errors occur when timing is off.