IVS-C02-synchronization_and_multiplexing_v2.0.pdf

Internet Video Streaming Synchronization and Multiplexing Enrico Masala Version 2.0  Media Types When more media are present (audio, video, subtitles etc.) it is necessary to synchronize them to avoid bad quality of experience...

Internet Video Streaming Synchronization and Multiplexing Enrico Masala Version 2.0  Media Types When more media are present (audio, video, subtitles etc.) it is necessary to synchronize them to avoid bad quality of experience Voice arriving after moving lips Unsynchronized sound/actions Delayed / anticipated subtitles … IVS – © 2022 Enrico Masala 2 Humans and Synchronization Audio/video synchronization Humans typically perceive A/V synchronization errors if the difference in time (“skew”) is higher than about 80 ms "Lip-sync" Moreover, auditory process is not independent from video perception for understanding purposes McGurk effect: “Hearing Lips and Seeing Voices”, Nature, 1976 IVS – © 2022 Enrico Masala 3  Audio and Video Sync Need to: Sync different media streams (audio and video) using time references for specific points in media E.g., video frames (pictures) and audio frames Embed everything (content and time references) into a single data flow Need for data multiplexing Examples: single media files (“.mp4”), DVB (digital TV) Not always necessary (e.g., 2 RTP sessions, or separate HTTP flows) IVS – © 2022 Enrico Masala 4  Audio and Video Multiplexing MPEG standards provide a mechanism to achieve Gli standard MPEG forniscono un meccanismo multiplexing with proper synchronization per raggiungere multiplexing con la corretta sincronizzazione Typically, "Part 1 - Systems” realizza il multiplexing e la sincronizzazione TDM (Time Division Multiplexing) is used to insert audio, video, data, …, as chunks in a single stream Chunks are named "packets" in the standard Audio Video Audio Audio Video Audio Video Video packet packet packet packet packet packet packet … packet Data type, Data bits timestamp header IVS – © 2022 Enrico Masala 5  Stream Types Codec syntax without additions is named Elementary Stream (ES) sequenza di bits codificati oppure Nalus Sequence of encoded bits (video) or NALUs (since AVC) Sequence of encoded frames (audio) For multiplexing purposes, two syntaxes are provided in MPEG mirato ad applicazioni prive di errori Program stream: aimed at error-free applications, e.g., files on reliable storage - disks Transport stream: adapted from Program stream, aimed at error-prone applications, e.g., bit errors or packet losses (DVB – digital TV, with a single physical channel) IVS – © 2022 Enrico Masala 6 Program and Transport Stream Block diagram of MPEG "Systems" tutte le televisioni che girano oggigiorno sono transport stream ES ES Elementary Stream Packetized Elementary Stream IVS – © 2022 Enrico Masala 7 Packetized Elementary Stream PES Packets PES header Video PES packet packet packet … packet Audio PES packet packet packet … packet Stream ID, Data bits timestamp,... header In MPEG Systems, additional groups are formed, named "packs", not shown here IVS – © 2022 Enrico Masala 8  PES Stream IDs Support for 32 audio streams (e.g. different languages) 16 video streams (e.g. different "views" or "TV channels") other streams for information and control Private, proprietary, other info such as program guide (TV), etc. IVS – © 2022 Enrico Masala 9  Transport Streams Fixed length (188 bytes) packets with header, derived by PES stream Fixed size helps in transmission: error recovery codes “FEC”, etc. PES header PES Stream Split in parts to fit in 188-byte segments Transport Stream Transport header Stuffing Transport packet (188 bytes) dei bit che verranno scartati IVS – © 2022 Enrico Masala 10  Synchronization TS headers contain a master clock PCR (Program clock reference), typically 27 MHz PS headers contain the sync clock for data Typically 48 KHz (audio), 90 KHz (video), locked to PCR In the headers, two clock values are present PTS (Presentation Time Stamp) Time at which decompressed data should be played back DTS (Decoding Time Stamp) Time at which compressed data should be decoded In practice it sometimes differs from PTS only for video with B frames IVS – © 2022 Enrico Masala 11  MPEG Systems Limitations MPEG Systems was mostly designed for transmission (TS - Transport Stream) Although Program Stream (PS) can be used for storing media files, additional features would be desirable For instance: precise random access (seek), indexing, support for non-MPEG codecs, etc. IVS – © 2022 Enrico Masala 12  Container Formats Specifications of a binary syntax suitable to contain media data encoded with various codecs Specifiche di una sintassi binaria Typically include additional metadata adatta a contenere dati multimediali codificati con vari codec Seek support, etc. Examples Audio: WAV, AIFF Images: TIFF Audio and video: many (next slide) IVS – © 2022 Enrico Masala 13  Mixed-Media Formats MP4 (a.k.a. MPEG-4 Part 14) AVI (Audio/video interleave) Many use "FourCC" standard to identify codecs MKV (Matroska) MOV (QuickTime container, similar to MP4) VOB (for DVD Video) 3GP (for mobile, based on ISO Base Media File Format) ASF (for Microsoft WMA, WMV) MXF (Material Exchange Format, professional usage) MPEG Program Stream (PS) and Transport Stream (TS) IVS – © 2022 Enrico Masala 14  MPEG-4 File Format (MP4) [ https://mpeg.chiariglione.org/standards/mpeg-4/iso-base-media-file-format ] MPEG-4 Part 14 Derived from the more generic MPEG-4 Part 12 (ISO Base Media File Format), in turn derived from Apple Quick Time Format Flexible and extensible Designed to facilitate managing, editing, exchange and presentation of media content Independent of the transmission protocol (often used as opaque binary download over HTTP) IVS – © 2022 Enrico Masala 15  Example of MP4 Structure Initial Object Binary format for scene Atom tag Descriptor description Pointers to access units Flexible but rather complex header, difficult to reconstruct if lost IVS – © 2022 Enrico Masala 16 Matroska Multimedia Container Open standard, open format, royalty-free (.mkv) Specs at https://www.matroska.org Supported by a non-profit French organization Based on Extensible Binary Meta Language (EBML) Generalized file format for any kind of data, first developed for Matroska Sort of binary equivalent to XML Suitable for streaming: index not required at start More and more support (out-of-the-box) Windows10; Google WebM based on Matroska IVS – © 2022 Enrico Masala 17