Internet Video Streaming Protocols PDF

Internet Video Streaming Protocols for Multimedia Communications Enrico Masala Version 1.2  How to Improve (w.r.t. IP)? Build on the existing platform Design new protocols with features useful for time- sensitive communications (e.g., RTP) Desi...

Internet Video Streaming Protocols for Multimedia Communications Enrico Masala Version 1.2  How to Improve (w.r.t. IP)? Build on the existing platform Design new protocols with features useful for time- sensitive communications (e.g., RTP) Design new schemes aimed at improving performance (e.g., bitrate adaptation, multiple HTTP requests, …) Change the paradigm Provide some form of quality of service (controlled packet loss ratio and delay) by changing the network architecture, i.e., routers (Diffserv (RFC 2475) and similar approaches) Can be applied only in fully controlled domains (ISP, large organizations) IVS – © 2023 Enrico Masala 2 Protocols for Multimedia over IP Depends on the aim Transport of actual multimedia data (RTP) Monitoring, description, control, and signaling of the communication (RTCP, SDP, RTSP, H.323 and other helper protocols) Depends on the application logic Push-based protocols (bandwidth control at server) Server manages the session (RTP, RTCP, SDP, RTSP, H.323 etc) Pull-based protocols (bandwidth control at client) Client decides how to pull content on the server (HTTP ! ) IVS – © 2023 Enrico Masala 3 Protocols for Multimedia over IP Depends on the application requirements Low-delay communications, interactive applications such as VoIP Typically UDP-based: RTP, RTCP, H.323 (which also relies on RTP) “Streaming”-like services, on-demand or live RTSP HTTP !, even for transport of multimedia data (some delay is tolerable) IVS – © 2023 Enrico Masala 4  Media transport protocols IVS – © 2023 Enrico Masala 5 Why a Media Transport Protocol? Many multimedia application have common requirements requisiti comuni time sensitive-> dipendenti dal ritardo “Real-time”: data is not a simple sequence of bits: every chunk has specific temporal references Detect problems in the communication: Missing packets, reordered packets Monitor the communication: PLR statistics Use of different multimedia formats Autonomously manage issues bandwidth adaptation, transmission rate, etc. IVS – © 2023 Enrico Masala 6  RTP Most of the previous requirements have been addressed by the Real-time Transport Protocol (RTP) Developed by the Audio-Video Transport Working Group of the IETF (Internet Engineering Task Force) RFC 1889 (year 1996), updated with RFC 3550 IVS – © 2023 Enrico Masala 7  RTP Goals Facilitate transport of “real-time” data, i.e., chunks that have temporal references Scalability support of unicast and multicast with large la reta si occupa di duplicare il pacchetto e di distribuirli a tutti i number of users destinatari Support multimedia applications without constraining their logic. Does not specify senza bloccare il funzionamento algorithms for adaptation support for QoS, resource reservation etc. IVS – © 2023 Enrico Masala 8  RTP Characteristics The application decides how to segment data into packets è l'unica incaricata di decidere come dividere i dati in pacchetti Because only the application knows how to best transport the data solo l'app sa com'è bene trasportare i dati Employs an end-to-end paradigm solo partecipanti alla comunicazione RTP In accordance with the Internet philosophy High flexibility, support sessions non è così scontato perché le sessioni costano in termini di risorse Lightweight sessions, some aspects only defined into “profiles”, i.e., separate documents for specific types of data IVS – © 2023 Enrico Masala codice scritto per gestire gli stadi della sessione 9  RTP Session A group of participants that communicate by means of RTP un gruppo di partecipanti che comunica tramite RTP ONE (single) media Session identification (for each participant): A network address A pair of ports (typically even and odd consecutive values) to transmit A pair of ports to receive (may be the same as before) NB: Every RTP session has an associate RTCP session for monitoring (2nd port of the pair) IVS – © 2023 Enrico Masala 10  RTP Session: Details No means to exchange information about network addresses and ports Must be known (acquired by external means) Multimedia sources are identified by means of an RTP mechanism In RTP ogni sorgente multimediale è identificata da un meccanismo RTP Synchronization source (SSRC) Avoids to use the network address because data can be sent by special sources such as mixers Evita di utilizzare l'indirizzo di rete perché i dati possono essere inviati da fonti speciali come miscelatori IVS – © 2023 Enrico Masala 11  Example Two sessions: audio and video End system A RTCP Video RTCP Audio RTP Video RTP Audio Network RTCP Video RTCP Audio RTP Video RTP Audio End system B IVS – © 2023 Enrico Masala 12  RTP Header In UDP dobbiamo specificare una porta sorgente ed una destinazione 0 15 31 UDP Source Port Destination Port header Length Checksum V X P CC M PT Sequence Number RTP Timestamp header Synchronization Source (SSRC) ID Contributing Source (CSRC) ID (optional) Payload.... prime tre righe: 4+4+4 = 12 BYTES IVS – © 2023 Enrico Masala 13  RTP Header Fields V (2bit) version: RTP version used (e.g. v.2) X (1bit) extension: if set to 1, header is followed by an extension P (1bit) padding: if set to 1, the packet contains one or more padding bytes at the end of the multimedia payload These bytes may be necessary to facilitate cryptographic operations that work with fixed length blocks IVS – © 2023 Enrico Masala 14  RTP Header Fields numero abbastanza limitato di bit PT (7bit) payload type: identifies the format of the data in the RTP payload Type of multimedia format included in the packet (e.g., MPEG, GSM-AMR, MP3, etc.) CC (4bit) CSRC count: number of contributing source (CSRC) identifiers le app audio segnalano che il frame è di tipo voice M (1bit) marker: meaning is defined by the application. If set to 1, signals particular events, such as voiced frame or last packet of a video frame IVS – © 2023 Enrico Masala 15  RTP Header Fields Timestamp (32bit) : sampling time of the first byte in the packet payload. It allows to estimate: Jitter at the receiver tempo di campionamento del primo byte nel payload del pacchetto RTP Ci permette di fare tutta una serie di calcoli che sono specificati nello standard del pacchetto RTP Delay Synchronize the media playback with others (e.g., audio and video in different sessions) Note: all the slices belonging to the same video frame have the same timestamp IVS – © 2023 Enrico Masala 16  RTP Header Fields Sequence number (16bit): monotonically increasing sequence number, increased at every packet transmission, used to identify: Missing packets Out-of-order packets Duplicate packets The initial value is randomly chosen to prevent plain- text attacks in encrypted communications IVS – © 2023 Enrico Masala 17  RTP Header Fields SSRC (32bit) synchronization source: uniquely identifies the media source (or better, the packet source) of an RTP stream All packets with the same SSRC use the same clock to create timestamps and sequence number generator, so that the receiver can group them correctly Different media sources (e.g., audio and video), which belong to different RTP sessions, have different SSRC The SSRC is randomly chosen to ensure uniqueness, collisions are handled by RTCP by leaving and re- entering the session IVS – © 2023 Enrico Masala 18  RTP Header Fields CSRC (32bit) contributing source: up to 15 fields, SSRC of the sources that contributed to create a stream produced by a mixer. The number of fields is given by CC Payload (variable): media content, dependent on the application Note: no packet length field is present The lower-layer protocols can be used to compute it Note 2: IP+UDP+RTP = 40 bytes header: big! Voice: 20ms @ 8KHz = 160 bytes (20% is header!) IVS – © 2023 Enrico Masala 19  RTP Timestamp Based on a clock Typically, 8 KHz for audio codecs and 90 KHz for video codecs (90 is for compatibility reasons with other legacy transport mechanisms) First value is randomly chosen (for cryptographic reasons) It is synchronized to the clock of the media source E.g., sampling frequency It increases continuously even if non- consecutive elements are joined together IVS – © 2023 Enrico Masala 20  RTP Timestamp After timestamp computation and insertion into packets, packets can be sent in any order Example with video P frames have higher timestamp value than B frames that precede them in display order, but P frames are sent first The clock frequency, to be useful, must be high enough to allow synchronization among different media and to measure delay variations in the network That is, KHz is a suitable order of magnitude IVS – © 2023 Enrico Masala 21  RTP Transmission Model Every sender sends the data to ALL the other participants Every RTP session has an associated RTCP session for monitoring purposes RTCP = RTP Control Protocol Monitoring, NOT control RTCP also provides information to synchronize playback across different RTP sessions Special entities: mixer and translators, to handle special topologies / communication requirements IVS – © 2023 Enrico Masala 23  RTP Payload IETF has produced a number of RFCs that specify the payload format for many multimedia data encoding (both audio and video) Example RFC # Title 2032 RTP Payload Format for H.261 Video Streams. 2190 RTP Payload Format for H.263 Video Streams. 2250 RTP Payload Format for MPEG1/MPEG2 Video. 3984 RTP Payload Format for H.264 Video. IVS – © 2023 Enrico Masala 24  RTP Payload for MPEG-2 ogni pacchetto deve contenere solo un numero intero di slice Every packet must contain only an INTEGER number of slices (RFC 2250) The payload must start with the first byte of the slice (or other headers higher than the slice: frame, GOP; sequence) Marker bit Set to 1 if data refer to the last packet of the frame MPEG Video Specific Header Repeats some information such as the number of frame in the GOP to increase resiliency in case of packet losses IVS – © 2023 Enrico Masala 25  RTCP Always used together with RTP Does not transport media data: only monitoring information and statistics (binary protocol as RTP) Requires that RTCP packets are periodically sent from each participant (not only the one that sends the media data) to ALL the other participants RFC 2250 recommends to send packets every about 5 seconds, and in general not to exceed 5% of the RTP traffic, since it is basically overhead IVS – © 2023 Enrico Masala 26  RTCP Allows to get feedback information about the RTP session ci serve per chi trasmettere PLR, jitter, round trip time Main packet types: Receiver report (RR), Sender Report (SR) Estimated PLR, Jitter, RTT Feedback is useful to the application to adapt itself to the current network conditions (e.g., by changing rate to help in reducing congestion, changing encoding format, etc.) Allows to synchronize media from different RTP sessions IVS – © 2023 Enrico Masala 27  RTCP SENDER RTCP RTP INTERNET RTCP RTCP RTP RTP RECEIVER RECEIVER IVS – © 2023 Enrico Masala 28 RTP Profiles & Payload Formats A complete specification to use RTP for a specific applications requires An RTP profile sono un modo di isolare e mettere insieme delle funzionalità che abbiano un senso per la nostra applicazione E.g. RFC 3551 “RTP Profile for Audio and Video Conferences with Minimal Control” or E.g. RFC 4585: “Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)” deve dire come usare RTP per le parti che non sono state standardizzate One or more documents that specify the payload format for the used codecs E.g., RFC 2250 for MPEG-2 video IVS – © 2023 Enrico Masala 29  Description Protocols IVS – © 2023 Enrico Masala 30 Distributing Session Information The SDP protocol (Session Description Protocol), RFC 2327, is useful to describe a multimedia session Information is encoded in plain text Usually inserted into an RTSP communication Type of information included Name and aim of the session, time availability, multimedia format, information to join the session, bandwidth used, responsible of the session Still used today mainly in WebRTC IVS – © 2023 Enrico Masala 31  SDP Example From RFC 2327 : 3 parts v=0 o=mhandley 2890844526 2890842807 IN IP4 126.16.64.4 s=SDP Seminar i=A Seminar on the session description protocol Description u=http://www.cs.ucl.ac.uk/staff/M.Handley/sdp.03.ps part [email protected] (Mark Handley) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 Temporal info a=recvonly part m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 31 Multimedia m=application 32416 udp wb resources a=orient:portrait part IVS – © 2023 Enrico Masala 32  SDP Example v=0 o=mhandley 2890844526 2890842807 IN IP4 126.16.64.4 s=SDP Seminar i=A Seminar on the session description protocol u=http://www.cs.ucl.ac.uk/staff/M.Handley/sdp.03.ps [email protected] (Mark Handley) c=IN IP4 224.2.17.12/127 TTL value for multicast t=2873397496 2873404696 Start in receive only mode a=recvonly Profile Payload Type m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 31 Port mimetype: application/wb (whiteboard) m=application 32416 udp wb a=orient:portrait Media type (other values: data or control) IVS – © 2023 Enrico Masala 33  SDP Example Use of dynamic payload type m=audio 49230 RTP/AVP 96 97 98 a=rtpmap:96 L8/8000 a=rtpmap:97 L16/8000 a=rtpmap:98 L16/11025/2 Payload Types Clock frequency IVS – © 2023 Enrico Masala 34  Control Protocols IVS – © 2023 Enrico Masala 35  equivalente di HTTP per controllare le comunicazioni server-client per contenuti multimediali RTSP The Real Time Streaming Protocol (RTSP), RFC 2326 Basato su un paradigma client-server dove i server trasmettono in streaming al client utilizzando RTP Based on a client-server paradigm where servers stream to client using RTP Application level protocol to control the fruition of multimedia contents in an on-demand fashion, with typical VCR functionalities (play, pause, stop, …) Originally developed by RealNetworks, Netscape Communications and Columbia University IVS – © 2023 Enrico Masala 36  RTSP, RTP/RTCP Commands and requests RTSP Status information Server Video Client RTP Audio RTP Monitoring RTCP RTP Monitoring IVS – © 2023 Enrico Masala 37  RTSP Text-based Similar to HTTP Easily human-readable Message structure similar to HTTP Header and body for both request and response Messages are numbered (CSeq) since RTSP can use also unreliable transports such as UDP Different from HTTP Keep status information on the server ogni richiesta è indipendente dall'altra in HTTP It is bidirectional IVS – © 2023 Enrico Masala 38  RTSP Request Example METHOD requestedURI RTSPversion General header Request header Body-related header CRLF Body //come la scriverebbe il protocollo DESCRIBE rtsp://foo/twister RTSP/1.0 CSeq: 1 IVS – © 2023 Enrico Masala 39  RTSP Response Example RTSPversion statusCode extendedDescription General header Response header Body-related header CRLF Body RTSP/1.0 200 OK CSeq: 1 Content-Type: application/sdp Content-Length: 164 … IVS – © 2023 Enrico Masala 40  RTSP Methods Mandatory (6): DESCRIBE, OPTIONS, PAUSE, PLAY, SETUP, TEARDOWN Optional (5): ANNOUNCE, RECORD, REDIRECT, SET_PARAMETER, GET_PARAMETER IVS – © 2023 Enrico Masala 41  RTSP Finite State Machine Describe Describe Teardown Init Ready Teardown Setup Setup Pause o Teardown Setup Play Playing sto spedendo i dati multimediali al client Describe Play IVS – © 2023 Enrico Masala 42  RTSP Nowadays streaming mostly happens over HTTP, using the HTTP protocol Where is RTSP still useful? When there is a (simple) stateful server that should send multimedia content on demand and with low latency (RTP). Example: IP Cameras, IoT Devices, Drones, etc. How does it work? fare uno streaming su UDP è molto più semplice Various RTP parameters are exchanged in the message headers (ports, initial RTP seq. num. and timestamp, …) RTSP can also be used from client to server as it can interleave the actual RTP flow (useful for content ingestion) IVS – © 2023 Enrico Masala 43  Summary Livello Applicativo RTSP (application layer) SDP Livello di Trasporto RTP / RTCP (transport layer) TCP UDP Livello di Rete (network layer) IP IVS – © 2023 Enrico Masala 44  ITU Multimedia Protocols International Telecommunication Union ragionando non come se fosse una rete a pacchetto ma come se si stesse simulando una chiamata telefonica ITU-T H.2xx series (https://www.itu.int/rec/T-REC-H/en) Defined protocols for signaling and control Connection opening and management Gateway management … Modeled on the typical architecture of a telephone network, but adapted to packet switched networks such as Internet IVS – © 2023 Enrico Masala 45  H.323 H.323: Packet-based multimedia communication systems standard che descrive come secondo la filosofia ITU bisognerebbe fare le chiamate su reti a pacchetto Describes the architecture of the system and of the terminals (formats and protocols to handle), also with support of other protocols Relies on RTP / RTCP to transport multimedia data Still used in very few context Corporate professional videoconferencing systems and telecoms Can easily interconnect with traditional public switched telephone network (PSTN) via gateways IVS – © 2023 Enrico Masala 46  IETF for audio/video calls SIP (Session Initiation Protocol) RFC 2543 Application layer protocol, can run on TCP, UDP or SCTP semi-affidabile (SCTP is “intermediate” between TCP and UDP) Does not try to mimic PSTN but draw ideas from other IETF protocols E.g., human readable, similar to HTTP posso fare girare anche protocolli diversi da UCP e UTP IVS – © 2023 Enrico Masala 47  SIP Used, for instance, by many VoIP providers instead of H.323 because it is simpler and easier to integrate in the IP infrastructure When voice service is carried directly over ADSL, VDSL, FTTC, FTTH etc. IVS – © 2023 Enrico Masala 48  Example: VoIP Services VoIP providers often use SIP protocol Codecs: G.711 A-law, G.729 Packetization time: 20 ms DiffServ, i.e., marking of packets with priority ID (DSCP) Valid within the ISP network References ("free modem"): https://www.vodafone.it/portal/Aziende/Partita- IVA/Supporto/ADSL-e-Rete-Fissa/Station-e-altri- dispositivi/Vodafone-Modem-Libero/Parametri-di- configurazione https://assistenzatecnica.tim.it/at/Internet_privati/mod em_generico#604002 https://www.fastweb.it/adsl-fibra-ottica/dettagli/altri- modem/ [ From: https://assistenzatecnica.tim.it/at/Internet_privati/modem_generico#604002 ] IVS – © 2023 Enrico Masala 49