Introduction To Multimedia Pt1.pdf

Introduction to Multimedia Diganta Baruah, Associate Professor, Dept. of IT and PhD Scholar, IIT Guwahati Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 2...

Introduction to Multimedia Diganta Baruah, Associate Professor, Dept. of IT and PhD Scholar, IIT Guwahati Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 2 Course Outcome COURSE OUTCOME (CO) OF IT1704 [Multimedia Computing & Communications] Course Description of Course Outcome Outcome (CO) CO1 To understand multimedia, basics of multimedia networks, multiplexing technologies in multimedia networks, wired and wireless networks, multipath fading with special emphasis on various fading models, radio propagation models, and to become familiar with multimedia network communications and applications with special emphasis on multimedia communication standards and protocols. CO2 To understand digitization principles of analog signals with encoder and decoder design, and to become familiar with multimedia information representation for text, images, graphics, audio, and video with special emphasis on PCM speech, basic concept of broadcast Television, and digitization formats of digital video. CO3 To apply various compression algorithms such as Static Huffman Coding, Arithmetic Coding, Lempel-Ziv-Welsh coding for text compression, and to understand various image compression techniques such as Joint Photographic Expert Group (JPEG) with special emphasis on Transform Coding along with principle of operation of Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT). CO4 To understand audio and video compression with special emphasis on audio compression techniques such as Pulse Code Modulation (PCM), Differential Pulse Code Modulation (DPCM), Predictive Differential Pulse Code Modulation, Adaptive Differential Pulse Code Modulation (ADPCM) with subband coding, Perceptual Coding with special emphasis on sensitivity of the human ear, frequency masking, temporal masking, and key concepts of Linear Predictive Coding (LPC) with special emphasis on its mathematical modeling.  Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 3 Syllabus of Multimedia Computing & Communications [IT1704] 1. Multimedia – Introduction – Components of multimedia – Multimedia data and multimedia systems–classification of multimedia systems–Characteristics of multimedia data– Uses of multimedia. 2. Multimedia Information Representation : Introduction – Digitization Principles : Analog signals, Encoder design, Decoder design– Text: Unformatted text, Formatted text, Hypertext –Images : Graphics, Digitized Documents, Digitized Pictures : color principles, raster-scan principles, Pixel depth, Aspect ratio, Digital cameras and scanners –Audio : PCM speech, CD-quality audio, Synthesized audio–Video : Introduction – Types of video signals–Basic Concept of Broadcast Television : Scanning sequence, Color signals, Chrominance components, Signal bandwidth, Digital video : Digitization formats. 3. Text and Image Compression – The need for compression – lossless compression and lossy compression – text compression algorithms : Static Huffman Coding, Arithmetic Coding, Lempel- Ziv-Welsh coding– Image compression : redundancy and relevancy of image data, image compression techniques, lossless image coding, Transform Coding. 4. Audio and Video Compression – The need for audio compression – Audio compression : Differential Pulse Code Modulation, Adaptive Differential Pulse Code Modulation, Adaptive Predictive Coding, Linear Predictive Coding, Code Excited Linear Predictive Coding, Perceptual Coding, MPEG audio coders – Introduction to video compression – Video compression based on Motion Compensation. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 4 Syllabus of IT1704 5. Multimedia Networks – Basics of Multimedia Networks – Multiplexing Technologies – Local Area Networks and Wide Area Networks – Access Networks. 6. Multimedia Network Communications and Applications: Quality of Multimedia Transmission – Multimedia over IP – Multimedia over ATM Networks – Multimedia Communication Standards and Protocols 7. Wireless Networks – Wireless versus Wired Technology – Basics of Wireless Communication – Wireless Networks – Radio Propagation Models. 8. Multimedia Databases -- Multimedia Databases, Desirable features of Multimedia Databases, Standards for Metadata. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 5 Text book and Reference books TEXT BOOK: 1. Multimedia Communications : Applications, Networks, Protocols, and Standards, Fred Halsall, Pearson Education. REFERENCE BOOK(S): 1. Fundamentals of Multimedia, Ze-Nian Li and Mark S. Drew, Pearson Education. 2. Multimedia Systems, Parag Havaldar and Gerard Medioni, Cenage Learning. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 6 Introduction to Multimedia Multimedia comes from the latin word multus (meaning numerous) and media (meaning middle). In latin, multi means many, much, multiple. Medium means middle or intermediary. Multimedia, therefore means "multiple intermediaries" or "multiple means". In general, we mean medium as a means for distribution and presentation of information. Examples of medium are text, image, graphics, speech, music, video etc. Media can be classified with respect to different criteria. For example, according to * Perception * Representation * Storage * Transmission * Information Exchange Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 7 The perception medium The question is : How do humans perceive information in a computer environment? Perception of information occurs mostly through seeing or hearing. For perception of media through seeing, the visual media such as text, image , and video are used. For perception of information through hearing auditory media such as music, speech, noise are relevant. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 8 The Representation Medium Representation media are characterised by internal computer representations of information. The central question is: How is the computer information coded? Various formats are used to represent information in a computer. e.g., -> A text character can be coded in ASCII. ->An audio stream can be represented using a simple PCM (Pulse Code Modulation) with a linear quantization of 16 bits per sample. ->An image can be coded as a JPEG format. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 9 Presentation Medium Presentation media refers to the tools and devices for input and output of information. The key question is: Through which medium is information delivered by the computer or introduced into the computer ? Keyword, mouse, camera, microphone etc. are input media. Monitor, speaker etc. are used to deliver the information by the computer (output media). Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 10 Storage medium and transmission medium STORAGE MEDIUM Storage media refer to a data carrier which enables storage of information. The question is : Where will the information be stored? Hard disk, CD-Rom ,etc are examples of storage medium. THE TRANSMISSION MEDIUM The transmission medium characterizes different information carriers that enable continuous data transmission. The question is: Over what will the information be transmitted? Information is transmitted over networks which use wire and cable and fibre optics as well as free air space transmission which is used for wireless traffic. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 11 Information Exchange Medium The information exchange medium includes all information carriers for transmission, i.e., all storage and transmission media. The main question is: Which information carrier will be used for information exchange between different places? Information can flow through intermediate storage media to destination through direct transmission using computer networks or through combined usage of storage and transmission media. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 12 Test yourself Differentiate between perception medium and representation medium. Differentiate between presentation medium and storage medium. Distinguish between transmission medium and information exchange medium. “The real world is analog in nature”. Justify the statement. Give two examples of analog signal. What is a digital signal? Explain Give two examples of digital signals. Differentiate between analog signals and digital signals Distinguish between continuous time signals and discrete time signals. Explain the need for digital representation of signals. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 13 Digital Representation Before discussing digital representation, let us understand what are analog quantities and digital quantities. Analog quantity: An analog quantity is a physical phenomena that varies continuously over space and/or time. Physical phenomena that stimulate human senses. e.g., light and sound, can be thought of as continuous waves of energy in space. These phenomena can be measured by instruments called sensors which transform the captured physical variable into another time/space dependent quantity called signal. If the signal is also continuous, then it is analogous to the measured variable, and hence the term “analog signal”. Analog signals are usually electrical in nature. Examples : a microphone converting sound energy to electrical signals, a solar cell converting solar radiation into electrical signals. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 14 Digital Representation Digital quantity: Digital quantities are not continuous over space and time, instead they are discrete in nature. Digital quantities exist or have values only at certain instants in time or points in space, and not at other instants in time or points in space. Digital quantities consist of sets of discrete values corresponding to instants in time or points in space. Digital quantities are represented as binary numbers consisting of 0s and 1s, because the processor and memory of a computer are actually integrated circuits(Ics) which are made up of transistors, and each transistor can behave as a bipolar switch which can be either ON or OFF Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 15 Digital Representation Digital quantities are not continuous over space or time. They are discrete in nature. Q. What is need for digital representation? Digital computers can recognize digital values only. Furthermore, digital quantities can easily be manipulated in the computer. Q. How do we represent natural phenomena like light, image, sound etc. as the digital quantities? Light, image, sound, video etc. occur in the world as natural phenomena which can be captured in the form of analog signals. These are analog signals because they are analogous to the natural phenomena, i.e., these analog signals can depict the continous variation in light, image, sound, video etc. that occur in the real world. All these signals need to be converted into digital form in order to process them using a computer. While converting an analog signal into digital form sampling is the first step. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 16 Sampling rate and Nyquist Sampling Theorem SAMPLING RATE It is the rate of frequency at which sampling occurs. It denotes the number of samples taken per second. NYQUIST SAMPLING THEOREM When converting the analog signal into digital form, the sampling frequency must be greater than twice the bandwidth of the input signal in order to be able to reconstruct the original signal accurately from the sampled version. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 17 Quantization The validity of the sampling theorem makes it possible to transmit or to process an analog signal by digital means. We need not take note of the analog signals at all times, but only at sampling times, and hence in the intervals between samplings we shall have time to convert each sample to digital form. The process of digitizing samples involves making an approximation. This process of a approximation is called quantization. When a finite number of bits are used, each sample can only be represented by a corresponding number of the discrete levels. If +vmax and -vmax is the maximum positive and maximum negative signal amplitude and n is the number of bits used, then the magnitude of each quantization interval is given by q=2Vmax/2n Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 18 Audio There are two types of audio signals: 1. Speech Signals (as used in a variety of interpersonal applications including telephony and video telephony). 2. Music (as used in applications such as CD-on demand and broadcast television). Audio can be produced by: i. Naturally by using microphone. ii. Electronically by using some form of synthesizer. A microphone generates time varying analog signals. In order to store these signals in the memory of a computer, and to transmit them over a digital network, they must be converted into digital form using an audio signal encoder. Also, on receiving the digitized bit streams they must again be converted back to the analog form, because the loudspeaker operates using an analog signal. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 19 Audio The bandwidth of a typical speech signal is from 50 Hz to 10 KHz, and that of a music signal ranges from 15 Hz to 20 KHz. Hence sampling rate for speech signal should be greater than its Nyquist rate which is 2× 10 KHz (i.e should be greater than 20 KHz or 20 ksps). Similarly, sampling rate for music should be greater than 2×20 KHz (i.e 40 KHz o 40ksps). The number of bits per sample must be chosen so that the quantization noise (or quantization error) generated by the sampling process is at an acceptable level. Test on Speech and Music suggests minimum 12 bits/sample for speech, and 16 bits/sample for music. For stereophonic music, since two signals must be digitized, therefore the bit rate is double that of monoaural(mono) signal. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 20 Audio Problem: Suppose the bandwidth of a speech signal is from 50 Hz to 10 KHz, and that of a music signal is from 15 Hz to 20 KHz.. Derive the bit rate that is generated by the digitization procedure in each case. Assume that the Nyquist sampling rate is used with 12bits/sample for speech signal and 16bits/sample for music signal. Also, calculate memory required for storing 10 minutes of stereophonic music. For Speech signal: Sampling rate=20ksps With 12 bits/sample, bit rate=12 bits/sample×20 kilo samples/sec=240 kbps For Music: Sampling rate=40ksps With 16 bits/sample, bit rate =16 (bits/sample)×40 (kilo samples/sec) =640kbps For stereophonic music, bit rate generated is 2×640 kbps (i.e 1280 kbps for stereo). Memory required= Bit rate in bps × time in seconds =(1280×103×600)bits =[(1280×103x600)/8]bytes =96X106 bytes =96 Mbytes Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 21 Sampling a signal at a rate which is lower than the Nyquist rate Sampling a signal at a rate which is lower than the Nyquist rate results in additional frequency components being generated that are not present in the original signal. Thus, the original signal becomes distorted. As an example, let us consider a sinusoidal signal of frequency 6KHz which is sampled at a rate of 8KHz or 8ksps. Obviously the sampling rate is lower than the Nyquist rate of 12ksps (i.e, 2×6KHz). Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 22 Effect of undersampling Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 23 Digitization The media elements, namely, audio, image, video etc. occur in natural world as analog quantities. Each of these analog quantities needs to be digitized before we can use them in multimedia applications. The conversion process is performed by an analog to digital converter (ADC) and involves three steps: Sampling, quantization, and codeword generation. The following figure shows these three digitization steps: Figure : Digitization steps Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 24 Digitization-Sampling (i) Sampling The first step in converting an analog quantity to a digital form is called sampling. Sampling involves breaking the continuous signal into discrete set of points. Sampling rate is determined based on Nyquist sampling theorem Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 25 Digitization-Sampling When converting the analog signal into digital form, the sampling frequency must be greater than twice the bandwidth of the input signal in order to be able to reconstruct the original signal accurately from the sampled version. This is also referred to as sampling rate. After sampling, the analog signal is represented as discrete amplitude values at sampling instants. For time varying signal such as audio signals, this is referred to as time discretization. In an electronic circuit, sampling clock pulses can be generated instantly, but recording the sample value needs some finite time. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 26 Digitization-Sampling Hence, the sample value must be held constant during the conversion process. This operation is called Sample-and-Hold. By using Sample- and-Hold circuitry, the current sample value is held constant until the next sample is obtained. Analog signal is sampled every TS secs. Ts is referred to as the sampling interval. fs = 1/Ts is called the sampling rate or sampling frequency. There are 3 sampling methods: Ideal - an impulse at each sampling instant Natural - a pulse of short width with varying amplitude Flattop - sample and hold, like natural but with single amplitude value Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 27 Digitization-Sampling Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 28 Digitization-Sampling The following figure shows sampling for various types of signals considering ideal sampling: Figure: Sampling for various signals (x axis-time, y axis-amplitude) Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 29 Digitization-Sampling For an intuitive example of the Nyquist theorem, let us sample a simple sine wave at three sampling rates: fs = 4f (2 times the Nyquist rate), fs = 2f (Nyquist rate), and fs = f (one-half the Nyquist rate). It can be seen that sampling at the Nyquist rate can create a good approximation of the original sine wave. Oversampling can also create the same approximation, but it is redundant and unnecessary. Sampling below the Nyquist rate does not produce a signal that looks like the original sine wave. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 30 Digitization-Sampling Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 31 Digitization-Sampling Question: A complex low-pass signal has a bandwidth of 200 kHz. What is the minimum sampling rate for this signal? Answer: The bandwidth of a low-pass signal is between 0 and f, where f is the maximum frequency in the signal. Therefore, we can sample this signal at 2 times the highest frequency (200 kHz). The sampling rate is therefore 400,000 samples per second. Question : A complex bandpass signal has a bandwidth of 200 kHz. What is the minimum sampling rate for this signal? Answer: We cannot find the minimum sampling rate in this case because we do not know where the bandwidth starts or ends. We do not know the maximum frequency in the signal. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 32 Digitization-Sampling [ A Mathematical Formulation] The media elements, namely, audio, image, video occur in natural world as analog quantities. Each of these analog quantities needs to be digitized before we can use them in multimedia applications. The conversion process involves three steps: Sampling, quantization, and codeword generation. These processes are discussed below. Sampling: The first step in converting an analog quantity to a digital form is called sampling. Sampling involves breaking the continuous signal into discrete set of points. Sampling rate is determined based on Nyquist sampling theorem When converting the analog signal into digital form, the sampling frequency must be greater than twice the bandwidth of the input signal in order to be able to reconstruct the original signal accurately from the sampled version. This is also referred to as sampling rate. After sampling, the analog signal is represented as discrete amplitude values at sampling instants. For time varying signal such as audio signals, this is referred to as time discretization. In an electronic circuit, sampling clock pulses can be generated instantly, but recording the sample value needs some finite time. Hence, the sample value must be held constant during the conversion process. This operation is called Sample-and-Hold. By using Sample-and- Hold circuitry, the current sample value is held constant until the next sample is obtained. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 33 Digitization-Sampling [ A Mathematical Formulation] Suppose, an analog signal is sampled every TS seconds. Ts is referred to as the sampling interval, and fs = 1/Ts is called the sampling rate or sampling frequency. Let us assume one dimensional signal in the time domain, with an amplitude given by. The sampled signal is given by s Here, Ts is the sampling period, and is the sampling frequency. We are using square brackets with index n to indicate discrete signals , and parentheses with time t to denote continuous signals. Here, s , s , s , and so on. If we reduce the sampling period (also known as the sampling interval) Ts (i.e., increase the sampling frequency fs), the number of samples increases, and correspondingly the storage requirement also increases. Similarly, if we increase the sampling period (also known as the sampling interval) s (i.e., decrease the sampling frequency fs), the number of samples collected for the signal decrease, and the storage requirement also decreases. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 34 Digitization-Sampling [ A Mathematical Formulation] If Ts is too large, the signal might be undersampled, thus leading to artifacts or aliasing effects. If Ts is too small, the signal will require large amount of storage, and may contain redundant data as well. For an intuitive example of the Nyquist theorem, let us sample a simple sine wave or a cosine wave at three sampling rates: fs = 4f (2 times the Nyquist rate), fs = 2f (Nyquist rate), and fs = f (one-half the Nyquist rate). It can be seen that sampling at the Nyquist rate can create a good approximation of the original sine wave or cosine wave. Oversampling can also create the same approximation, but it is redundant and unnecessary. Sampling below the Nyquist rate does not produce a signal that looks like the original sine wave. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 35 Digitization-Quantization [A Mathematical Formulation] (ii) Quantization: The second step is called quantization. The output of the sampling step is a set of sample values, which represents the changing analog signals. Quantization refers to the actual number of levels for representing the sampled values. Quantization level is usually indicated by a parameter called bit-depth which refers to the number of bits required to represent the requisite number of levels. For example, an 8 bit representation of audio signal implies that we can have a maximum of 256 different amplitude values to represent the time varying audio signal. Quantization deals with encoding the signal at every sampled location with a predefined precision, defined by the number of levels. In other words, the answer to the following question is given by the number of levels: “How many bits do we use to represent the value of a signal at each sampling instance?” Each sample can be represented by b bits. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 36 Digitization-Quantization [A Mathematical Formulation] The quantization operation can be represented as follows: Here, Q represents a rounding function that maps the samples to the nearest digital value using b bits. Using b bits corresponds to N= levels, thus producing a quantization step size of , Where, R is the entire range of the signal. Since each quantized sample is represented by a finite number of bits, the quantized value will differ from the actual signal value, thereby introducing an error known as the quantization error. The maximum quantization error is limited to half the quantization step. The quantization error decreases when we use more number of bits to represent the samples. The number of bits to be used depends on the type of signal and its intended use. For example, audio signals representing music must be quantized using 16 bits, whereas audio signals representing speech requires only 8 bits. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 37 Digitization- Codeword generation (iii) Codeword generation: After the quantization step, we have a set of sampled values quantized to a specific number of levels occurring at specific instants of time (or space). Codeword generation involves expressing these amplitude levels in terms of binary codes, because this is how the data would be represented within a computer. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 38 Digitization-Quantization Now, let us elaborate the quantization operation in detail: The output of the sampling step is a set of sample values, which represents the changing analog signals. Quantization refers to the actual number of levels for representing the sampled values. Quantization level is usually indicated by a parameter called bit-depth which refers to the number of bits required to represent the requisite number of levels. For example, an 8 bit representation of audio signal implies that we can have a maximum of 256 different amplitude values to represent the time varying audio signal. In many lossy compression applications, we are required to represent each source output using a small number of codewords. The number of possible distinct source output values is generally much larger than the number of codewords available to represent them. The process of representing a large ( or possibly infinite) set of values with a much smaller set is called quantization. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 39 Digitization-Quantization Let us consider a source that generates numbers between -10 and +10. A simple quantization scheme would be to represent each output of the source with the integer value closest to it. For example, if the source output is 2.47, we can represent it as 2 or 2.5, and if the source output is 3.14, we can represent it as 3 or 3.5. This approach reduces the size of the alphabet required to represent the source output. For example, the infinite set of values between -10 and +10 are represented with a set of 8 values. At the same time, we have also forever lost the original value of the source output. For example, if we know that the reconstructed value is 3, we can not tell whether the source output was 2.95, 3.14, 3.05, or any other value. The loss of information is the reason for the use of the word “lossy” in many lossy compression schemes. Although quantization is a simple process, the design of a quantizer has a significant impact on the amount of compression obtained and loss incurred in a lossy compression scheme. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 40 Digitization-Quantization The quantizer consists of two mappings: an encoder mapping and a decoder mapping. The encoder divides the range of values that the source generates into a number of intervals. Each interval is represented by a distinct codeword. The encoder represents all the source outputs that fall into a particular interval by the codeword representing that interval. As there can be many distinct sample values that can fall in any given interval, the encoder mapping is irreversible. Knowing the codeword only tells us the interval to which the sample value belongs. It does not tell us which of the many values in the interval is the actual sample value. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 41 Digitization-Quantization Quantization deals with encoding the signal at every sampled location with a predefined precision, defined by the number of levels. In other words, the answer to the following question is given by the number of levels: “How many bits do we use to represent the value of a signal at each sampling instance?” Each sample can be represented by b bits. The quantization operation can be represented as follows: Here, Q represents a rounding function that maps the samples to the nearest digital value using b bits. Using b bits corresponds to L= levels, thus producing a quantization step size of , Where, R is the entire range of the signal. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 42 Digitization-Quantization Sampling results in a series of pulses of varying amplitude values ranging between two limits: a min and a max. The amplitude values are infinite between the two limits. We need to map the infinite amplitude values onto a finite set of known values. This is achieved by dividing the distance between min and max into N steps, each of height  Step size,  = (max - min)/L Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 43 Digitization-Quantization The midpoint of each step (i.e., level) is assigned a value from 0 to L-1 (resulting in L values) Each sample falling in a level is then approximated to the value of the midpoint. Each level is then assigned a binary code. The number of bits required to encode the various levels, or the number of bits per sample as it is commonly referred to, is obtained as follows: b = log2 L Given our example, b = 3 Suppose, the codewords for the 8 levels are: 000, 001, 010, 011, 100, 101, 110, and 111 Assigning codes to levels: 000 will refer to level -20 to -15 001 to level -15 to -10, etc. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 44 Digitization-Quantization Assume we have a voltage signal with amplitutes Vmin=-20V and Vmax=+20V. We want to use L=8 quantization levels. Level width = (20 - -20)/8 = 5 The 8 levels are: -20 to -15, -15 to -10, -10 to -5, -5 to 0, 0 to +5, +5 to +10, +10 to +15, +15 to +20 The midpoints are: -17.5, -12.5, -7.5, -2.5, 2.5, 7.5, 12.5, 17.5 Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 45 Digitization-Quantization When a signal is quantized, we introduce an error, i.e., a quantized value is an approximation of the actual amplitude value. The difference between actual and quantized value (e.g., the midpoint of a level) is referred to as the quantization error. If there are more levels, then we have smaller step size, , which results in smaller errors. However, with more levels, we require more bits to encode the samples which results in higher bit rate Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 46 Digitization-Quantization Since each quantized sample is represented by a finite number of bits, the quantized value will differ from the actual signal value, thereby introducing an error known as the quantization error. The maximum quantization error is limited to half the quantization step. The quantization error decreases when we use more number of bits to represent the samples. The number of bits to be used depends on the type of signal and its intended use. For example, audio signals representing music must be quantized using 16 bits, whereas audio signals representing speech requires only 8 bits. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 47 Digitization-Quantization The following figure shows assignment of levels so as to have a maximum quantization error of s/2. Figure : Assignment of quantization levels to quantizer Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 48 Digitization-Quantization The following figure shows relationships among actual input signal, quantized signal, and quantization error: Figure: Depiction of actual signal, quantized signal, and quantization error Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 49 Digitization-Quantization Question: State the meaning of the term “dynamic range” as applied to an analog signal, and show how this is expressed in decibels. How does this influence the number of bits to be used in the quantizer of ADC (Analog to Digital Converter)? Answer: The ratio of peak amplitude of a signal to its minimum amplitude is known as the dynamic range of the signal, and is denoted as D. It is quantified using a logarithmic scale known as decibels (dB). When determining the quantization interval (thereby determining the number of bits to be used), it is necessary to ensure that the level of quantization noise relative to the smallest signal amplitude is acceptable. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 50 Digitization-Quantization Question: An analog signal has a dynamic range of 40 dB. Determine the magnitude of the quantization noise relative to the minimum signal amplitude if quantization uses (i) 6 bits and (ii) 10 bits. Here, and quantization noise= Hence, 40 dB= 2= Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 51 Digitization-Quantization With n=6 bits, With n=10 bits, As can be seen, with n=6 bits, the quantization noise is greater than and hence is unacceptable. With n=10 bits, the quantization noise is less than. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 52 Digitization- Quantizer Design and codeword generation After the quantization step, we have a set of sampled values quantized to a specific number of levels occurring at specific instants of time ( or space). Codeword generation involves expressing these amplitude levels in terms of binary codes, because this is how the data would be represented within a computer. The following figure shows a mechanism of quantizer design and codeword generation: Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 53 Digitization-Design of a quantizer As shown in the following figure, , , are three reference voltages connected to the inverting inputs of the comparators or operational amplifiers (op-amps). 𝑉 can be slowly time varying input voltage or sampled values of analog signals. When 𝑉 is greater than a reference voltage of a particular op-amp (comparator), the output of the op-amp is high, otherwise the output is low. Suppose, 𝑉 is greater than 2.5 V but less than 3.75 V, and as a result only the upper comparator output is LOW, and all the other comparator outputs are HIGH. Necessary logic circuitry is used to convert the outputs of comparators( op-amps) to the required number of bits. Figure: Design of a simple quantizer and encoding mechanism for codeword generation. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 54 Digitization A pictorial representation of the three processes of digitization (i.e. ADC) is depicted below: Figure: Depiction of sampling, quantization, and codeword assignment Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 55 Digitization Quality of digitization The sampling rate and bit depth together decide the quality of the output signal. As we increase the sampling rate, we get more information about the analog signal which correspondingly increases its quality. However, increasing the number of samples per second also means that more number of bits ( i,.e., more bit-depth) are required to represent these samples, thereby requiring more storage space to store them in computer, and possibly requiring powerful processors to process them, which will also increase the cost of system. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 56 Digitization If we use a high sampling rate and low bit-depth, we will not be able to represent all the sample values. On the other hand, if we use low sampling rate and higher bit depth, then we will have provision for representing large number of samples, but only a fraction of it will be utilized because we may not have enough samples (due to low sampling rates) to fill up all the slots provided by high bit-depth. In this situation, the advantage of higher bit-depth will be lost. Although sampling rate and bit-depth are independent of each other, they play a combined role in deciding the final quality of the digital output. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH Filtering As we know, digitization consists of three steps to digitize an analog signal: 1. Sampling 2. Quantization 3. Binary encoding (i.e., codeword generation)  Before we sample, we have to filter the signal to limit the maximum frequency of the signal as it affects the sampling rate.  Filtering should ensure that we do not distort the signal, i.e., remove high frequency components that affect the signal shape. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 58 Filtering In any signal processing application, the function of a filter is to remove unwanted parts of the signal, e.g., random noise and undesired frequencies, and to extract useful parts of the signal, e.g., the frequency components lying within a certain frequency range. The sampling requirement to convert an analog signal to digital signal is dictated by the sampling theorem. By knowing the frequency range of the input signal can enable this conversion. However, in practice, the range of frequencies needed to record an analog signal is not known beforehand. Therefore, a universal sampling rate cannot be derived. However, the frequency range of interest can always be determined. For example, human voice does not contain frequencies beyond 4KHz because of physical or structural limitation of the larynx. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 59 Filtering When voice is recorded, the signal coming to the microphone may also contain higher frequencies which may be caused by noise from the environment, noise from the microphone itself, or by resonance caused by sound waves reflected from objects in the environment. Thus, the maximum frequency content of the signal to be recorded may be considerably higher than 4KHz in this case. By adding a filter prior to sampling ensures that the frequencies above a certain cutoff limit are eliminated from the signal. Therefore, for recording human voice, an analog filter with 4KHz cutoff frequency is inserted between the microphone and the recording device. The resulting signal is then sampled at 8KHz. It should be noted that sampling is always accompanied by filtering. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 60 Filters Analog filtering techniques are commonly used to capture commonly used signals such as audio and images. In the digital world, digital data manipulation also requires filters known as digital filters. Filtering can be done prior to sampling or as a postprocessing operation after sampling. There are broadly two kinds of filters: analog filters and digital filters. Analog filters and digital filters are quite different in their physical structure and their way of operation. An analog filters uses analog electronic circuits made up of components like resistors, capacitors, and operational amplifiers to produce the required filter. Such filtering circuits are widely used in applications such as noise reduction, video signal enhancement, graphic equalizers in Hi-Fi systems etc. A digital filter, on the other hand, uses digital numerical computations on sampled, quantized values of the signal. The processing may be done on a general purpose computer or on a specialized Digital Signal Processor (DSP) chip. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 61 Advantages of digital filters A digital filter is programmable, i.e., a program stored in the processor’s memory determines its operation. Therefore, a digital filter can be changed easily without affecting the hardware circuitry. On the contrary, an analog filter can only be changed by redesigning the hardware circuitry. Digital filters can be easily designed, implemented, and tested on a general purpose computer. Digital filters can be cascaded in series or combined in parallel with minimal software requirements. This is not easily possible with analog circuitry without complete redesigning. The characteristics of analog filter circuits containing active components are dependent on temperature and may also drift. Digital filters do not have these issues, and therefore, are stable with respect to both time and temperature. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 62 Advantages of digital filters Digital filters can handle low frequency signals with more precision compared to analog filters. Although, analog filters were dominantly used in the past to handle high frequency signals, now-a-days digital filters are also increasing applied to high frequency signals in the radio frequency (RF) domain because of continuous improvement in DSP technology. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 63 Categories of analog and digital filters Both analog and digital filters are classified into three categories: low pass filters, high pass filters, and band pass filters. Low pass filters remove high frequency components from the input signal. Such filters are used to avoid aliasing signals while sampling. High pass filters are used to remove low frequency from a signal, and are used to enhance edges and sharpen an image. Band pass filters allow frequency components of a definite range to pass through it, and they are commonly used in subband filtering for audio signals. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 64 Digital filters In any signal processing application, the function of a filter is to remove unwanted parts of the signal, e.g., random noise and undesired frequencies, and to extract useful parts of the signal, e.g., the frequency components lying within a certain frequency range. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 65 Test Yourself Define the term “channel bandwidth” in relation to a transmission channel. Hence, explain the meaning of the term “ band limiting channel”. Explain the role of (i) band limiting filter, (ii) sample and hold circuit, and (iii) quantizer in a signal encoder. Determine the rate of the sampler and the bandwidth of the band limiting filter in an encoder which is to be used for the digitization of an analog signal which has a bandwidth from 15 Hz to 10 KHz assuming that the digital signal is to be stored within the memory of a computer. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 66 Test Yourself Determine the rate of the sampler and the bandwidth of the band limiting filter in an encoder which is to be used for the digitization of an analog signal which has a bandwidth from 15 Hz to 10 KHz assuming that the digital signal is to be transmitted over a channel which has a bandwidth from 200 Hz to 3.4 KHz. “The design of a quantizer has a significant impact on the amount of compression obtained and loss incurred in a lossy compression scheme”. Justify the statement. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 67 A note on quantization In uniform quantization, the output range of the signal is divided into fixed and uniformly separated intervals depending on the number of bits to be used for each sample. This is an effective arrangement when all the values in the range of the signal are equally likely, and thus the quantization error is equally distributed. However, for some signals where the distribution of all output values is nonuniform, it is more appropriate to make the quantization intervals nonuniform. For example, the output values of audio signals such as human voice are more likely to be concentrated on lower intensities rather than at higher intensities in the dynamic audio range. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 68 A note on quantization Since the distribution of output values in such signals is not uniform over the entire dynamic range, quantization errors should also be distributed nonuniformly. Nonuniform quantization can also be realized by using compressor before sampling, and then using a uniform quantization as part of analog to digital conversion (i.e. using linearADC). In this case, an expander circuit will have to be used at the destination after linear digital to analog (linear DAC) conversion. An example of this is PCM (Pulse Code Modulation) speech. In PCM speech, the speech input signal is initially passed through a bandlimiting filter, and then it is compressed, and then it is sent to linear ADC as part of the encoding process. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 69 A note on quantization Prior to the input signal being sampled and converted into digital form by the ADC, it is passed through the compressor which effectively compresses the amplitude of the input signal. The level of compression (and hence the quantization intervals) increases as the amplitude of the input signal increases. The resulting compressed signal is then passed to the ADC, which in turn performs a linear quantization on the compressed signal. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 70 A note on quantization As part of signal decoding at the receiver, the received bits are converted to analog signal by using a linear DAC, and then the signal is passed through expander, and finally it is passed through low pass filter to obtain the speech output signal. To be precise, at the destination, the received codewords are fed to a linear DAC. The analog output from the DAC is then passed through the expander which performs the reverse operation of the compressor circuit. Multimedia Computing and Communications (IT1704) /Multimedia Fundamentals [IT219A2/IT219A8] by DIGANTA BARUAH 71 Textbook and Reference books Text Book: 1. Ze-Nian Li, Mark S. Drew, Jiangchuan Liu “Fundamentals of Multimedia”, Springer, Third Edition, 2021 Reference Books: 1. P. Havaldar and G. Medioni “Multimedia Systems – Algorithms, Standards and Industry Practices”, Cengage Learning – First Edition, 2009. 2. W. Burger & M. Burge “Digital Image Processing: An algorithmic introduction using Java”, Springer - Second Edition, 2016 3. F. Halsall, “ Multimedia Communications: Applications, Networks, Protocols, and Standards”, Pearson.