Lecture 3: Convolution and Its Properties - ELEC 421 DSP - 2024 PDF

Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing ELEC 421 Digital Signal and Image Processing Siamak Najarian, Ph.D., P.Eng....

Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing ELEC 421 Digital Signal and Image Processing Siamak Najarian, Ph.D., P.Eng., Professor of Biomedical Engineering (retired), Electrical and Computer Engineering Department, University of British Columbia Siamak Najarian 2024 1 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Course Roadmap for DSP Lecture Title Lecture 0 Introduction to DSP and DIP Lecture 1 Signals Lecture 2 Linear Time-Invariant System Lecture 3 Convolution and its Properties Lecture 4 The Fourier Series Lecture 5 The Fourier Transform Lecture 6 Frequency Response Lecture 7 Discrete-Time Fourier Transform Lecture 8 Introduction to the z-Transform Lecture 9 Inverse z-Transform; Poles and Zeros Lecture 10 The Discrete Fourier Transform Lecture 11 Radix-2 Fast Fourier Transforms Lecture 12 The Cooley-Tukey and Good-Thomas FFTs Lecture 13 The Sampling Theorem Lecture 14 Continuous-Time Filtering with Digital Systems; Upsampling and Downsampling Lecture 15 MATLAB Implementation of Filter Design Siamak Najarian 2024 2 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Lecture 3: Convolution and its Properties Siamak Najarian 2024 3 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Table of Contents Review of impulse response Running example: computing a system's response to a signal in different ways Direct computation Using the convolution sum: adding up shifted and scaled copies of the impulse response Flipping and sliding one signal against the other MATLAB example of flipping and sliding Understanding h[n-k] The convolution array (a fast method for convolving short signals) Convolving infinite-length signals The sum of a finite geometric series Symbolic sum of a series in MATLAB Properties of convolution/LTI systems Commutative property Distributive property Associative property Causality and the impulse response The step response and its relationship to the impulse response Differential and difference equations Siamak Najarian 2024 4 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Review of impulse response H Goal: We want to know what would happen if we put an arbitrary signal into the system. We showed that the output was called the convolution of the input with the impulse response. H For the above to be true, the system should be LTI. Siamak Najarian 2024 5 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Running example: computing a system's response to a signal in different ways; Direct computation Example: We would use double underlined to remind ourself where x is. We can also use a vertical arrow. Shorthand for input signal. Shorthand for output signal. Siamak Najarian 2024 6 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Using the convolution sum: adding up shifted and scaled copies of the impulse response impulse response We can just read off the impulse response, 1, -2, 3 in the shorthand, which is the same as 1, -2, 3 in equation (1). We are simply writing down the coefficients on each of the (1) delay versions of the signal. Siamak Najarian 2024 7  Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Using the convolution sum: adding up shifted and scaled copies of the impulse response (1) What we are doing here is weighting the delayed versions of the impulse response, (1). The weighting factors are x[k]’s. For example, for xh[n-1], the Original impulse Start here weighting is x (or the coefficient response Shift by 1 of the impulse signal) and the delayed version of the impulse Shift by 1 signal is h[n-1]. Zero element is right here (at y = 1). We are adding up the corresponding sticks. Siamak Najarian 2024 8 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Flipping and sliding one signal against the other Here, k is like our dependent variable and n is like a constant. h[-k], B, is the mirror image of h[k], A. A h[n-k], C, looks like h[-k] slid to the right by n units. In h[n-k] graph, what used to be 0 will begin at n. B The convolution sum tells us that first we need to find out x[k], then find out h[n-k] and finally multiply corresponding entries up and sum them C up together. That gives us one value of y. Siamak Najarian 2024 9 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing MATLAB example of flipping and sliding Siamak Najarian 2024 10 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing MATLAB example of flipping and sliding How does the convolution work? What is going to happen is we are going to see that h[k] is going to flip around. Then we are going to start click it across the x[k] (1) and at each point we multiply the corresponding entries at the bottom and add them up. Here, h[-k] is the filliped version, i.e., (2) graph (1). In (2), we get one entry (y). Then we click the signal h[-k] over by 1. Now, we get h[1-k] or graph (3). We add the corresponding entries up and we get another entry, y, and so on. (3) Siamak Najarian 2024 11 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing MATLAB example of flipping and sliding Siamak Najarian 2024 12 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing MATLAB example of flipping and sliding Siamak Najarian 2024 13  Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Understanding h[n-k] Example: Can we still use the previous process when there is an impulse response beyond zero on the left hand side? The answer is “yes”. So, if we have negative values like h[-1], the same idea applies. We do not usually like systems like that, but the same flipping, shifting computation method is applied. And that is where it is really important to keep track of the zero element (see the example). In this example, we also have to think about what happens when we move h[-k]. That means we may get some y-values that are non-zero for negative n’s. The final output signal is given in graph (1). (1) MATLAB Code for Convolution: In MATLAB, the disp(y) command is used to display the value of the variable y in the Command Window. This function shows the value without printing the variable name, making it useful for displaying results or messages in a clean format. Siamak Najarian 2024 14 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The convolution array (a fast method for convolving short signals) Convolution Array Method: We take the elements of x and the elements of h and we make a table, A. Then we are going to fill this table up with products of A values of x and h, as shown in B. For example, 1 (encircled green)×1 (encircled red) is 1 (encircled brown), and -2×1 is -2, etc. Then we look at the diagonals of this table and we add up corresponding values. For example, on the diagonal line of “(a)”, we have only one element: (1) = 1 and on the diagonal line of “(b)”, we have: (-2) + (1) = -1, etc. So, here we get [1, -1, 2, 1, 3]. How do we know what is the zero element of the result? First, we double-underline both the zero element of the input (x) and the zero element of the impulse response (h) and where those two things (a) cross is the zero element of the output. That is, B wherever that double-underline crossing lands is (b) where we n = 0 for y. In this case, they land on the 1 encircled brown. Because the brown element is on the first diagonal, so the result of our calculations for the first diagonal will give us the location of the zero- value element. In this example, the first element in the final output sequence is our zero-element. Siamak Najarian 2024 15 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The convolution array (a fast method for convolving short signals) Example: Let us use the Convolution Array Method to do this example. We see that this method is much easier than any of the previous methods we covered so far. The crossing of the double-underlined -1 and 1 gives us an element that is encircled by the brown circle, i.e., -1. This means that the zero element will be the outcome of our computation for this particular diagonal. This is the zero-value element. Siamak Najarian 2024 16 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The convolution array (a fast method for convolving short signals) Example: Another example that is solved by the Convolution Array Method. Siamak Najarian 2024 17 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Convolving infinite-length signals Example: Note: We cannot use the convolution array method for every input signal, specifically, the ones that are infinitely long. In this example, instead of having just solid numbers, we also have symbols, i.e., 𝜶. But this exponential decay goes on forever and for any value of n, no matter how large, we are still going to get a little non-zero value of x. So, let us investigate what is going to be the convolution between these two things. One nice thing about convolution is that it does not matter which signal we flip and shift first. We can either choose x or we can choose h. In this case, it is going to be a little bit easier for us to choose h. Siamak Najarian 2024 18 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Convolving infinite-length signals Graph A In Graph A, when we are at 0 in h[-n] plot, for y, we get 1 from x[n] plot times 1 from h[-n] plot. So, y = 1×1 = 1. Graph B In Graph B, for h[-n] plot, we click 1 unit over and then we repeat the same process. So, y = 1 + (𝜶)(𝟏) = 1+ 𝜶. Graph C In Graph C, for h[-n] plot, we click 1 unit over and then we repeat the same process. So, y = 1 + 𝜶 + 𝜶𝟐 (𝟏) or y = 1 + 𝜶 + 𝜶𝟐. Siamak Najarian 2024 19 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Convolving infinite-length signals Finding the Pattern: Now we see there is a pattern. All the elements before y[-1] are zero. Siamak Najarian 2024 20 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Convolving infinite-length signals The Pattern: Siamak Najarian 2024 21  Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The sum of a finite geometric series A Sidebar: Proof: Siamak Najarian 2024 22 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The sum of a finite geometric series Using the pervious formulas, we get the following: Siamak Najarian 2024 23 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The sum of a finite geometric series Graph B * Graph A One way to think about y[n] is that we are making a cumulative sum of the values of x. In the beginning, we are adding relatively larger sticks to our sum (shown by the region of * in Graph A). So, the sum will grow faster in the beginning and then as we move our “h[-n]” along the signal x[n], the next little bit that we add from the original x[n] is pretty small. It gets smaller and smaller, and finally, we are going to reach a point of diminishing returns and the output signal would look like Graph B. Eventually, the graph tapers off and there is going to be an asymptote (shown by horizonal dashed-line) that we never quite reach. Siamak Najarian 2024 24 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The sum of a finite geometric series Let us now use the Convolution Sum. What does the product of the two step functions in (1) look like? Here, everything is a function of k and when k is negative, the step function of u[k] is 0. So, this is like a step function that does not turn on until k = 0. It means that we can immediately remove u[k] entirely and the sum now turns into a sum that starts (1) at k = 0. This is because we know nothing can be happening before 0. In (2), u[n-k] is like a step function that is flipping and then shifting n units. It is equal 1 for k ≤ n. (2) Based on (3), we managed to get rid of u[n-k] in (2). (3) Siamak Najarian 2024 25 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Symbolic sum of a series in MATLAB MATLAB built-in function symsum(f, k, a, b): The symsum function is powerful for symbolic computations, allowing us to work with exact expressions rather than numerical approximations. The symsum(f, k, a, b) function in MATLAB computes the symbolic sum of a series. Here is a breakdown of its usage:  f: The expression defining the terms of the series.  k: The summation index.  a: The lower bound of the summation.  b: The upper bound of the summation. Example 1: To find the sum of (k2) from (k = 1) to (k = 10): syms k; F = symsum(k^2, k, 1, 10); disp(F); This will output: 385 Example 2: syms k n; F = symsum(k^2, k, 1, n); disp(F); This will output: (n*(n + 1)*(2*n + 1))/6 Example 3 (Indefinite Sum): The following symsum returns the indefinite sum (antidifference): syms k alpha; F = symsum(alpha^k, k, 0, Inf); disp(F); This will output: 1/(1 - α) with the assumption that 0 < α < 1. Siamak Najarian 2024 26 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Properties of convolution/LTI systems A real world system is usually composed of lots of little subsystems put together either in series or in parallel. Property #1: So, we need to know how we can analyze those systems. We want to have an effective impulse response for the entire system and not getting involved in computing the convolution from one response and then go to the next one, and so on. That would be challenging and tedious. Property #1 or Impulse Response: This is basically saying that as long as we know h[n], all we have to do next is to convolve x with h and then we will know how the system acts for any input signal. Siamak Najarian 2024 27 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Properties of convolution/LTI systems Example: How can we test if 𝑦[𝑛] = 𝑛𝑥[𝑛] is an LTI system? Here, instead of a general proof method, let us use the method we discussed before, i.e., the method known as the counterexample, which relies on the fact that a single counterexample that violates the claim is sufficient (enough) to prove that the claim is wrong. Assume that the system was LTI, (the claim is that system is LTI). Then consider the following inputs 𝑥1[𝑛] = 𝛿[𝑛] and 𝑥2[𝑛] = 𝛿[𝑛−1] with corresponding outputs 𝑦1[𝑛] = 0⋅𝛿[𝑛]= 0 for all 𝑛 and 𝑦2[𝑛] = 1.𝛿[𝑛−1] = 𝛿[𝑛−1]. Now if the system is actually LTI, then its output should satisfy 𝑦2[𝑛] = 𝑦1[𝑛−1]. However, one can see that this is not the case, as 𝑦2[𝑛] = 𝛿[𝑛−1] ≠ 𝑦1[𝑛−1] = 0. Hence, this counterexample has proved that the claim was wrong; i.e., the system is not LTI. In fact, what we have actually shown is that the system is not time-invariant, but then the fact that it is also not LTI follows as a consequence. Here, we used the sifting property of the impulse function, i.e., 𝑓[𝑛]𝛿[𝑛−𝑎] = 𝑓[𝑎]𝛿[𝑛−𝑎], which becomes specifically as 𝑛.𝛿[𝑛] = 0.𝛿[𝑛] = 0, ∀𝑛. Siamak Najarian 2024 28 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Commutative property; Distributive property Property #2: Property #2 or Commutative Property: This basically says that if we were to interchange the order of the signals, when we convolve them, we get the same answer. We can keep x the way it is and we flip h and we run h across the signal. We do not have to flip h, we can flip x instead and run it around across h. Let us show that why that is true. We can Property #3: do that by a simple change of variables. Let m = n - k. Here, k goes from minus infinity to plus infinity. That means m also goes from minus infinity to plus infinity. That is the same thing as if we were to interchange the signal order. In the real world engineering, we may find that it is easier to flip one signal than Property #3 or Distributive Property: This states to flip the other signal. that the convolution of a signal x[n] with the sum of two signals h1[n] and h2[n] is equal to the sum of the individual convolutions of x[n] with h1[n] and h2[n]. It has applications in parallel systems. Siamak Najarian 2024 29 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Distributive property Example for Distributive Property: If we have a signal, x[n], that goes Diagram (1) through two pieces, Diagram (1), and h1 then we add those pieces up later to get y[n], an equivalent system is what is shown in Diagram (2). h2 Diagram (2) h1 + h2 Siamak Najarian 2024 30 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Associative property Property #4 or Associative Property: This states that in equation (1), we can put the parentheses wherever we want. We could basically say that we can move these parentheses around or we could (1) remove the parentheses altogether. And, this is important in terms of the way that systems are composed. Note: Composed Systems refer to systems that are constructed by connecting multiple basic digital signal Examples for Associative Property: processing building blocks or subsystems together. In (2), we put a signal through a system with one (2) impulse response and then we put it through some h1 h2 other system and we get our result, y[n]. We could equivalently put x[n] through a single system, shown h1 h2 (3) by (3), whose impulse response is the convolution of h1 and h2. And, if we combine this with the (4) communitive property, shown in (4), we could also h2 h1 interchange the order of these two h’s (because we would get the same answer). Finally, as shown in (5), h2 h1 (5) we could take this apart again and say that it does not matter what order we put the systems in. That is, we can switch the systems themselves. In summary, we can effectively combine all our LTI systems into one glove and also we can put the systems in any order that we want (we should get the same answer). Siamak Najarian 2024 31 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Associative property; Causality and the impulse response Example for which Associative Property fails: What we discussed about the associative property only works for LTI systems. 2 x2 Namely, the same is definitely not true for nonlinear systems. x2 2 Let us look at an example where we have a system in which, first, we multiply x by 2 and then we square. What we get out would be 4x2. Whereas if we were to square the signal first, then multiply by 2, what we would get would be 2x2. So, here, this is a (1) situation where interchanging these two blocks does not give us the same answer. The reason is that x2 block is not linear. In this case, we cannot use the LTI properties. Graph A The system interchangeability only 0 * applies to LTI systems. Property #5 or Causality Property: Causality means that y[n] does not depend on x[n+k] for k > 0. In the convolution sum equation, (1), when k < 0 in x[n-k], it means we are looking to the future of x. So, if we have a causal system, for this to work, the corresponding impulse response value has to be 0 after multiplying future values of x with 0. The impulse response has to look like Graph A. They do not start before 0. The system cannot be doing stuff before it sees the impulse. All the values shown in region *, for h[k] vs. k graph, should be 0. Siamak Najarian 2024 32  Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing The step response and its relationship to the impulse response Property #6 or Step Response: Since the step signal u[n] looks like the sum of the delta functions up to a certain point, (1), then the step response s[n] is equivalently the sum of values of the impulse response h[k] up to a certain point, (2). So, if we know this impulse response, we can get the step response and vice versa. (1) Conclusion: If we know what the step s[n] = u[n] * h[n] response is, we can go back and compute the (2) impulse response and thus compute the response to anything. Practical Application of these Properties: The reason we are studying all these properties is that it simplifies the computational work. They will come in handy in cases where we have to compute what may look like tricky parallel or serial combinations of systems. Before we just brute force our way through a lot of convolutions, we should think about “Can we take pieces of the system and combine them in ways that could possibly make our life easier?”. Siamak Najarian 2024 33 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Differential and difference equations In (1), ak’s and bk’s are called linear constant coefficient. We solve the difference equations such as (1) the same way that we solve differential equations. As shown in (2), the solution is going to be a homogeneous solution plus a particular solution. As a special case of (1), we can have (3). Here, we (1) only have one term, y[n], on the left and other terms are on the right. That is like the systems that we were talking about so far. That is called a finite impulse response system (FIR system). Here, (2) there is really nothing to solve necessarily. We just use convolution to get the answer, whereas once (3) we start combining previous values of y[n], then we have to use (1), the recursive solution, to get the difference equation answer. This is where we have to go through the machinery of homogeneous solution and particular solution. Siamak Najarian 2024 34 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing Differential and difference equations Later on, we will see that convolution and the responses of LTI systems can be greatly simplified with the tools of frequency domain analysis, such as, Fourier transform, and z-transform. To solve differential equations, the Laplace transform was one of the best ways to do it. The same thing is true for difference equations. Here, the z-transform is a powerful tool. Siamak Najarian 2024 35 Lecture 3: Convolution and its Properties ELEC 421: Digital Signal and Image Processing End of Lecture 3 Siamak Najarian 2024 36