Computer Vision: A Practical Introduction PDF

COMPUTER VISION A PRACTICAL INTRODUCTION TO COMPUTER VISION WITH OPEN CV 1 Digital Image Processing Representation of binary images on a computer is done using zeros and ones. Each image consists of rows and columns containing a set of pixels, where a pixel is the smallest unit in...

COMPUTER VISION A PRACTICAL INTRODUCTION TO COMPUTER VISION WITH OPEN CV 1 Digital Image Processing Representation of binary images on a computer is done using zeros and ones. Each image consists of rows and columns containing a set of pixels, where a pixel is the smallest unit in the image. The clarity of the image increases with a greater number of pixels. Digital Image Processing means processing digital image by means of a digital computer. We can also say that it is a use of computer algorithms, in order to get enhanced image either to extract some useful information. Digital image processing is the use of algorithms and mathematical models to process and analyze digital images. The basic steps involved in digital image processing are: 1. Image acquisition: This involves capturing an image using a digital camera or scanner, or importing an existing image into a computer. 2. Image enhancement: This involves improving the visual quality of an image, such as increasing contrast, reducing noise, and removing artifacts. 3. Image restoration: This involves removing degradation from an image, such as blurring, noise, and distortion. 4. Image segmentation: This involves dividing an image into regions or segments, each of which corresponds to a specific object or feature in the image. 5. Image representation and description: This involves representing an image in a way that can be analyzed and manipulated by a computer, and describing the features of an image in a compact and meaningful way. 6. Image analysis: This involves using algorithms and mathematical models to extract information from an image, such as recognizing objects, detecting patterns, and quantifying features. 7. Image synthesis and compression: This involves generating new images or compressing existing images to reduce storage and transmission requirements.  8. Digital image processing is widely used in a variety of applications, including medical imaging, remote sensing, computer vision, and multimedia. Image processing mainly includes the following steps: 1. Importing the image via image acquisition tools; 2. Analyzing and manipulating the image; 3. Output in which result can be altered image or a report which is based on analyzing that image. What is an image? An image is defined as a two-dimensional function (x, y), where x and y are spatial coordinates, and the amplitude of F at any pair of coordinates (x, y) is called the intensity of that image at that point. When x, y, and amplitude values of F are finite, we call it a digital image. In other words, an image can be defined by a two-dimensional array specifically arranged in rows and columns. Digital Image is composed of a finite number of elements, each of which elements have a particular value at a particular location. These elements are referred to as picture elements, image elements, and pixels. A Pixel is most widely used to denote the elements of a Digital Image. Types of an Image 1. BINARY IMAGE – The binary image, as its name suggests, contains only two pixel elements, i.e., 0 & 1, where 0 refers to black and 1 refers to white. This image is also known as Monochrome. 2. BLACK AND WHITE IMAGE – The image which consists of only black and white colors is called a BLACK AND WHITE IMAGE. 3. 8-bit COLOR FORMAT – It is the most famous image format. It has 256 different shades of colors in it and is commonly known as 3 Grayscale Image. In this format, 0 stands for Black, and 255 stands for White, and 127 stands for Gray. 4. 16-bit COLOR FORMAT – It is a color image format. It has 65,536 different colors in it. It is also known as High Color Format. In this format, the distribution of color is not the same as in Grayscale images. A 16-bit format is actually divided into three further formats which are Red, Green, and Blue. That famous RGB format. PHASES OF IMAGE PROCESSING 1. ACQUISITION – It could be as simple as being given an image in digital form. The main work involves: a) Scaling b) Color conversion (RGB to Gray or vice-versa) 2. IMAGE ENHANCEMENT – It is among the simplest and most appealing areas of Image Processing. It is also used to extract some hidden details from an image and is subjective. 3. IMAGE RESTORATION – It also deals with the appeal of an image but it is objective (Restoration is based on a mathematical or probabilistic model or image degradation). 4. COLOR IMAGE PROCESSING – It deals with pseudo-color and full-color image processing; color models are applicable to digital image processing. 5. WAVELETS AND MULTI-RESOLUTION PROCESSING – It is the foundation of representing images in various degrees. 6. IMAGE COMPRESSION – It involves developing some functions to perform this operation. It mainly deals with image size or resolution. 7. MORPHOLOGICAL PROCESSING – It deals with tools for extracting image components that are useful in the representation & description of shape. 8. SEGMENTATION PROCEDURE – It includes partitioning an image into its constituent parts or objects. Autonomous segmentation is the most difficult task in Image Processing. 9. REPRESENTATION & DESCRIPTION – It follows the output of the segmentation stage; choosing a representation is only part of the solution for transforming raw data into processed data. Advantages of Digital Image Processing: 1. Improved image quality: Digital image processing algorithms can improve the visual quality of images, making them clearer, sharper, and more informative. 2. Automated image-based tasks: Digital image processing can automate many image-based tasks, such as object recognition, pattern detection, and measurement. 3. Increased efficiency: Digital image processing algorithms can process images much faster than humans, making it possible to analyze large amounts of data in a short amount of time. 4. Increased accuracy: Digital image processing algorithms can provide more accurate results than humans, especially for tasks that require precise measurements or quantitative analysis. Disadvantages of Digital Image Processing: 1. High computational cost: Some digital image processing algorithms are computationally intensive and require significant computational resources. 2. Limited interpretability: Some digital image processing algorithms may produce results that are difficult for humans to interpret, especially for complex or sophisticated algorithms. 3. Dependence on quality of input: The quality of the output of digital image processing algorithms is highly dependent on the quality of the input images. Poor quality input images can result in poor quality output. 4. Limitations of algorithms: Digital image processing algorithms have limitations, such as the difficulty of recognizing objects in cluttered or poorly lit scenes, or the inability to recognize objects with significant deformations or occlusions. 5. Dependence on good training data: The performance of many digital image processing algorithms is dependent on the quality of the training data 5 used to develop the algorithms. Poor quality training data can result in poor performance of the algorithm. Difference between Image Processing and Computer Vision Image processing and Computer Vision both are very exciting fields of Computer Science. Image processing and Computer Vision are both very exciting fields of Computer Science. Computer Vision:  In Computer Vision, computers or machines are made to gain high-level understanding from the input digital images or videos with the purpose of automating tasks that the human visual system can do. It uses many techniques and Image Processing is just one of them. Image Processing:  Image Processing is the field of enhancing the images by tuning many parameters and features of the images. So, Image Processing is the subset of Computer Vision. Here, transformations are applied to an input image and the resultant output image is returned. Some of these transformations are sharpening, smoothing, stretching etc. Now, as both the fields deal with working in visuals, images and videos, there seems to be a lot of confusion about the difference between these fields of computer science. In this article, we will discuss the difference between them The stages of image analysis processing can be outlined as follows: 1. Pre-processing This stage is used to identify and remove noise (such as dots, speckles, and scratches) and irrelevant visual information that does not affect the results of the areas to be processed later. 2. Data Reduction: This stage is used to reduce the data in the spatial domain and transfer the result to another place called the frequency domain. We record properties (frequency domain = spatial domain) of the analysis used for processing. 7 Primary processing is divided into sections: 1. Image engineering for a specific internal region or a particular internal copy will use the derived features of a specific region called ROI, where certain operations are adjusted by spatial coordinates used in image engineering processes, including (Group) or (Zoom) for enlargement, reduction, or transfer rotation. Subsequently, a partial image is obtained for further processing. 1- Zoom Process Method: 1. There are methods for processing (Zoom), and the first method is the Zero-Order-Hold method, which involves repeating the pixel values of previous rows and columns, for example cleaning up a row to zoom in on rows or adding rows and columns simultaneously to enlarge the matrix or column. Example// You have the following part of the required image: 2. Zoom it using the (Zero-Order-Hold) row by row method. 3. Zoom it using the (Zero-Order-Hold) column by column method 4. Zoom it using the (Zero-Order-Hold) row and column method. Solution: 1. The output will be a matrix of size 3×6.  40 40 20 20 10 10 70 70 50 50 30 30 90 90 80 80 10 10 2. The result will be a matrix with a size of 3×6. 40 20 10 40 20 10 70 50 30 70 50 30 90 80 10 90 80 10 3. The output is a 3×6 matrix 40 40 20 20 10 10 40 40 20 20 10 10 70 70 50 50 30 30 70 70 50 50 30 30 90 90 80 80 10 10 90 90 80 80 10 10 How to find the average: Finding the average between two adjacent pixel values and putting the value between them, such as 4-8. We add it = 12. We divide by 2, so the average value becomes 6. The result is written as 4-6-8. If we use this method by averaging row pixels, the columns will increase, and if we use columns, the rows will increase. 9 We can work with two pixels in each row and each column, and we can expand the columns and rows together: This method enlarges the capacity of the N*N matrix to become an image matrix of size (2n-1-2n-1). Example If we have a 3×3 matrix that represents part of the values of the digital image, we need to expand the columns and rows together. Solution: The size of the matrix becomes 5 x 5 Example for clarification: We have the following matrix on which the rows and columns will be expanded together. Expand The rows and columns are operated on the resulting matrix, not the original.  4. Zoom using factor (k): This means, for example, that the image (matrix) is enlarged, for example, 3 times its size. This means that the factor K=3 multiplies it by the capacity of the matrix. If what is required is to enlarge a matrix (part of an image), enlarge it three or four times or something else. We use what is called the k factor and we do the following: 1. Subtract the value of each of two adjacent values. 2. Divide the result by the magnification factor (K). 3. Add the result to the smallest value and continue adding to all elements by (k-1). 4. Apply these steps to rows and columns. Example: You have a portion of the following image that you want to enlarge by 3 times its original size Solution: We take each of two adjacent values, subtract the smaller one from the larger one, then divide the result by 3, then add the result of the division to the smallest value, and it becomes 11 That is, we add 5 twice, and the result becomes two numbers between 125 and 140 [ 125 130 135 140 ] Then we take the other two adjacent numbers, which are 140 and 155 The matrix becomes as follows: Computer vision modeling Image Algebra Algebraic operations are divided into mathematical operations (arithmetic operations) and logical operations. Mathematical Calculations: Addition: The addition process is used to collect information from two images by combining the elements of the first image with the second, starting with the first element of the first image with the first element of the second image, and so on for the rest of the elements. We use the addition method to restore or number the image. Image Restoration and to add noise to the image (as a type of encryption). Example: You have parts of the following two images. The first image is I1 and the second image is I2. What is required to add these two parts? Solution Example: If the plural is used for Noise, what is the way to break the two images back? Answer: We will depend on the result, so we subtract one of the two matrices from the result. The matrix resulting from the subtraction in the noise matrix starts from the target and we penetrate the target’s probabilities using the probabilities of the search space. Subtraction The subtraction process is used to subtract information from two images, so that we subtract every element in the first image. Example: You have the following two images. You need to subtract the two images? 13 Solution: Multiplication The process is done by multiplying the matrix elements of the image by a factor greater than one and is used. Increase or shrink the image For example: The factor K must be greater than one when you want to enlarge the image Example: You have the following image to scale up and down using one of the digital image jigsaw operations. Answer: Using the multiplication process: 1- Increase it 2- Decrease it We use multiplication as algebraic mathematical operations, for example, we multiply this matrix by a factor (here we choose the factor now, it was not specified for us in the question) The coefficient K=3 is greater than one if it increases If the image is reduced, multiply by 3-less than one. 15 Example: we have the matrix (part of an image), which is the result of the increment process, and (K=3) required to find the original matrix? Answer: There are two ways to solve: 1- We divide the matrix by the factor K, where each value in the resulting matrix is divided by Parameter (3) produces the original matrix. 2- In terms of the decrease, such that you use the factor K less than one, for example, K=1/3. Note: 1 < K if increased makes the image tend to be whiter 1 > K in case of decrease in (shrinking) and here the image tends to black (darkness) Division: - The elements of the given image are divided by a factor greater than one. The division process makes the image dark. - Such as // You have the following matrix, which is part of an image. You need to divide the image by a factor of K=4 Solution Logical operations: Logical AND operation: Logical operations are applied to the elements of the image after converting each element of the image to the binary state, so that logical operations can be used in it through the (ROI) method. AND is considered similar to the multiplication process, meaning that the image tends to be white and is done through a white square with Image elements so that the output is the part of the image corresponding to the white square. (AND: makes the background of the part we want white, while OR makes the background of the part we want black) Logical OR operation it is done by taking a black square and a white background for the required image data from the original image, and the OR process is similar to the addition process. The logical operation NOT It is used to give negative values to the original image, meaning it deviates the opposite of the image (e.g., negative camera film). 17 That is, the image data is reversed, i.e., black becomes white and white becomes black Example: If you have the following image part to use NOT? The image resulting from the NOT process is close to black, and the data of this image must be converted to (Binary) (0,1) format. Example: Apply an AND gate operation to an element of the image so that the first element is 88 and the second element is 111 Solution: We convert the 88 to the binary form (1,0) so that: In the case of NOT, it is for one of the two numbers, so that every zero becomes a one and every one is a zero. First Number Second Number Note: The values here are distorted, so I used a second method to deal with these parameters for logical operations and convert them to binary, and also for gates. (NAND, NOR, XOR) 19 Image enhancement (spatial filters) Filter means a process that filters the image from any remaining impurities, that is, it highlights the features of the part of the image that we want by removing noise and impurities. We use spatial filters to remove noise or improve the image, as these filters apply directly in the image field (directly on the image elements) and not in the frequency domain (transformation), where the image elements are used using one of the transformations such as the Fourier transform and the cos transform. Filters are divided into three types: 1- Mean Filter 2- Median Filter 3- Enhancement Filter The first and second types are used to remove noise, in addition to some applications that give. The form of smoothing the image, 1- Removing noise 2- Smoothing The third type is used to clarify the edges and details in the image, where spatial filters are applied, either by using the elements directly without using and your name, or by using a wrap mask with the elements and their neighbors. The results of the mask can be known as follows: 1. If the sum of the mask’s parameters equals 1, it means high illumination of the image. 2. If the sum of the coefficients is equal to 0, then the image illumination loses, that is, it tends to become black. 3. If the coefficients are negative and wavelike, this means information about the edges. 1- If the coefficients are only waves, there is some kind of distortion in the image. 21 1- Mean Filter - It is a linear filter whose elements are: All of its elements are positive, and because they are all positive, there is distortion in the image, and since the sum of the mask’s elements equals 1, then there is a high loss. Example: Apply the Mean mask to the following image: Solution: We know that the result is two points, so we connect them and the shape becomes linear. 2- -Median Filter It is a nonlinear filter that acts on image elements immediately after selecting a mask through elements where the center of the image is replaced by the value in the middle. Example: Apply Median Filter to the following image part? There is no masker in Median, and we create a masker for it from the elements of the matrix, where we take elements The image and we make a mask in it, and that is in the order of the image elements in ascending order, so it becomes: 1- The first step: Arrange the items in ascending order 2- The second step: We divide the number of elements by 2 to extract the middle location. 3- Step Three: We see what the value of the fifth position is The value of the fifth position is 5 It is not necessary that the value be equal to the intermediate position -4 The fourth step: We change the middle location in the matrix, which is the fifth location (4), which is “equal to the 23  value 5 instead of the middle element in the original matrix, which is 4 (that is, we put 5 instead of 4), and it becomes Example: You have the following image fragment Solution: 1- We take the part (3*3) and arrange it in ascending order 2- We select the element in the middle 3- We find the value in the middle 3=5 4- We replace the element in the middle and write the matrix We take the next part of the matrix, which is also (3*3), and arrange it in ascending order.  Enhancement Filter Image Quantization: - The difference between compression and shrinkage: Image shrinkage is the process of transferring image data by removing some image information by projecting a group of image elements to a single point. This process of shrinkage (quantization) takes place. - As for compression: we deal with the image itself as a file, while shrinking may delete part of the image and we deal with the values of the image. - There are two ways to reduce the image: 1- Gray Level Reduction - That is, we reduce the color levels of the image, and here it is done on I(r,c). 2- Special Reduction - Here, work is done on the coordinates of the image elements (r, c), the location, for example (1,1). 1- Gray Level Reduction - We can explain its sections as follows: Reducing the gray level consists of three points: 25 A- The first method: Threshold A specific value of color levels is chosen. This value is called a threshold. Any value of image data that is higher than the threshold value becomes one, and if it is lower, its value becomes zero. This means that the number with 256 color levels is converted into binary images. like If the threshold value is 127, apply it to the following values: Solution: - We see that the highest value is 251 and the lowest value is 11 according to the binary model, the threshold is 127, so it is: Example: If the threshold value is 127, apply it to the following values: Solution: All of this value is less than the threshold of 127, so it is zero. In this case, we determine the largest and the least, and we take the middle, so the smallest value is 2, and the largest value is 25, so the middle between them is 12 or 13. B- The second method: It is the OR, AND process, not using the mask Here the bits per pixel are reduced. Example: We want to reduce or reduce the information to eight. If the standard probability for us is 256, for the gray level to 32 levels, we use the AND method to explain this. The smallest number of each place is in AND. Solution: So, these color spectra must be reduced from 256 to 32 spectra, meaning that every 8 bits we put in a cell, after which we take the lowest value in each cell, so here the first cell will have the lowest value, 0, the second, 8, and the third, 16, until it reaches 256, so that the number of these extracted numbers is 32 numbers. But if the OR method takes the largest number in the box, why? Answer: Because R does not take a zero, and the numbers are less than AND by one number starting from the number The second) becomes.............. 32 15 7 to OR takes the largest number from each digit Example: If you have the number of standard levels (256) do you want to reduce it to 16? 27  Answer: This means that it is all 16 bits in one place, so the division is by 16, so we start with the first place from 0 to 16, and so on. C- The third method is ANE and OR using the mask This method is used to shrink the image (reduce it) using a specific mask. Example // If you have diamonds, the following is required to use the AND catcher method to shrink this part of the image depending on the number of bits for each element, which is 8? Solution: The law of gray levels Any 8 comes from (0, 7) In binary, the 7 becomes (111), so the masker is 8 bits and equals 00000111. Then we take the number of a number from the matrix and convert it to binary form and multiply it by the masker, so you get the number zero, which in binary is 00000000 After hitting it with musk, it will be: And 10, which we convert to binary, is 1010 As for the number 255, it is And so on for the rest of the numbers in the matrix 29 2- Special Reduction Reducing space is done in three ways: 1- Average 2- Median 3- Reduction  The first method: the rate This is a method that takes a group of adjacent elements and takes their average. Example: It is required to use the rate method for the following image part Solution: If the overall image rate is: Total Average of Image = (33 +17)/4= 13 If the average is for rows, it is the sum of the row divided by the number of numbers in one row.  The second method: the mediator In this method, the image elements are arranged in ascending order and the value in the middle is taken. Example: You have the following array. What is required: 1- Taking the median of all elements 2- Using a specific mask? - We arrange the numbers in ascending order, so they become: - If the median is the sixth element, i.e. - If the masker is used and the masker is assumed to be 3*3 and we arrange the elements in ascending order, the order of the elements of the first matrix will be 3*3. So, the median is the fifth element and its value is 5 The order of the elements of the second matrix is 3*3 So, the median is the fifth element and its value is also 5 The third method is reduction 31 - Some image data is deleted, for example, the image size is reduced by 2. Here, each row is taken or a column from the image and delete the row and the next column/ Example: You have the following image part that needs to be reduced by 2 by columns? So, the second and fourth columns are deleted, meaning the matrix becomes as follows: \ If the reduction is by 3, we delete two columns, the second and third, meaning 2 = 1 - 3. So, the matrix becomes a sequence In this case, if what is required is to reduce by 3 rows, the answer is not possible because the matrix consists of 3 rows, not  Histogram modification - A chart that uses the gray levels of an image distributes these levels of the image so that the part of the image that contains the information fills the chart and the rest of the space is empty, depending on the values of the chart's image points. There are many of these modified levels that we can mention as follows: 1. Histogram with a small spread of low contrast levels 2. Image. 3. Histogram with a large spread of High Contrast gray levels 4. Image. 5. Histogram clustered at the low-end Dark Slide Image 6. Histogram clustered at the top end White Slide Image The process of changing the histogram is done in three ways: Histogram Stretching Histogram Shrink (compressed) Slide of Histogram The first method: Histogram Stretching 33 The histogram can be expanded according to the following law: whereas: 1. The largest gray level value in the image I (r,c)max 2. The minimum gray level value in the image is I(r,c) min 3. The possible minimum and maximum gray level values depend on (255.0) Max & Min Example: You have the following image part to expand this part of the image using the histogram expansion method?  The second method is histogram shrinking The histogram can be reduced according to the following law: 35 whereas: 1. The largest gray level value in the image is I(r,c)max 2. The minimum gray level value in the image is I(r,c)man 3. Depends on the maximum and minimum Shrinkmax & Shrinkmin gray level values the potential is (0, 255). Example: You have the following image part to shrink this part of the image using method Shrink histogram?  The third method: Shift the histogram slide The histogram can be shifted by a certain distance according to the following law: Slide (I (r,c) ) = I (r,c) - OFFSET.................(8) whereas: Offset: The amount by which the histogram is offset by a distance. Example: You have the following part of the image that needs to be shifted by a distance of 10 units using the Histogram slide method. 37 Introduction Computer vision is the automatic analysis of images and videos by computers in order to gain some understanding of the world. Computer vision is inspired by the capabilities of the human vision system and, when initially addressed in the 1960s and 1970s, it was thought to be a relatively straightforward problem to solve. However, the reason we think/thought that vision is easy is that we have our own visual system which makes the task seem intuitive to our conscious minds. In fact, the human visual system is very complex and even the estimates of how much of the brain is involved with visual processing vary from 25% up to more than 50%. 1.1 A Difficult Problem The first challenge facing anyone studying this subject is to convince them that the problem is difficult. To try to illustrate the difficulty, we first show three different versions of the same image in Figure 1.1. For a computer, an image is just an array of values, such as the array shown in the left-hand image in Figure 1.1. For us, using our complex vision system, we can perceive this as a face image but only if we are shown it as a grey scale image (top right). Computer vision is quite like understanding the array of values shown in Figure 1.1, but is more complicated as the array is really much bigger (e.g. to be equivalent to the human eye a camera would need around 127 million elements), and more complex (i.e. with each point represented by three values in order to encode colour information). To make the task even more convoluted, the images are constantly changing, providing a stream of 50–60 images per second and, of course, there are two streams of data as we have two eyes/cameras. 39  67 67 66 68 66 67 64 65 65 63 63 69 61 64 63 66 61 60 69 68 63 68 65 62 65 61 50 26 32 65 61 67 64 65 66 63 72 71 70 87 67 60 28 21 17 18 13 15 20 59 61 65 66 64 75 73 76 78 67 26 20 19 16 18 16 13 18 21 50 61 69 70 74 75 78 74 39 31 31 30 46 37 69 66 64 43 18 63 69 60 73 75 77 64 41 20 18 22 63 92 99 88 78 73 39 40 59 65 74 75 71 42 19 12 14 28 79 102 107 96 87 79 57 29 68 66 75 75 66 43 12 11 16 62 87 84 84 108 83 84 59 39 70 66 76 74 49 42 37 10 34 78 90 99 68 94 97 51 40 69 72 65 76 63 40 57 123 88 60 83 95 88 80 71 67 69 32 67 73 73 78 50 32 33 90 121 66 86 100 116 87 85 80 74 71 56 58 48 80 40 33 16 63 107 57 86 103 113 113 104 94 86 77 48 47 45 88 41 35 10 15 94 67 96 98 91 86 105 81 77 71 35 45 47 87 51 35 15 15 17 51 92 104 101 72 74 87 100 27 31 44 46 86 42 47 11 13 16 71 76 89 95 116 91 67 87 12 25 43 51 96 67 20 12 17 17 86 89 90 101 96 89 62 13 11 19 40 51 99 88 19 15 15 18 32 107 99 86 95 92 26 13 13 16 49 52 99 77 16 14 14 16 35 115 111 109 91 79 17 16 13 46 48 51 Figure 1.1 Different versions of an image. An array of numbers (left) which are the values of the grey scales in the low-resolution image of a face (top right). The task of computer vision is most like understanding the array of numbers 1.2 The Human Vision System If we could duplicate the human visual system then the problem of developing a computer vision system would be solved. So why can’t we? The main difficulty is that we do not understand what the human vision system is doing most of the time. If you consider your eyes, it is probably not clear to you that your colour vision (provided by the 6–7 million cones in the eye) is concentrated in the centre of the visual field of the eye (known as the macula). The rest of your retina is made up of around 120 million rods (cells that are sensitive to visible light of any wavelength/colour). In addition, each eye has a rather large blind spot where the optic nerve attaches to the retina. Somehow, we think we see a continuous image (i.e. no blind spot) with colour everywhere, but even at this lowest level of processing it is unclear as to how this impression occurs within the brain. The visual cortex (at the back of the brain) has been studied and found to contain cells that perform a type of edge detection (see Chapter 6), but mostly we know what sections of the brain do based on localised brain damage to individuals. For example, a number of people with damage to a particular section of the brain can no longer recognise faces (a condition known as prosopagnosia). Other people have lost the ability to sense moving objects (a condition known as akinetopsia). These conditions inspire us to develop separate modules to recognise faces (e.g. see Section 8.4) and to detect object motion (e.g. see Chapter 9). We can also look at the brain using functional MRI, which allows us to see the concentration of electrical activity in different parts of the brain as subjects perform various activities. Again, this may tell us what large parts of the brain are doing, but it cannot provide us with algorithms to solve the problem of interpreting the massive arrays of numbers that video cameras provid 41 1.3 Practical Applications of Computer Vision Computer vision has many applications in industry, particularly allowing the automatic inspection of manufactured goods at any stage in the production line. For example, it has been used to: Inspect printed circuits boards to ensure that tracks and components are placed correctly. See Figure 1.2. Inspect print quality of labels. See Figure 1.3. Inspect bottles to ensure they are properly filled. See Figure 1.3. Figure 1.2 PCB inspection of pads (left) and images of some detected flaws in the surface mounting of components (right). Reproduced by permission of James Mahon Figure 1.3 Checking print quality of best-before dates (right), and monitoring level to which bottles are filled (right). Reproduced by permission of Omron Electronics LLC  Guide robots when manufacturing complex products such as cars. On the factory floor, the problem is a little simpler than in the real world as the lighting can be constrained and the possible variations of what we can see are quite limited. Computer vision is now solving problems outside the factory. Computer vision applications outside the factory include: The automatic reading of license plates as they pass through tollgates on major roads. Augmenting sports broadcasts by determining distances for penalties, along with a range of other statistics (such as how far each player has travelled during the game). Biometric security checks in airports using images of faces and images of fingerprints. See Figure 1.4. Augmenting movies by the insertion of virtual objects into video sequences, so that they appear as though they belong (e.g. the candles in the Great Hall in the Harry Potter movies). 30.8 30.0 29.1 28.3 27.5 26.6 25.8 25.0 24.2 23.3 22.5 °C  Figure 1.4 Buried landmines in an infrared image (left). Reproduced by permission of Zouheir Fawaz,  Handprint recognition system (right). Reproduced by permission of Siemens AG  Assisting drivers by warning them when they are drifting out of lane.  Creating 3D models of a destroyed building from multiple old photographs. 43   Advanced interfaces for computer games allowing the real time detection of players or their hand-held controllers.  Classification of plant types and anticipated yields based on multispectral satellite images. Detecting buried landmines in infrared images. See Figure 1.4. Some examples of existing computer vision systems in the outside world are shown in Figure 1.4. 1.4 The Future of Computer Vision The community of vision developers is constantly pushing the boundaries of what we can achieve. While we can produce autonomous vehicles, which drive themselves on a highway, we would have difficulties producing a reliable vehicle to work on minor roads, particularly if the road marking were poor. Even in the highway environment, though, we have a legal issue, as who is to blame if the vehicle crashes? Clearly, those developing the technology do not think it should be them and would rather that the driver should still be responsible should anything go wrong. This issue of liability is a difficult one and arises with many vision applications in the real world. Taking another example, if we develop a medical imaging system to diagnose cancer, what will happen when it mistakenly does not diagnose a condition? Even though the system might be more reliable than any individual radiologist, we enter a legal minefield. Therefore, for now, the simplest solution is either to address only non-critical problems or to develop systems, which are assistants to, rather than replacements for, the current human experts. Another problem exists with the deployment of computer vision systems. In some countries the installation and use of video cameras is considered an infringement of our basic right to privacy. This varies hugely from country to country, from company to company, and even from individual to individual. While most people involved with technology see the potential benefits of camera systems, many people are inherently distrustful of video cameras and what the videos could be used for. Among other things, they fear (perhaps justifiably) a Big Brother scenario, where our movements and actions are constantly monitored. Despite this, the number of cameras is growing very rapidly, as there are cameras on virtually every new computer, every new phone, every new games console, and so on. Moving forwards, we expect to see computer vision addressing progressively harder problems; that is problems in more complex environments with fewer constraints. We expect computer vision to start to be able to recognise more objects of different types and to begin to extract more reliable and robust descriptions of the world in which they operate. For example, we expect computer vision to  become an integral part of general computer interfaces;  provide increased levels of security through biometric analysis;  provide reliable diagnoses of medical conditions from medical images and medical records;  allow vehicles to be driven autonomously;  automatically determine the identity of criminals through the forensic analysis of video. 45 Figure 1.5 The ASIMO humanoid robot which has two cameras in its ‘head’ which allow ASIMO to determine how far away things are, recognise familiar faces, etc. Reproduced by permission of Honda Motor Co. Inc Ultimately, computer vision is aiming to emulate the capabilities of human vision, and to provide these abilities to humanoid (and other) robotic devices, such as ASIMO (see Figure 1.5). This is part of what makes this field exciting, and surprising, as we all have our own (human) vision systems which work remarkably well, yet when we try to automate any computer vision task it proves very difficult to do reliably. 2 Images Images play a crucial role in computer vision, serving as the visual data captured by devices like cameras. They represent the appearance of scenes, which can be processed to highlight key features before extracting information. Images often contain noise, which can be reduced using basic image processing methods. 2.1 Cameras A camera includes a photosensitive image plane that detects light, a housing that blocks unwanted light, and a lens that directs light onto the image plane in a controlled manner, focusing the light rays. 47 48 2.1.1The Simple Pinhole Camera Model The pinhole camera model is a basic yet realistic representation of a camera, where the lens is considered a simple pinhole through which all light rays pass to reach the image plane. This model simplifies real imaging systems, which often have distortions caused by lenses. Adjustments to address these distortions are discussed in Section 5.6. Y X I Z J Image plane Focal Figure 2.1 illustrates the pinhole camera model, demonstrating how the 3D real world (right side) relates to images on the image plane (left side). The pinhole serves as the origin in the XYZ coordinate system. In practice, the image plane needs to be enclosed in a housing to block stray light. In homogeneous coordinates, www acts as a scaling factor for image points. fif_ifi and fjf_jfj represent a combination of the camera's focal length and pixel sizes in the I and J directions. (ci,cj)(c_i, c_j)(ci,cj) are the coordinates where the optical axis, a line perpendicular to the image plane passing through the pinhole, intersects the image plane. 2.2 Images An image is a 2D projection of a 3D scene captured by a sensor, represented as a continuous function of two coordinates (i, j), (column, row), or (x, y). For digital processing, the image needs to be converted into a suitable format. To process an image digitally, it is sampled into a matrix with MMM rows and NNN columns and then quantized, assigning each matrix element an integer value. The continuous range is divided into intervals, commonly k=256k = 256k=256. 2.2.1 Sampling Digital images are formed by sampling a continuous image into discrete elements using a 2D array of photosensitive elements (pixels). Each pixel has a fixed photosensitive area, with non-photosensitive borders between them. There is a small chance that objects could be missed if their light falls only in these border areas. A bigger challenge with sampling is that each pixel represents the average luminance or chrominance over an area, which might include light from multiple objects, especially at object boundaries. The number of samples in an image determines the ability to distinguish objects within it. A sufficient resolution (number of pixels) is crucial for accurately recognizing objects. However, if the resolution is too high, it may include unnecessary details, making processing more difficult and slower. 2.2.2Quantization Each pixel in a digital image f(i,j)f(i, j)f(i,j) represents scene brightness as a continuous function. However, these brightness values must be discretely represented using digital values. Typically, the number of brightness levels per channel is k=2bk = 2^bk=2b, where bbb is the number of bits, commonly set to 8. Figure 2.2 Four different samplings of the same image; top left 256x192, top right 128x96, bottom left 64x48 and bottom right 32x24 49 50 The essential question is how many bits are truly needed to represent pixels. Using more bits increases memory requirements, while using fewer bits results in information loss. Although 8-bit and 6-bit images appear similar, the latter uses 25% fewer bits. However, 4-bit and 2-bit images show significant issues, even if many objects can still be recognized. The required bit depth depends on the intended use of the image. For automatic machine interpretation, more quantization levels are necessary to avoid false contours and incorrect segmentation, as seen in lower-bit images. Figure 2.3 Four different quantization of the same grey-scale image; top left 8 bits, top right 6 bits, bottom left 4 bits and bottom right 2 bits