Computer Vision Module Notes - PDF

MODULE 4 - COMPUTER VISION Computer vision is a field of artificial intelligence (AI) and computer science that focuses on enabling computers to interpret and understand the visual world. It involves the development of algorithms and models to analyze images, videos, and...

MODULE 4 - COMPUTER VISION Computer vision is a field of artificial intelligence (AI) and computer science that focuses on enabling computers to interpret and understand the visual world. It involves the development of algorithms and models to analyze images, videos, and other visual data to extract useful information. The ultimate goal of computer vision is to replicate the human ability to see and interpret the visual environment. Key Goals of Computer Vision: 1.​ Image Understanding: Understanding the content of images, such as recognizing objects, scenes, and activities. 2.​ Feature Detection and Analysis: Identifying specific details like edges, corners, textures, or motion. 3.​ Object Detection and Recognition: Detecting objects within an image or video and identifying what they are. 4.​ Segmentation: Dividing an image into meaningful parts or regions, such as separating the foreground from the background. 5.​ Tracking: Following objects as they move through a sequence of frames in a video. 6.​ 3D Reconstruction: Rebuilding 3D models from 2D images or videos. Computer Vision leverages large datasets, advanced algorithms, and powerful computing resources to mimic human visual perception. With the rise of deep learning, its capabilities have significantly advanced, enabling breakthroughs in fields like autonomous driving and real-time video analytics. 4.1 BASIC COMPUTER IMAGE PROCESSING Computer Image Processing involves the manipulation and analysis of digital images using computational techniques. It is widely used in fields like photography, medical imaging, robotics, and computer vision. Tools and Libraries ​ OpenCV: A comprehensive library for real-time image processing. ​ Pillow: Simple image manipulation library in Python. ​ MATLAB: Popular in academia for prototyping. ​ NumPy: Used for low-level operations on image arrays. Applications ​ Object Detection: Locating and identifying objects in images. ​ Face Recognition: Analyzing facial features for identification. ​ Medical Imaging: Enhancing and analyzing X-rays, MRIs, etc. ​ Satellite Imagery: Processing images from space for environmental studies. Computer Vision deals with enabling machines to interpret and analyze visual data, and at its core lies image representation and processing techniques. Image Representation Image representation in computer vision refers to the process of converting an image into a numerical or symbolic form that can be easily understood and processed by a computer. Images are typically represented as a collection of pixels, where each pixel corresponds to a specific color or intensity value. The goal of image representation is to extract relevant features and information from the image, enabling the computer to perform various tasks, such as object recognition, image classification, and image segmentation. Techniques There are several common techniques for image representation in computer vision: Grayscale representation: Images are represented using a single channel where each pixel contains a grayscale value ranging from 0 (black) to 255 (white). This representation is commonly used for tasks that do not require color information. Color representation: For color images, the most common representation is the RGB (Red, Green, Blue) format. Each pixel is represented by three color channels (R, G, and B), with each channel containing an intensity value ranging from 0 to 255. Feature extraction: Instead of using raw pixel values, computer vision algorithms often extract relevant features from images. These features can be edges, corners, textures, or more complex representations like histograms of oriented gradients (HOG) or deep features from convolutional neural networks (CNNs). Histograms: Image histograms represent the frequency distribution of pixel intensities in an image. They can provide valuable information about the image's contrast, brightness, and overall content. Local descriptors: These are representations that describe local regions of an image, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features), which are used for object recognition and matching. Global descriptors: These representations describe the entire image and are often used for image classification tasks. Examples include bag-of-words models, histogram of gradients (HOG), and deep learning-based features like activations from pre-trained CNN models. Deep Learning-based Representations: Convolutional neural networks (CNNs) have revolutionized image representation in recent years. CNNs automatically learn hierarchical representations from images, enabling them to capture complex patterns and features effectively. The choice of image representation depends on the specific computer vision task at hand. Different representations may be more suitable for different applications, and the selection of the most appropriate representation significantly impacts the performance and accuracy of the computer vision system. Image Processing Techniques Image processing is the process of transforming an image into a digital form and performing certain operations to get some useful information from it. The image processing system usually treats all images as 2D signals when applying certain predetermined signal processing methods. Types of Image Processing There are five main types of image processing: ​ Visualization - Find objects that are not visible in the image ​ Recognition - Distinguish or detect objects in the image ​ Sharpening and restoration - Create an enhanced image from the original image ​ Pattern recognition - Measure the various patterns around the objects in the image ​ Retrieval - Browse and search images from a large database of digital images that are similar to the original image Image Filtering An image filter is a technique through which size, colors, shading and other characteristics of an image are altered. An image filter is used to transform the image using different graphical editing techniques. Image filters are usually done through graphic design and editing software. Image Filtering refers to the process of modifying or enhancing an image by applying a mathematical operation (filter or kernel) to its pixels. This operation can either emphasize specific features or suppress unwanted noise and distortions in the image. Image filtering plays a crucial role in preprocessing images for further analysis, such as object detection, edge detection, and noise reduction. Applications of Image Filtering in Computer Vision 1.​ Noise Reduction: ○​ Preprocessing to remove artifacts, like sensor noise or distortion, improving the quality of images for further processing. 2.​ Feature Extraction: ○​ Enhances important features like edges, corners, or textures, which are critical for tasks such as object detection and recognition. 3.​ Object Detection: ○​ Filters such as edge detection help isolate objects in an image by highlighting their boundaries. 4.​ Image Enhancement:  ○​ Filters like sharpening or contrast adjustment enhance the visual appeal or detail in images. 5.​ Preprocessing for Machine Learning: ○​ Images are often filtered to reduce irrelevant information (e.g., noise) and improve the signal, making it easier for machine learning models to analyze the data. Image Enhancement Image Enhancement refers to the process of improving the visual quality of an image or making specific features in an image more apparent. The goal is to make the image more suitable for analysis by increasing the contrast, sharpness, or visibility of particular regions or details. Image enhancement techniques are often applied as a preprocessing step before further analysis, such as object detection, segmentation, or recognition. Applications of Image Enhancement in Computer Vision 1.​ Medical Imaging: ○​ Enhancing X-rays, MRIs, or CT scans to improve visibility of features like tissues, tumors, or fractures. 2.​ Remote Sensing: ○​ Enhancing satellite or aerial images to better analyze geographical features, vegetation, or urban areas. 3.​ Object Detection:  ○​ Improving the visibility of objects in images for better recognition and tracking, especially in low-light or noisy environments. 4.​ Surveillance Systems: ○​ Enhancing video footage to make faces or objects more distinguishable, especially in challenging conditions like low light or motion blur. 5.​ Robotics: ○​ Enhancing images for better object recognition, navigation, and interaction with the environment. 6.​ Image and Video Editing: ○​ Enhancing images for artistic purposes, improving the overall look, or restoring old photos with degraded quality. Feature Extraction The process of identifying and extracting distinctive characteristics or patterns from an image that are useful for tasks like object recognition, classification, and matching. The goal is to reduce the amount of data while retaining essential information, making it easier for machine learning models or algorithms to analyze and understand the image. Types of Features in Computer Vision 1.​ Low-Level Features: ○​ Basic visual features like edges, corners, textures, and colors. 2.​ High-Level Features: ○​ More abstract representations of objects, patterns, or shapes that are used  in higher-level tasks like object recognition. Common Feature Extraction Methods in Computer Vision 1.​ Edge Features ○​ Purpose: Identify boundaries of objects or regions in an image. ○​ Methods: ​ Sobel Operator: Detects edges by computing the gradient of pixel intensities in both horizontal and vertical directions. ​ Canny Edge Detection: A multi-step algorithm that detects a wide range of edges with high precision by applying Gaussian smoothing, gradient computation, non-maximum suppression, and edge tracing by hysteresis. 2.​ Corner Detection ○​ Purpose: Detect points in an image where the gradient changes sharply in different directions. ○​ Methods: ​ Harris Corner Detection: Identifies corners by analyzing the eigenvalues of the image’s gradient covariance matrix. ​ Shi-Tomasi Corner Detection: A modification of the Harris detector, focusing on selecting points with a strong response to corner-like structures. ​ FAST (Features from Accelerated Segment Test): A high-speed corner detection method that is particularly useful in real-time  applications. 3.​ Texture Features ○​ Purpose: Capture the pattern or texture of surfaces in an image. ○​ Methods: ​ Gray Level Co-occurrence Matrix (GLCM): Measures how often pairs of pixel with specific values occur in a specified spatial relationship, providing texture features like contrast, correlation, energy, and homogeneity. ​ Local Binary Pattern (LBP): A texture descriptor that compares a pixel with its neighbors and encodes the result into a binary number, capturing local texture information. ​ Gabor Filters: Used to capture texture and edge information by convolving the image with a set of Gabor filters at different frequencies and orientations. 4.​ Shape Features ○​ Purpose: Extract geometrical or structural features that describe the shape of objects in an image. ○​ Methods: ​ Hough Transform: Detects geometric shapes like lines, circles, or ellipses by transforming image coordinates to a parameter space. ​ Shape Descriptors: Examples include Hu Moments, Fourier Descriptors, and Zernike Moments, which are used to describe shapes in terms of invariant properties. 5.​ Keypoint Detection and Descriptors ○​ Purpose: Identify and describe distinctive key points (points of interest) in an image that are invariant to scale, rotation, and translation. ○​ Methods: ​ SIFT (Scale-Invariant Feature Transform): Detects and describes keypoints that are invariant to scaling, rotation, and noise. The descriptors are robust and widely used in object recognition and matching. ​ SURF (Speeded-Up Robust Features): A faster alternative to SIFT that detects keypoints and generates descriptors for matching objects across images. ​ ORB (Oriented FAST and Rotated BRIEF): A fast feature extraction method that combines the FAST keypoint detector and the BRIEF descriptor, optimized for real-time applications. ​ AKAZE (Accelerated KAZE Features): A feature extractor designed for non-linear scale spaces, providing better performance in real-time applications than SIFT and SURF. 6.​ Histogram-Based Features ○​ Purpose: Describe the color distribution or intensity distribution of an image or region of interest. ○​ Methods: ​ Color Histograms: Count the occurrences of color intensities in each color channel (RGB, HSV, or LAB). Common applications  include color-based object recognition and image retrieval. ​ Histogram of Oriented Gradients (HOG): Describes the distribution of edge directions (gradients) in localized portions of an image. HOG is widely used in object detection, particularly for human detection. 7.​ Deep Learning Features ○​ Purpose: Automatically extract complex and abstract features using neural networks, especially Convolutional Neural Networks (CNNs). ○​ Methods: ​ Pre-trained CNNs (e.g., VGG, ResNet, Inception): CNNs are capable of learning hierarchical features from raw image data. By using pre-trained networks, you can extract features from various layers (like convolutional layers) and use them for tasks like classification, detection, or transfer learning. ​ Fully Convolutional Networks (FCNs): Used for segmentation tasks to detect and label each pixel in an image based on learned features. 8.​ Scale-Invariant Feature Transformation (SIFT) and SURF ○​ Purpose: Identify keypoints that are invariant to scaling, rotation, and translation for object recognition and matching. ○​ Methods: Both SIFT and SURF use a multi-step process of detecting keypoints, filtering out noise, and generating descriptors that describe the local features of the image around each keypoint.  9.​ Bag of Visual Words (BoVW) ○​ Purpose: Represents an image as a collection of visual words, similar to how text is represented as a collection of words. ○​ Methods: ​ Keypoint Descriptor Matching: First, extract keypoints from the image using methods like SIFT or SURF, then quantize these keypoints into a vocabulary of "visual words" through clustering (typically using k-means). ​ Bag of Features: The image is represented as a histogram of visual word occurrences, which can then be used for classification tasks. 4.2 AUTOENCODER What is Autoencoder? Autoencoders are a specialized class of algorithms that can learn efficient representations of input data with no need for labels. It is a class of artificial neural networks designed for unsupervised learning. Learning to compress and effectively represent input data without specific labels is the essential principle of an automatic decoder. This is accomplished using a two-fold structure that consists of an encoder and a decoder. The encoder transforms the input data into a reduced-dimensional representation, which is often referred to as “latent space” or “encoding”. From that representation, a decoder rebuilds the initial input. For the network to gain meaningful patterns in data, a process of encoding and decoding facilitates the definition of essential features. The autoencoder consists of two main parts: 1. Encoder: ​ The encoder takes the input data and compresses it into a smaller, fixed-size representation (often called the latent vector or bottleneck). ​ The encoder typically involves several layers of the neural network, such as fully connected layers or convolutional layers (for images). ​ The encoder reduces the dimensionality of the input data while retaining its essential features. 2. Decoder: ​ The decoder takes the compressed latent vector from the encoder and reconstructs it back to the original data format (such as an image or a sequence of words). ​ The decoder can mirror the structure of the encoder, with layers that progressively expand back to the original data dimensions. Types of Autoencoders: 1.​ Vanilla Autoencoder (Basic Autoencoder): ○​ The simplest form, where both the encoder and decoder are symmetric networks. It is typically used for dimensionality reduction or feature learning. 2.​ Convolutional Autoencoder: ○​ Uses convolutional layers instead of fully connected layers. These are commonly used for image data as they are better at capturing spatial hierarchies and features. 3.​ Variational Autoencoder (VAE): ○​ A probabilistic version of autoencoders. Instead of learning a deterministic encoding, VAEs learn the parameters of a probability distribution (typically Gaussian) for each data point, allowing for generation of new, similar data points. This makes them powerful for tasks like generative modeling. 4.​ Denoising Autoencoder: ○​ Trains the autoencoder to reconstruct the original data from a noisy version of it, helping it learn more robust features. This is useful for tasks like noise reduction in images. 5.​ Sparse Autoencoder: ○​ Adds a sparsity constraint to the hidden layer, encouraging the network to learn more efficient representations by using fewer active neurons. Architecture Of Autoencoder ​ Encoder: The encoder part of the autoencoder compresses the input image into a lower-dimensional representation (latent vector). It typically consists of convolutional layers (for image data) followed by a fully connected layer or a bottleneck layer that holds the compressed feature. ​ Latent Space: This is the compressed representation of the image, typically a smaller dimension than the input image. ​ Decoder: The decoder takes the latent vector from the encoder and reconstructs the original image using deconvolution or transpose convolution layers (for images). The decoder output should ideally resemble the original input image. Train an Autoencoder for Image Reconstruction Step 1: Data Preparation Step 2: Build the Autoencoder Model Step 3: Train the Autoencoder Step 4: Reconstruct Images Using the Trained Autoencoder Applications Of Autoencoders In Image Compression And Denoising Autoencoders are widely used for tasks like image compression and denoising, as they can efficiently learn compressed representations and robust features from images. Image Compression Using Autoencoders Image compression aims to reduce the size of an image while retaining as much important information as possible, allowing for efficient storage or transmission. Autoencoders are highly effective for this purpose, as they can learn compact, low-dimensional representations of images in their latent space. Image Denoising Using Autoencoders Image denoising involves removing noise from an image while preserving the essential structures and features. In this case, autoencoders can be trained to clean noisy images by learning to reconstruct the original (clean) images from their noisy versions. Applications ​ Image Compression for Web and Media ​ Noise Removal in Satellite Images ​ Restoring Old or Damaged Images ​ Improving Image Quality in Surveillance  4.3 TRANSFER LEARNING ​ Explanation​ In transfer learning, a model that was pre-trained for one task is fine-tuned for a new, related task. This allows organizations to avoid the time-consuming and intensive process of training a new model from scratch. ​ Example​ For example, a model that can identify images of dogs can be trained to identify cats using a smaller image set that highlights the differences between the two. ​ Benefits​ Transfer learning can be used to: ​ Solve regression problems in data science ​ Train deep learning models ​ Train convolutional neural networks (CNNs) Key Concepts Of Transfer Learning Pre-trained Model: A model that has already been trained on a large dataset (e.g., ImageNet, BERT) and is reused for a new, related task. Source Task and Target Task: ​ Source Task: The task on which the model is initially trained (e.g., image classification on ImageNet). ​ Target Task: The new task for which the model is adapted (e.g., medical image  classification). Fine-Tuning: Adjusting a pre-trained model for the target task by training it on a smaller dataset, usually by modifying the last layers or adding new layers. Feature Extraction: Using the pre-trained model to extract useful features, and then adding a new classifier for the target task without retraining the entire model. Domain Adaptation: Adapting the model to a new domain where the data distribution is different (e.g., adapting a model trained on natural images to work on medical images). Multi-task Learning: Training the model on multiple tasks simultaneously, sharing knowledge between tasks. Zero-Shot Learning: The ability of a model to perform tasks it was never explicitly trained on, using general knowledge learned from other tasks. Negative Transfer: Occurs when knowledge from the source task harms performance on the target task, usually when the tasks are too different. Knowledge Distillation: A smaller model (student) learns from a larger model (teacher), making the student model more efficient while retaining the teacher’s knowledge. Key Techniques Of Transfer Learning Fine-Tuning: ​ Description: Adjusting the pre-trained model for the target task by retraining it on the new data, typically by modifying or adding layers. ​ When to use: When you have a similar task and enough data to fine-tune the model effectively. Feature Extraction: ​ Description: Using the pre-trained model to extract useful features from the input data, then training a new classifier (e.g., a fully connected layer) on top of those features. ​ When to use: When you have a small target dataset and want to use the pre-trained model's learned features. Domain Adaptation: ​ Description: Adapting a model trained on one domain (source) to work well on a different but related domain (target), even when the data distributions differ. ​ When to use: When the source and target tasks are related but come from different domains (e.g., natural images to medical images). Multi-task Learning:  ​ Description: Training a model to solve multiple related tasks at once, sharing the learned knowledge across tasks. ​ When to use: When you have multiple tasks that are related, and learning them together can improve performance. Zero-Shot Learning: ​ Description: Enabling the model to perform tasks it hasn’t been directly trained on, by leveraging knowledge learned from similar tasks. ​ When to use: When you want the model to generalize to completely new tasks without task-specific training data. Knowledge Distillation: ​ Description: Training a smaller, simpler model (student) to mimic the behavior of a larger, complex model (teacher), retaining most of the teacher's performance in a smaller model. ​ When to use: When you want to deploy a smaller model that still retains the performance of a larger model. Fine-Tuning Fine-tuning is the process of adjusting a pre-trained model on a new, specific task by retraining it with a smaller dataset, usually modifying the last layers of the model to suit the new task. Pre-Trained Models Hyperparameter tuning​ Optimize hyperparameters like batch size, dropout rate, and the number of layers in the model to improve performance. Parameter-efficient fine-tuning (PEFT)​ Update only a small portion of the model parameters to improve performance while minimizing the number of trainable parameters. Evaluate the model​ Use evaluation metrics like accuracy, precision, recall, and F1 score to determine if the model meets your needs. Time and resources​ Fine-tuning usually takes less time and requires fewer resources than training a model from scratch. Leveraging Transfer Learning In Computer Vision Applications Transfer learning is a method where you use a model that’s already been trained on similar data. Instead of starting from scratch, you take a model that already knows how to recognize basic features like shapes and colors. Then, you adjust it to fit your specific task. This approach is faster and easier.