Dimensionality Reduction

Reducing Dimensionality

• Complexity in classification relies on the number of features (d) and data samples (N) within a d-dimensional feature space.
• As the number of features increases, the amount of training data required increases, which refers to the "curse of dimensionality".
• Computational complexity is reduced when minimizing the number of features.
• Removing unnecessary features reduces extraction costs.
• Data is more easily analyzed and understood with fewer features.
• Humans discern patterns best in one, two, or three dimensions. However, this drastically degrades for four or more dimensions.
• Classifiers perform poorly when retaining features with little discriminative power due to high correlation.
• Simple models need smaller datasets.
• Increasing the number of features will improve classifier performance, but only up to a critical value (N).
• Increasing the number of features above the critical value will reduce performance and result in overfitting.
• The "peaking phenomenon" suggests that discarding certain features can cause the information lost to be compensated by a more accurate mapping in the lower-dimensional space.
• Two main methods exist for reducing dimensionality including feature selection and feature extraction.
• Selecting k features out of d that offer maximum information is known as feature selection. This discards the other (d - k) features.
• Inter/intraclass distance and subset selection are methods for feature selection.
• Feature extraction finds a new set of k (< d) features, which may be supervised or unsupervised.
• Principal Components Analysis (PCA) and Linear Discriminant Analysis (LDA) are the most widely used feature extraction methods. PCA is unsupervised, while LDA is supervised.

What is Feature Extraction?

• Feature extraction is the process of obtaining meaningful patterns (features) from raw data.
• The reasons to use it are to reduce complexity and preserve critical information.
• Examples of feature extraction with:
  • Image Data (Edges, colors, shapes)
  • Text Data (Word frequency, sentence length)
  • Audio Data (Tone, amplitude)

Why Use Feature Extraction?

• Feature extraction is used for dimensionality reduction, illustrated by transforming 1000 pixels into 10 features.
• Feature extraction leads to faster training and higher accuracy, which improves model performance.
• Irrelevant details like background noise can be filtered (noise reduction).

Types of Data

• Images: Feature examples are edges, textures, corners; for example, cat ears in an image.
• Text: Feature examples are word frequency and TF-IDF scores; for example, the word "excellent" in a product review.
• Audio: Feature examples are frequency and amplitude; for example, the tone of a singer's voice.
  • Frequency measures wave cycles over time, and amplitude measures wave intensity.
• Time Series: Feature examples are trends and seasonality; for example, stock price.

Feature Extraction vs. Feature Selection

• Feature extraction creates new features from raw data; for example, extracting edges from an image.
• Feature selection selects existing features; for example, picking "income" for a loan model.

Examples

• Images:
  • Cat images can be parsed with ear shape, tail length, and fur color to classify cat versus dog.
• Text:
  • "The plot was amazing!" is a movie review with a TF-IDF of "amazing" equal to 0.8, that can be used in sentiment analysis.
• Audio:
  • Voice recordings are parsed with frequency(200 Hz), that can be used for speaker identification.

Introduction to Feature Extraction

• An image recognition system may extract edges, textures, or shapes as features. This keeps it from using all the pixels in an image as features.

Feature Extraction Approaches

• Feature extraction techniques can be broadly categorized into four approaches:
  • Geometrical Feature Extraction
  • Statistical Feature Extraction
  • Transformational Feature Extraction
  • Texture-Based Feature Extraction
• The features are then used as inputs to a classifier (neural network or decision tree) for prediction.

Geometrical Feature Extraction

• Focuses on capturing the spatial structure and shape of objects within the data.
• Geometrical, like shape and structure, features are very useful in the field of image processing and computer vision.
• Examples include the size of a region or object (area).
• Other examples include the length of the boundary of an object (perimeter) and the ratio of height of width (aspect ratio).
• Determining how close an object is to a perfect circle is called circularity.
• Geometrical features such as area, aspect ratio, and circularity can be extracted in autonomous driving systems to detect and classify objects (vehicles, traffic signs) in real time.
  • Vehicles: Have a wider aspect ratio and larger area
  • Traffic Signs: Often have a circular/rectangular shape, with specific aspect ratios.
• The system is able to classify quickly and accurately the objects, and it enables safe navigation by extracting geometrical features.

Statistical Feature Extraction

• Captures the distribution and variability of the data.
• Widely used in signal processing, time series analysis, and general data analysis.
• Feature examples are:
  • Mean, median, mode (central tendency of data).
  • Variance and Standard Deviation (spread of data).
  • Skewness, Kurtosis (the shape of data distribution: asymmetry and "tailedness").
  • Correlation (the relationship between variables).
• The medical profession is a good way to describe the system for statistics extraction.
• Statistical features like mean, variance, and correlation can be extracted from patient data (blood pressure, heart rate, cholesterol levels) to predict the risk of diseases such as diabetes or heart disease.
• Mean Blood Pressure indicates the average blood pressure over time.
• Variance in Heart Rate measures variability in heart rate, which can indicate stress or arrhythmia.
• Correlation between cholesterol levels and blood pressure helps identify relationships between risk factors.

Transformational Feature Extraction

• Converts data into a different domain to extract meaningful information.
• The methods are particularly useful for image analysis, signal processing, and time series data.
• Examples are:
  • Fourier Transform converts time-domain data into frequency-domain data.
  • Wavelet Transform captures both time and frequency information.
  • Principal Component Analysis (PCA) reduces dimensionality while preserving variance.
• Transformational features such as Fourier Transform and Wavelet Transform are used to extract frequency components from the audio signal in a speech recognition system.
  • Fourier Transform converts the audio signal into its frequency components, which are used to identify phonemes (basic units of sound).
  • Wavelet Transform captures both time and frequency information, allowing the system to handle variations in speech speed.

Texture-Based Feature Extraction

• Captures the surface characteristics of objects, such as roughness, smoothness, and patterns.
• The extracted features are widely used in computer vision and image processing.
• Texture Features:
• Gray-Level Co-occurrence Matrix (GLCM) captures the spatial relationship between pixels.
• Local Binary Patterns (LBP) describe the local texture of an image.
• Gabor filters capture texture information at different scales and orientations.
• In satellite image analysis, land cover types (forest, urban, water) can be classified by extracting texture feature such as GLCM and LBP to analyze the texture of different regions:
• Forest: Typically has a rough texture with high contrast in pixel intensities.
• Urban areas: Have a more uniform texture with regular patterns (buildings, roads).
• Water: Has a smooth texture with low contrast in pixel intensities.

Face Recognition Example

• The techniques are used in face recognition systems, with the goal of identifying individuals based on their facial features.
• Geometrical Features:
  • Facial landmarks (eyes, nose, mouth) are extracted and geometrical features are calculated, like the distance between eyes, aspect ratio of the face, and circularity of the eyes.
• Statistical Features:
  • Computing statistical features like the mean and variance of pixel intensities in different facial regions.
• Transformational Features:
  • Applying the Fourier Transform to extract frequency components from the facial image, capturing patterns like wrinkles and texture.
• Texture Features:
  • Using Filters to extract texture information from the facial image, capturing details like skin texture and hair.
• Accurately identifying individuals can be achieved by combining these features, even in challenging conditions like varying lighting or facial expressions.