AI Course: Intro and Data Quality

Introduction to Artificial Intelligence Degree Course

• Introduction and Data Quality, lectured by Prof. Dante Conti and Prof. Sergi Ramirez.

Course Resources

• The Guia Docente & Web Site / Atenea is available at:
• https://www.fib.upc.edu/es/estudios/grados/grado-en-inteligencia-artificial/plan-de-estudios/asignaturas/PMAAD-GIA
• https://ramia-lab.github.io/AdvancedModelling/

Project Details

• Project groups consist of 5-6 people, with a total of 4 groups per laboratory.
• Practical Work:
  • Choose a "real-world" problem or case study.
  • Implement algorithms and methods.
  • Write a technical/managerial report.
  • Oral defense.
• R is the primary language, but Python is also acceptable.

Importance of Data in Learning

• Learning involves an iterative process between real-world facts and hypothesized theories.
• Deduction moves from idea/theory/hypothesis to observation.
• Induction moves from observation to idea/theory/hypothesis.
• Statistics, Data mining, and Data Science focus on the inductive phase of learning.
• Data can be represented as Data = Fit + Noise.

Trends Leading to Data Flood

• Exponential increase in data generation and storage,
  • including bank, telecom, scientific, web, text, e-commerce, and social network data.
• Increase in data formats:
  • relational tables, non-structured tables, log files, textual data, and image data.
• Real-time, streaming data sources.

Big Data Overview

• Big Data = Transactions + Interactions + Observations
• Encompasses various inputs like sensors, mobile web, user click streams, and weblogs.
• Includes outputs like user-generated content, sentiment analysis, social interactions, spatial coordinates, and external demographics.
• Characterized by increasing data variety and complexity.

The 8 Vs of Big Data

• Velocity: The speed at which data is generated, collected, and analyzed.
• Volume: The amount of data generated each second.
• Value: The worth of the extracted data.
• Variety: Describes the different types of data generated, often referring to unstructured data.
• Veracity: How trustworthy the data is.
• Validity: Accuracy of the data for its intended use.
• Volatility: The age of the data; fresh data can quickly make stored data irrelevant.
• Visulalisation: Challenges in using the data, as impacted by factors like scalability and functionality.

Data as a Valuable Resource

• Stored data contains information about the generating phenomenon (statistical regularity).
• The goal is to reveal information like models, patterns, associations, trends, and clusters hidden in the data.
• Data is a treasure for organizations if the data quality is reliable.
• All digital interactions can be valuable data sources that can be enhanced through collected data analysis.
• Reveal interesting data insights by selecting & reporting
• SQL queries alone are not sufficient.
• Assembling historical data consistently to store is called data warehousing or data lakes and these are the memory of the company.
• Need to learn from the data.

Data Warehouse Architecture

• Data flows from sources through staging areas to a warehouse, then to data marts, and finally to users who perform analysis, reporting, and mining.
• Key components include operational systems, ERP, CRM, flat files, meta data, summary data, and raw data.

Interdisciplinarity in Machine Learning

• Computer Science is for developing machines or algorithms that solve problems.
• Statistics is designed to make inferences with confidence measures.
• Machine learning is based on statistics to create machines/algorithms that self-program to solve tasks.

Rosetta Stone Comparison of Statistics and Machine Learning

• Statistics terms map to machine learning equivalents:
  • Variables are attributes/features.
  • Individuals are instances.
  • Explanatory variables are inputs.
  • Response variables are outputs/targets/concepts.
  • Models are networks/trees.
  • Coefficients are weights.
  • Fit criteria are cost functions.
  • Estimation is learning/training.
  • Classification is clustering/unsupervised classification.
  • Discrimination is supervised classification.

Data Science Venn Diagram

• Data Science combines Computer Science/IT
• Math and Statistics
• Domains/Business Knowledge.

Data Science Process Steps

• Data Collection
• Data Preparation
• Model Fitting
• Model Evaluation
• Hyperparameter Tuning

Multivariate Data Definition

• Multivariate data arise when researchers/users record values of several variables/attributes on a set of units.
• This leads to a vector-valued or multidimensional observation for each unit.

Data File Characteristics

• Data is multivariate.
• Tables can contain individual variables or counts.
• Useful data types
  • Transaction's data
  • Graphs
  • Similarity matrices, Link data
    • Textual data - Documents, Html/Xml
    • Stream data - Sensors, Podcasting
    • Image data - Medical, Instagram

The Data Matrix

• A multivariate data matrix, X ∈ R^(n×p), has form with x_ij representing the value of the jth variable for the ith unit.
• Theoretical entities describing univariate distributions are denoted by random variables X_1,..., X_p.
• Rows (columns) of X can be written as vectors x_1,x_2,...,x_n or (x_(1), x_(2), ..., x_(p)).

Sample Mean Vector and Covariance Matrix

• The sample mean of the jth variable is calculated as x̄_j = (1/n) Σ x_ij.
• The sample mean vector is x̄ = (x̄_1, ..., x̄_p)^T.
• The sample covariance of the jth and kth variables is s_jk = (1/(n-1)) Σ (x_ij - x̄_j)(x_ik - x̄_k).
• The p × p matrix S = (s_jk) is the sample covariance matrix.

Rows and Columns Characteristics

• Rows of a data file represent individuals or instances, of tens to millions.
• Also referred to as samples, examples, or records.
• Represent repeated units forming a population under study and can be classified, associated, or clustered and characterized by a predetermined set of attributes.
• Use all available data.

Columns Characteristics:

• Each row is defined some features, variables or attributes.
• Variable are individuals that can take several forms (according to a distribition)
• Attribute types:
  • Binary, nominal, ordinal, interval, ratio, textual,...
• Same variables measured in all individuals and same orders.
• Dictionary of variable in the first rows

Variable Types Hierarchy

• Data types includes discrete and continuous, quantitative and qualitative
  • Discrete data that include categorical (nominal/ordinal)
  • Continous that include quantitative (metric)

Nominal Variables

• Nominal variables have distinct categories represented by symbols, and these serve as labels or names.
• No relation is implied among nominal values, with no ordering or distance measure.
• Percentages and tables can be calculated, and bar plots are used for graphical representation. - *Ex: Citizenship (Mexican, German, French) or marital status (single, married, divorced, widow)
• A special case is the binary/dichotomous variable .
• DS algorithms cannot operate on nonimal data directly since the input should be numeric so Binarisation (one-hot-encoding) needs to be performed

Ordinal Variables

• Impose order on values, no distance between values defined.
  • *Ex: size in clothes (XXL > XL > L > M > S) Social status is ordinal
  • (upper class > middle/high > middle > middle/lower > lower class)
• Arithmetic calculations are not possible.
• Includes Tables and percentages and bar plots but emphasizing the ordering of values.
• Internal encoding of ordinal variables preserving the order. (lower class -> 1)

Count or Discrete Data

• Often are the result of counting a variable.
  • *Ex: attribue number of words in a sentence, number of students, number of bugs - Modeled by the poisson distribution

Understanding the Data

• Variables must have meaning & its described in the metadata file
• Role of Variables
  • Response Variables include target, model, and predicting, usually displayed as (Y) -Exploratory Variables include inputs& predictors that we can used to predict past variables (X)
• Data Origin can include what we collect (surveys/sampling) or the Secondary source (public data)

Two Types of Data Files

• Framework depending on with or without response variable. - *Ex: transactions, ecological, survey
• Data to explore ,describe ,association
• Data to build model and predict the response

Data Mining Chain (Batch Mode)

• 1. Preprocessing:
     • Summary, cleaning, Analysis
• 2. Summary
     • Unvariate, Bivariate
• 3. Multi Exploration - Visulization, clustering, profiling
• 4. Modeling - Optimal model, Estimation
• 5. Deployement Communications, usage for a certain context

Advanced Preprocessing

• Inventory Data Sources
• Fix Quality Issues
• Identify Important Features
• Apply Feature Engineering Libraries
• Validate Results
• Repeat or Complete

Additional Steps For Data Preprocessing

• Data profiling
• Data cleansing
• Data extraction
• Data transformation
• Data enrichment
• Data validation

Process

• Correlation between Features
• Statistical Tests
• Recursive Feature Elimination
• Variance Threshhold

Data Processing Steps

• Data Collections with label
• Data Preprocessing, Cleaning and Removing Duplicates, Split into Test, Validation
• Scale, balance and augment Data.

Preprocessing:

• feature selection: filtering variables
• feature extraction variables
• Transformations: -Recoding numerical for categorical -Quantifying a nominal variable -Normalized

Data Cleaning

• Errors & Typo, missing data, outliers
• Non-response is sample surveys
• Drop outs on longitudinal data
• Refusal to answer questions
• Complete case analysis missing data set
• Estimate quantities
• Imputation data

Outlier Detections

• A observation deviates with other observations generates a different mechanism
• Always by data with statistics
• Non-linked events with the current model
• Data that fails in current models

Univariate Detections

• box plot display for exploration, outliers are tagged
  • mild outliers and extreme outliers - Observation X declared an extreme outliers if is outside off Q1-3

Multivariate Detections

• Not univariate
• Then detection is based on computing Mahalanobis
• Distance between I and G that is called generation if is the value of g

Mahalanobis Distance

• The distances for a normal distributed point
• Allow establishment for outliers
• Short distanced occurs oreoften

MultiVariant Outliers

• Problem is corrupted outlies (V is corrupted)
• initialization is G and V
  • Compute Mahalanobis Rank is based on lower DM and updates V&G till convergence

Data Outliers (R)

• Metrics function
  • With Quantiles
• Values
  • Md Values of classical Mahalanobis Distance
  • Rd robust Mahalanobis Distance
  • The cut is in the outlier cuts

Not parametric

• Local Outliers
  • Algorithm is identification metrics based in local dist / DL(Acm.orgh)

Summary Table

• LOF to estimate of outliers and values greater than 1 suggest

Outlier: Random Forest

• Algorithm is Isolation Tree
  • Las Anomalias

Approaches

• Encoding
  • Rencustruction
    • It is important to have consideration in the analysis

Steps

• Cleaning data
• Detecting rare events
• Fraud , net, analysis
• Eliminate theme
• Weight the individuals and diminish stats
• Estimation with generating individual

Autoamte

• Goal number and deciding about automatic data faster
• Example EDA for R

Additional Library

• Data mad
• Python
• Data Explorer/YData
• SmartEDA Dtale
• The Sweet Viz Autoviz