Data Management: Concepts and Techniques

• The world is data rich, but information poor
• Key steps in data curation are cleaning, integration, selection, and transformation, leading to analysis and visualization

Data Lifecycle

• The stages in data lifecycle are generation, collection, processing, storage, interpretation, visualization analysis, and management
• Data is created or acquired from various sources during generation
• Data is gathered from different sources and prepared for processing during the Collection phase
• Raw data is processed and manipulated to be useable and consistent during processing
• Processed data is securely stored in databases or data warehouses during storage
• Results are interpreted to inform decision-making and drive actions during interpretation
• Insights are presented in graphical or visual formats for easier interpretation during visualization
• Data is examined to extract insights and patterns during data analysis
• Data is organized, maintained, and governed to ensure quality and accessibility during data management

Data Generation vs Data Collection

• Data generation can be either active or passive
• Common sources of data include human-generated data like surveys and forms, machine-generated data like IoT sensors and logs, transactional systems like purchases and banking records, and web scraping

Web Scraping

• Web scraping is the extraction of data from websites
• Considerations for web scraping include how the website content is presented, whether it is structured, and how the extracted content will be saved
• Sentiment analysis is a common application of web scraping
• Possible ways to do web scraping include using libraries in programming languages, browser automation tools, APIs for data retrieval, headless browsers, and no-code or low-code tools

Headless Browsers

• Headless browsers such as Puppeteer and Playwright, have applications in e-commerce price tracking and social media monitoring

Common Data Sources

• Database data, specifically relational databases, consist of tables with unique names, attributes (columns or fields), and tuples (records or rows), modeled using the Entity-Relationship Model
• Data warehouses are repositories of information collected from multiple sources under a unified schema, residing at a single site, and often modeled as a data cube

Data Objects

• Datasets are composed of data objects representing entities
• In sales data, objects can be customers, store items, and sales
• Data objects can also be referred to as samples, examples, instances, or data points
• Data objects stored in a database are known as data tuples
• Data objects are described by attributes and the rows of a database correspond to the data objects, while the columns correspond to the attributes
• The terms attribute, dimension, feature, and variable are often used interchangeably

Types of Data Attributes

• Nominal attributes are relating to name and are categorical, like gender, color, or city, without any inherent order
• Binary attributes are nominal attributes with only two categories and are also called Boolean
• Ordinal attributes have a specific order but lack equal intervals between categories, such as education level or product ratings (low, medium, high)
• Numerical attributes can be interval scaled with no true zero point, like temperature in Celsius or Fahrenheit, or ratio scaled, with a true zero point, like temperature in Kelvin Numerical attributes
• Continuous data has infinite possible values within a given range, such as height, weight, or temperature
• Discrete data has a finite number of values, such as the number of children or products sold

Statistical Description of Data

• Statistical description aims to provide an overall picture of the data
• Central tendency measures the location of the middle or center of a data distribution
• Dispersion measures how the data is spread out
• Common approaches for measuring central tendency include the mean, median, and mode

Summary Statistics using Pandas

• Size includes NaN values, while count excludes them
• value_counts is a shortcut to count the number of entries in each category of a variable
• Operations typically follow a Split-Apply-Combine strategy

Measuring the Dispersion of Data

• Range is calculated as the maximum value minus the minimum value
• Standard deviation measures the average distance between each data point and the mean
• Variance is the average of the squared differences between each data point and the mean
• Quartiles divide the dataset into four equal parts when sorted
• Q1 (1st quartile) is the median of the lower half (25th percentile)
• Q2 (2nd quartile) is the median of the dataset (50th percentile)
• Q3 (3rd quartile) is the median of the upper half (75th percentile)
• Interquartile Range (IQR) is the distance between the first and third quartiles and is calculated as IQR = Q3 - Q1

Measuring Data Similarity / Dissimilarity

• Measuring similarity and dissimilarity helps understand relationships between data points and is also called proximity measurement
• A similarity measure for two objects, i and j, typically returns a value of 0 if the objects are unalike
• The higher the similarity value, the greater the similarity between objects, where a value of 1 typically indicates complete similarity
• A dissimilarity measure works the opposite way
• Data can be represented as either a data matrix or a dissimilarity matrix

Data vs Dissimilarity Matrix

• A data Matrix stores n data objects in a relational table or an n-by-p matrix, where n is number of objects and p number of attributes and is also called a "two-mode" matrix
• A dissimilarity matrix stores a collection of proximities available for all pairs of n objects
• d(i, j) represents the measured dissimilarity or difference between object i and j
  • sim(i, j) = 1 - d(i, j), where sim is the similarity
  • Also called a one-mode matrix

Measuring Dissimilarity - Nominal Attributes

• Dissimilarity between nominal attributes is measured based on whether the attributes match or differ

Euclidean Distance

• Measures the straight-line distance
• Diagonal movement is allowed
• The shortest distance between any two points
• Calculated as d(x,y) = √(x1 - y1)² + (x2 - y2)² in a two-dimensional plane
• Extended to n dimensions: d(x,y) = √Σ(xi - yi)²

Manhattan Distance

• Measures the sum of the absolute differences of the coordinates
• Calculated as d(x, y) = |x1 - y1| + |x2 - y2| in a 2-D plane
• In n-dimensional space d(x, y) = ∑ |xi - yi|

Minkowski Distance

• Minkowski distance is a generalization of the Euclidean and Manhattan distances
• Defined as d(x,y) = (∑|xi-yi|^p)^(1/p) for p≥ 1
• If p=1, then Minkowski distance is the same as Manhattan distance (L1 - norm)
• If p=2, then the Minkowski distance is equivalent to the Euclidean distance (L2-norm)

Chebyshev / Supremum Distance

• Measures the maximum difference along any coordinate dimension between two points in a multidimensional space
• It is also known as Lmax, L∞ norm, or uniform norm
• dChebyshev(P, Q) = maxi(|pi - qi|)

Mathematical Properties

• Distance is always non-negative, d(x, y) ≥ 0
• Distance between a point and itself is always 0, d(x, y) = 0 if and only if x = y Symmetry
• The order of points doesn't matter in distance calculation, d(x, y) = d(y, x) Triangle inequality
• The maximum difference between x and z along any dimension cannot be greater than the sum of the maximum differences from x to y and from y to z
• d(x, z) ≤ d(x, y) + d(y, z)

Jaccard Distance

• Quantifies the dissimilarity between two sets of data
• Derived from the Jaccard Index (or Similarity Coefficient) Jaccard Distance
• 1-Jaccard Index
• Jaccard Index= |A∩B| / |AUB|
• A∩B: The number of elements common to both sets A and B (intersection)
• AUB: The number of unique elements in either set A or B (union)
• Jaccard Index ranges from 0 to 1, where 1 indicates identical sets and 0 indicates disjoint sets
• Use case : Compare documents or text similarity or compare sets of pixels in images for similarity

Explore data management principles covering creation, storage, access, and preservation. Learn about data quality assessment to address inconsistencies, and data integration techniques to resolve redundancies. Discover attribute selection and sampling methods for effective data reduction.