Data Wrangling, Cleaning and Preparation

Unit 2: Data Wrangling, Data Cleaning and Preparation

• Unit focuses on data wrangling, data cleaning, and data preparation
• Topics include data handling, data wrangling, data cleaning, and preparation techniques

Data Handling

• Large data can pose challenges, like overloaded memory and algorithms that never stop
• Handling large volumes of data effectively involves adapting your techniques
• Consider I/O and CPU usage to avoid speed issues because of RAM limitations
• Operating systems swap memory blocks to disks when data exceeds RAM, which is inefficient
• Most algorithms load entire data sets into memory, causing errors when dealing with large data
• Problems when handling large data include insufficient memory, endless processes, bottlenecks, and slow speed
• Solutions involve choosing the right algorithms, data structures, and tools

Techniques for Handling Large Data Volumes

• No direct mapping exists between problems and solutions; solutions can address lack of memory and computational performance
• Data set compression can solve memory issues by reducing data size
• Data set compression can affect computing speed by shifting from hard disk to CPU
• Hard disks store data persistently but take more time to write
• Algorithms for large data should avoid loading entire datasets into memory
• The algorithm should ideally support parallelized calculations

Online Learning Algorithms

• Full batch learning (statistical learning) involves feeding algorithm entire dataset
• Mini-batch learning involves feeding a subset of observations
• Online learning involves feeding one observation at a time

Algorithm Selection

• bcolz stores data arrays compactly and uses hard drive memory when needed
• Dask optimizes calculation flow and enables parallel calculations

MapReduce

• MapReduce is designed for easy parallelization and distribution
• Example: Counting votes in a national election with 25 parties, 1,500 offices, and 2 million people
• Options: centrally counting all tickets, or allowing local offices to count for each party before aggregating results

Data Structure Selection

• Key factors are algorithm performance and suitable data structure for storage
• Data structures influence CRUD operations
• Algorithms impact program, data storage matters
• Performance influenced by CRUD (create, read, update, delete)

Sparse Matrix

• Contains little information relative to entries or observations

Tree Data Structure

• Trees allow faster information retrieval than scanning tables
• A tree has a root value with subtrees of children
• Trees are used in databases, with databases using indexes to avoid scanning tables
• Indices based on trees and hash tables can speed up finding observations

Hash Tables

• Calculate a key for each data value and store keys in a bucket
• Retrieving data involves finding the right data bucket
• Python dictionaries are implementations of hash tables

Right Tool Selection

• Select the right tool based on algorithms and data structures
• Tools include Python libraries or tools controlled from Python

Available Tools (Python Libraries)

• Cython can specify data types during program development and runs faster because the compiler has info
• Numexpr evaluates numerical expressions for NumPy, is faster than NumPy
• Numba compiles code for speed improvements
• Bcolz overcomes memory issues and stores arrays in a compressed form, and uses Numexpr for calculations
• Blaze translates Python code to SQL and manages data stores such as CSV and Spark
• Theano works with the GPU, performs simplifications, and includes a just-in-time compiler
• Dask optimizes calculation flow, executes efficiently, and distributes calculations

Tips for Programming with Large Datasets

• Use existing tools and libraries
• Maximize hardware usage; optimize for maximum potential
• Reduce computing needs and minimize memory and processing usage

Utilizing Databases

• First step is to prepare analytical base tables inside a database
• Method preferred when preparing simple features
• Utilize user-defined functions and procedures for advanced modeling

Optimized Libraries

• Creating libraries requires expertise
• Optimized, uses state-of-the-art methods
• Focus on completion, not duplicating work

Hardware Exploitation

• Feeding compressed data to the CPU avoids CPU starvation
• Using the GPU benefits parallel computations
• The GPU offers higher throughput compared to the CPU
• Python threads allow for parallel computations on the CPU

Computing Needs

• Profiling code identifies slow parts
• Use compiled code for functions, especially loops
• Using generators avoids the batch processing of intermediate data storage
• Training data on a sample to minimize data usage if large data algorithm unavailable
• Using math skills simplifies calculations

Data Wrangling

• Is a transformation and mapping of data from "raw" form to more suitable form
• Data Wrangling is also known as "data munging"
• Goal: to ensure data is quality and useful

Data Wrangling Steps

• Discovery involves understanding underlying data
• Organization structures the data
• Cleaning removes outliers, formats data, and eliminates duplicates
• Data Enrichment: to proceed if you have enough data
• Validation: validates data, by confirming dataset consistency
• Publishing provides notes and creates access for users and other applications

Data Wrangling Use Cases

• Fraud detection involves identifying unusual behavior by examining information like emails
• Customer Behavior Analysis: tools can help businesses gain insights via customer behavior analysis

Data Wrangling Tools

• Spreadsheets/Excel Power Query is the basic manual tool
• OpenRefine is an automated cleaning tool
• Tabula if suited for all data types
• Google DataPrep explores, cleans, and prepares data
• Data wrangler is a cleaning and transforming tool
• Plotly (with Python) helps with maps and chart visualizations

Clean, Transform, and Merge

• Three pandas dataset steps are: cleaning, transformation, and merging

Pandas

• Pandas is a Python library providing data structures and tools designed to make analyzing data structured or relational
• Cleaning handles missing values, duplicates, and inconsistent data
• Transforming changes data formats and applies functions
• Merging combines datasets for analysis

Data Cleaning Code Example

• The program utilizes the pandas library to conduct basic data cleaning operations on a dataset containing names, ages, and cities of individuals
• Demonstrates how to fill missing values (NaN) in a DataFrame with default values, using 'Unknown' for the 'Name' and 'City' columns and the mean of existing values for the 'Age' column

Data Transformation Code Example

• "Age_Category" for each individual is determined, designating people younger than 30 as "Young" and others as "Old"
• Applies change to data type to integer
• Rename 'name' column to 'full_name'
• Creation of new column assigned as the square number of 'age'