Data Wrangling, Cleaning, and Preparation

Data Science Unit 2: Data Wrangling, Data Cleaning, and Preparation

• Unit 2 covers data wrangling, cleaning, and preparation, including techniques for handling large volumes of data.
• It also covers data wrangling processes like cleaning, transforming, merging, reshaping, combining, merging datasets, merging on index, concatenating, combining with overlap, reshaping, and pivoting
• As well as data cleaning and preparation techniques such as handling missing data, data transformation, string manipulation, summarizing, binning, classing, standardization, and outlier/noise & anomaly management.

Challenges in Handling Large Data

• Large data volume poses challenges like overloaded memory and algorithms that run endlessly.
• Adapting and expanding techniques becomes necessary due to the challenges of working with large data.
• Analysis can be affected by I/O (input/output) and CPU starvation, potentially causing speed issues.
• Computers have limited RAM; exceeding it leads to the OS swapping memory blocks to disks, reducing efficiency.
• Algorithms designed for smaller datasets can cause out-of-memory errors when loading large datasets.

Overcoming Large Data Challenges

• Insufficient memory, unending processes, components bottlenecking, and insufficient speed are common problems
• Solutions involve choosing suitable algorithms, appropriate data structures, and the correct tools.
• Data set compression can alleviate memory problems by reducing data set size and affecting computation speed.
• Hard disks offer permanent storage, but writing to them takes more time than changing data in RAM.
• Algorithms should ideally not require loading the entire dataset into memory and support parallelized calculations.

Online Learning Algorithms

• Full batch learning (statistical learning): Involves feeding the algorithm all data at once.
• Mini-batch learning: Feeds the algorithm a subset of observations (e.g., 100, 1000) at a time.
• Online learning: Feeds the algorithm one data observation at a time.

Algorithm Selection

• bcolz: Compactly stores data arrays and utilizes hard drive space when the array exceeds main memory.
• Dask: Optimizes calculation flow and facilitates easier, parallel calculations.
• MAPREDUCE: Useful for parallelizing and distributing tasks.
  • For instance, consider counting votes nationally across 25 parties, 1,500 offices, and 2 million people: tasks can either be gathered and counted centrally, or offices can hand over results to be aggregated by party.

Data Structure Considerations

• Data storage is as vital as the selection of algorithms.
• Data structures affect storage needs and CRUD (create, read, update, delete) operations performance.
• Sparse data sets have relatively little actual information in its entries.

Tree Data Structures

• Trees offer a way of retrieving information significantly faster than scanning tables.
• Trees contain a root value and subtrees of children, with each child potentially having its own children.
• They are often used by databases
• Databases use trees and other data structures to index data (eg using trees and hash tables) to avoid scanning the whole table.

Hash Tables

• Hash tables calculate a key for each data value and store keys in buckets.
• Data can then be retrieved quickly by looking up the right bucket.
• Python dictionaries are an implementation of hash tables.

Tool Selection for Data Handling

• Choosing a tool typically happens after selecting the right algorithms and data structures.
• The right tool is either a Python library or controlled via Python.
• A number of Python libraries exist that deal with large data using smarter data structures, code optimizers and just-in-time compilers.
  • Cython: Requires specification of data types during program development, making compilation and execution faster.
  • Numexpr: A numerical expression evaluator for NumPy, which is often significantly faster than NumPy itself.
  • Numba: Achieves higher speeds using just-in-time compilation to compile code right before it runs.
  • Bcolz: Overcomes the out-of-memory problem when using NumPy, storing and working with arrays in a compressed form. It also uses Numexpr in the background.
  • Blaze: Translates Python code into SQL, with support for handling data stores like CSV, Spark, and more.
  • Theano: GPU access and symbolic simplifications, coming with a just-in-time compiler.
  • Dask: Optimizes calculation flow, allowing efficient execution and distribution.

General Programming Advice

• Leverage existing tools and libraries to avoid reinventing the wheel.
• Maximize hardware potential through adaptations.
• Reduce computing needs by streamlining memory and processing requirements.
• The first reaction most data scientists have when working with large data sets is to prepare their analytical base tables inside a database
• Exploit databases for simple feature preparation.
• Adopt user-defined functions for advanced modeling when available.
• Use optimized libraries like Mahout and Weka.
• Only use advanced modeling when necessary

Hardware Optimization

• Feed the CPU compressed data.
• Switching to the GPU for parallelizable computations can provide higher throughput for computations.
• Use multiple Python threads for your CPU.

Reducing Computing Needs

• Remove as much of the work as possible before data is processed
• Use a profiler to detect slow parts of your code.
• Use compiled code or optimised numerical computation packages where appropriate.
• Compile the code yourself in lower-level languages such as C or Fortran.
• Read data in chunks rather than pulling it all into memory at once.
• Use generators to avoid intermediate data storage
• Use only the absolute minimum data you require.
• Simplify calculations as much as possible.

Data Wrangling Explained

• Data wrangling, also known as data munging, transforms raw data into a valuable, useable format for analytics.
• Data wrangling assures data quality and utility.
• Key operations involve discovery, organization, cleaning, enrichment, validation, and publishing.
  • Discovery: Understand the source data
  • Organization: Structure your raw data once gathered.
  • Cleaning: Remove things like outliers, nulls and duplication
  • Enrichment: Take a step back and check you have enough data to proceed
  • Validation: Apply validation rules to check for consistency throughout your dataset.
  • Publishing: Provision of data notes/documentation with access for other people to use.
• It can be used for things like corporate fraud detection via analysis of multi-party or multi-layered emails
• And to facilitate improved insights into things like customer behaviour analysis.

Data Wrangling Tools

• Spreadsheets/Excel with Power Query are basic manual data wrangling tools.
• OpenRefine is an automated cleaning tool requiring some programming skills.
• Tabula is suited to all data types.
• Google DataPrep is a data service that explores, cleans, and prepares data.
• Data Wrangler is a data cleaning and transforming tool.
• Plotly (with Python) is useful for map and chart data.

Common Data Operations

• When dealing with datasets in pandas, the three primary steps include cleaning, transforming, and merging.
  • Cleaning: Managing missing values, duplicates, and inconsistent data.
  • Transforming: Includes altering data formats, applying functions, and feature engineering.
  • Merging: Combining multiple datasets.

Working with Data in Pandas

• Clean your data by handling missing or inconsistent values using dropna() and fillna() functions in pandas.
• For example, missing values can be replaced with "Unknown" or the mean value using fillna().
• Functions such as apply() can be used to apply data transformations to certain columns.
• Datatypes for a column can be changed as appropriate with the astype() command
• An existing column can be renamed with the rename() command.
• Columns can have their values derived from other columns using the assign() command. df.assign(Age_Squared=df['Age'] ** 2) creates a column "Age_Squared" with the values equal to the age squared.