Pandas: Data Cleaning and Data Visualization

• Pandas is used for data cleaning.

Lecture Outline

• The lecture covers data objects and attribute types.
• Basic statistical descriptions of data is discussed.
• Data visualization is covered.
• Measuring data similarity and dissimilarity is examined.

Data Sets

• Data sets consist of data objects, each representing an entity.
• An attribute represents a property or characteristic of a data object.
• Attributes are also known as dimensions, features, or variables.
• A collection of attributes describes an object.
• Objects are also referred to as samples, examples, data points, or instances.

Attribute Types

• Nominal attributes are categories, states, or "names of things."
• Example of nominal attributes include hair color {auburn, black, blond, brown, grey, red, white}, marital status, and occupation.
• Binary attributes are nominal with only two states (0 and 1), such as diagnosis.
• Ordinal attributes have a meaningful order or ranking, but the magnitude between values is unknown.
• Examples of ordinal attributes include size {small, medium, large} and grades {A+, A, A-}.

Discrete vs. Continuous Attributes

• Discrete attributes have a finite or countably infinite set of values.
• Examples of discrete attributes include zip codes, profession, or the set of words in a collection of documents.
• Discrete attributes are sometimes represented as integer variables.
• Binary attributes are a special case of discrete attributes.
• Continuous attributes have real numbers as attribute values
• Examples of continuous attributes include temperature, height, or weight.
• Real values can only be measured and represented using a finite number of digits.
• Continuous attributes are typically represented as floating-point variables.

Types of Data Sets

• Datasets are collections of data used for analysis, insights, and decision-making in data analytics.
• Data sets are categorized based on their structure and format.

Structured Data

• Structured data is organized and stored in a predefined format, such as rows and columns.
• Structured data is easy to search, query, and analyze.
• It includes databases (SQL tables), spreadsheets (Excel), and CSV files.
• Common use cases include business reports, financial records, and inventory systems.

Unstructured Data

• Unstructured data lacks a predefined format or structure.
• It is difficult to store and process using traditional databases.
• It includes text(emails, social media posts), images, videos, and audio files.
• Common use cases for unstructured data include sentiment analysis, image recognition, and multimedia analytics.

Semi-Structured Data

• Semi-structured data doesn't conform to a rigid structure but has some organizational properties.
• It combines elements of both structured and unstructured data.
• It includes JSON, XML files, and NoSQL databases.
• Common uses cases include Web data, APIs, and log files.

Data Quality

• Key questions to ask during data quality assessment:
  • What kinds of data quality problems exist?
  • How can we detect problems with the data?
  • What can we do about these problems?
• Examples of data quality problems:
  • missing values
  • outliers
  • duplicate data

Missing Values

• Reasons for missing values:
  • Information is not collected (e.g., people decline to give their age and weight).
  • Attributes may not be applicable to all cases (e.g., annual income is not applicable to children).
• Handling missing values:
  • Eliminate data objects.
  • Estimate missing values.
  • Ignore the missing value during analysis.
  • Replace with all possible values (weighted by their probabilities).

Outliers

• Outliers are data objects with characteristics that are considerably different than most of the other data objects in the dataset.

Duplicate Data

• A data set may include data objects that are duplicates or almost duplicates of one another.
• Examples of duplicate data include:
  • Same person with multiple email addresses.
  • More than one person holding the same name and address.
• Data cleaning is the process of dealing with duplicate data issues.

Practical Application

• Titanic dataset is used for practical application and examples.