Pandas for Data Handling

Data Handling with Pandas

• Pandas is used for data handling
• The lecture will cover data handling with pandas

Lecture Agenda

• The lecture will cover the ML Pipeline
• The lecture will cover Prerequisites
• The lecture will cover what the pandas library offers
• Resources will be reviewed in the lecture
• Several Lecture Exercises
• Compulsory Assignment 1 will be described

ML Pipeline

• A diagram is displayed for Machine Learning Pipelines
• Feature Extraction and Scaling are essential for Machine Learning Pipelines
• Labels, Training Datasets, Learning Algorithms and Final Models are all parts of the process
• New Data can be predicted with Labels can Evaluation
• Model Selection, Cross-Validation, Performance Metrics, and Hyperparameter Optimization are all used

ML Pipeline: Preprocessing

• Preprocessing data is a crucial step in every ML application
• Raw data often needs processing to become a good format
• Many ML algorithms require scaling for good performance
• Dimensionality reduction is used during the ML Pipeline

ML Pipeline: Preprocessing

• Separation of data into training and test sets is required
• Models should generalize well
• Good data should create a good performance in training and test set
• Feature engineering can create new features from raw data by transformations

ML Pipeline: Learning algorithm

• Many algorithms are available for ML, optimization methods, etc
• Each algorithms has it's own strengths and weaknesses
• The best model is found by comparing performance
• One must choose their performance metrics (accuracy, AUC of ROC, etc.)
• Cross-validation is used for generalization testing
• Hyperparameter optimization is used when fine tuning models

ML Pipeline: Evaluation and prediction

• Estimate generalization error using unseen test data
• Track expected prediction performance for future data
• All transformations applied to the training data is applied to the test data using the same, user-set or ML algorithm-acquired parameters

Prerequisites

• Anaconda, Miniconda, or Python with a pip package manager should be installed
• The required packages should also be installed
• The environment YAML file is located on Canvas
• Conda can create an environment from the environment.yml file, which provides the different dependencies to the project
• Start environment with the command conda activate dat200_env
• Install package ipykernel - which provides the Ipython integration for Jupyter
• The command is python -m ipykernel install --user --name=dat200_env

Pandas Library

• Pandas is a free and open-source software library for Python
• Features fast and highly flexible structures for handling relational tables and time series
• Pandas is a tool for handling spreadsheets in Python
• The 2 main data structures are Series (1D Array) and DataFrames (DF; a 2D array with named rows and columns)
• Every row and/or column in a DF is a Series
• The pandas library is built on top of Numpy

Pandas Resources

• pandas website and documentation is a resource
• pandas community tutorials are offical pandas website incl videos
• RealPython is another resource
• Pandas is a powerful tool with many commands and options
• ChatGPT is a great tool to remember syntax
• A precise question on what to do is a helpful tool

Common tasks with Pandas

• Creating a DataFrame
• Indexing the rows and columns
• Pandas is a vectorized operation
• Examples showcase pandas capabilities

Lecture exercises 1

• Iris dataset should be loaded into a pandas datafram from the web
• The link for Iris dataset is: https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
• Set the column name to sepal_length, sepal_width, petal_length, petal_width, and types
• Set the row names to flower_1, flower_2, flower_3, ..., flower_150

Common tasks with Pandas 2

• Finding unique values in Pandas Dataframes
• Grouping rows in Pandas
• Creating a column based on a conditional in Pandas

Lecture exercises 2

• Find the unique values for the column types in a dataframe
• Compute the cloumn mean for each type
• Create a new column named sepal width >=3 that contains True or False depending on if the cloumn sepal width is >= 3 (True) or < 3 (False)
• Count how many times the sepal width is >= 3 (you can use the column sepal width >= 3 for that)

Common tasks with Pandas 3

• Filtering Pandas Dataframes
• Getting descriptive statistics for Pandas Dataframes
• Counting values in a Pandas Dataframe
• Searching a Pandas column for a value

Lecture exercises 3

• Count how many times each class occurs (Answer: 50 of each class)
• Create three data subsets from the original dataframe, one for each kind of flower
• Use conditional row selection based on the column types
• View the last 10 rows of the columns sepal_length and types

Common tasks with Pandas 4

• Dropping rows and columns in a Pandas DataFrame
• Selecting Pandas DataFrame rows based on conditions
• Sorting rows in Pandas DataFrames
• Applying Operations Pandas DataFrames
• Getting Pivot Tables in Pandas
• Selecting Pandas DataFrame Rows based on conditions

Lecture exercises 4

• View the rows where sepal_length > 5 and petal_width < 0.2
• Make a new DataFrame containig only rows where petal_width is exactly 1.8
• Get the descriptive statistics for the whole data frame and afterwards just columns for the cloumn petal_length
• Remove the rows named flower 55 and flower 77
• Remove the column sepal_width >= 3
• View all rows of sepal length where petal_width is exactly 1.8
• Get the values of the DataFrame stored in a numpy array
• Remove the column types and apply a function named computation to each cell in the DataFrame

Compulsory Assignment 1

• Compulsory Assignment 1 is posted on Canvas
• The tasks in CA1 is similar to the exercises in the lecture
• The CAs should be started early, or it will get overwhelming before the deadline.