Working with CSV files in Python

• CSV files store a large number of variables or data
• CSV files are simplified spreadsheets, similar to Excel, but the content is stored in plaintext
• The CSV module is a built-in function in Python that helps parse these types of files.
• Data in a CSV file is organized in rows and columns, separated by commas.
• Each line represents a row, and commas separate them to define cells.
• The csv module is used for data exported from spreadsheets and database in text file format
• Comma-separated value(CSV) format is identified by commas used to separate fields
• Use CSV module for importing and exporting spreadsheets and databases into Python interpreter

Steps to use CSV files with Python

• Save the Excel file with a '.csv' extension
• Save the CSV file in the same folder as the Python file
• Write code to read and write the CSV file

Reading a CSV file

• There are two ways to do this through the reader function and the DictReader class
• The DictReader class opens a CSV file, reads the file, and reads the file using DictReader() class
• DictReader() outputs the data in dictionary format
• In a program, m[:3] will return the first three rows
• To read the code using the reader() class, the row and column values which separate with comma are returned

Writing a CSV file

• Using the writer function or the DictWriter class, the csv module can be used to write to CSV file
• The DictWriter class opens a CSV file, field names are created, a writer is created and then is written into the CSV file

Looping through rows

• Open the CSV file using open(filename.csv) and then perform the operation
• The for loop contains each element from the list, and the second line which will print the row variable

Looping Through Rows

• Creates an empty list called 'model_no'
• Appends data from row[2] to the 'model_no' list
• The code will print a single list after execution

Extracting information from csv file

• Use row[] to extract the information required from particular column

Converting list to set in CSV file

• Import the csv module while manipulating with csv file
• The dataset function in the code removes the duplicated values

Pandas

• Pandas is an open-source Python library for data analysis, introducing two new data structure
• The new data structures are Series and DataFrames

Series

• Series is a one-dimensional labelled array capable of holding any data type
• A Series is a one-dimensional object similar to an array, list, or column in a table
• It will assign a labelled index to each item in the Series.
• By default, each item will receive an index label from 0 to N, where N is the length of the Series minus one.

Series Examples

• Using pd.Series() data structure, the values in the list are arranged in series
• The dtype in output is 'object' because the strings is taken as object data type
• The value in the output list is arranged in series with the index assigned
• An index can be specified for the elements in the list using index=[]
• A series constructor can convert dictionary by using the keys of dictionary as its index
• Use ‘index' keyword to output the index of the values
• Elements that are 'None' in the series will only print “None’ in the output
• Elements can also return with ‘NaN’ if all elements are numbers
• NaN is not the same as the None type

Querying a series

• Use loc() to query about the label
• Use iloc() to query the data using numeric value
• Use 'iloc[]' to query about the particular element in series using numeric position
• Use 'loc[]' to query about the particular element in series using label

Data Frame

• DataFrames is a tabular data structure comprised of rows and columns
• It is defined as a group of Series objects that share an index for the column names
• The Pandas data frame consists of three main components: the data, the index, and the column
• pd.DataFrame() function in Pandas is used to frame the different series object and output the result in two-dimensional form

DataFrame Example

• To display the first five records of the dataset, head() is implemented

Extracting data from DataFrame

• To extract the element by label, use the loc[] attribute
• Pass two values in df.loc[] function to also extract the element if we want only particular column by their mentioned index
• If the index is required along with the columns to be extracted, use this particular form: df.loc[:, ['cost', 'Student']]

Adding Columns to DataFrame

• To add new column use this form: df['Place']=['mall','road','chowk']

Rename A Column Name

• Use df.rename(columns={'Place':'location','Student':'students'}) to rename a column
• New column name which you want to mention needs to be written in this operation
• In this context 'df.rename(columns={}) ‘ syntax, to rename the column is applied

Inplace

• If inplace is False, the operation won't affect the underlying data
• If the inplace is True, there is nothing to print out

Drop

• Axis=1 is used if we want to drop the column, and Axis=0 is used if we want to drop the row.
• Use drop() function to drop the mentioned column

Querying the DataFrames

• The output can return True or False if it satisfies the condition for data
• The Where() statement takes the Boolean masking condition, applies it to the dataframe series and returns a new dataframe of the series of the shape shape
• The count() statement is used to count the occurrence in the dataframe
• Dropna() function is used to remove the row which contain not a number value
• Data can also be filtered or drop based on conditional code

Querying DataFrames Using Logical Operations

• &(and) operation can be used in the two condition and output the result if it satisfies the both condition
• |(or) operation can be used in the two condition and output the result if it satisfies either of the condition

Indexing A DataFrame

• The data is used for to display the index or rows of the dataframe
• Set_index() is used to set the column as an index in the dataframe
• Reset_index() is used to reset the index that is set using set_index().

Handling Missing Values

• Isnull() function returns True for a value if the value is null otherwise returns False.

Handle Missing Values In Pandas

• Tail() function is used to display the last five column from the data.
• Notnull() function returns True if the value is not null and False when value is null
• Fillna() is used to fill the missing values in the CSV file

Groupby

• The Groupby function is applied to analyze panda series by category
• The code finds the mean of the BIRTHS2012 column for each CTYNAME column
• The code can output the mean of BIRTHS2012 column with respect to city name 'Ada county'
• Code calculates the mean over across all the column for each CTYNAME
• use to specify multiple aggregation function at once

Agg() Function

• agg() function is used to aggregate the value for count, min, max, mean

Scales

• Ratio scale: Units are equally spaced, mathematical operations of +-/* are all valid, Ex: height and weight
• Interval scale: Units are equally spaced, but there is no true zero
• Ordinal scale: The order of the units is important, but not evenly spaced, Ex: Letter grades such as A+, A
• Nominal scale: Categories of data, but the categories have no order with respect to one another, Ex: Teams of a sport

Nominal Scales example

• '.AStype()' converts the datatype of one form to another.

Ordinal Scales example

• the ordered attribute is used to arrange the data in an ordered form

Converting to Datetime

• The .Tto_datetime()' statement, converts to datetime format

Timedeltas

• Used to express differences in time
• To find the difference between two timestamps we apply this function

Merging Dataframes

• The merge() function is used to merge the two datasets, by specifying parameters for the type of join (e.g., outer, inner) and the indexes to use for merging.