Pandas DataFrames: Data Analysis

Pandas DataFrames

• The code reads a CSV file into a Pandas DataFrame named cars.
• The .head() method displays the first few rows of the DataFrame.
• The .info() method provides a summary of the DataFrame, including data types and missing values.
• The .describe() method calculates descriptive statistics for numeric columns.

Working with Data Types

• unemployment.dtypes prints the data types of each column in the unemployment DataFrame.
• unemployment["2019"] = unemployment["2019"].astype(float) converts the data type of the "2019" column to a float.

Value Counts and Filtering

• unemployment['continent'].value_counts() counts the occurrences of each unique value in the 'continent' column, providing a frequency distribution.
• not_oceania = ~unemployment["continent"].isin(["Oceania"]) creates a boolean Series that is True for continents not in Oceania.
• unemployment[not_oceania] filters the DataFrame to exclude records related to countries in Oceania.

Descriptive Statistics and Visualization

• unemployment["2021"].min() and unemployment["2021"].max() calculate the minimum and maximum unemployment rates in 2021.
• sns.boxplot(data=unemployment, x="2021", y="continent") generates a boxplot of 2021 unemployment rates by continent using Seaborn.

Aggregation

• unemployment.loc[:, "2010":"2021"].agg(["mean", "std"]) calculates the mean and standard deviation of unemployment rates for the years 2010-2021.
• unemployment.groupby("continent").agg(["mean" , "std"]) calculates yearly mean and standard deviation grouped by continent.
• The code groups the unemployment data by continent and calculates the mean and standard deviation of the 2021 unemployment rate for each continent.

Handling Missing Values

• planes.info() provides a summary of the DataFrame to identify missing values.
• planes.isnull().sum() counts the number of missing values in each column.
• The code calculates a threshold for dropping columns based on the percentage of missing values.
• planes.columns[planes.isna().sum() < threshold] identifies columns with missing values below the threshold.
• planes.drop(columns=cols_to_drop, inplace=True) removes columns exceeding the threshold of missing values.
• planes.dropna(subset=["column_name"], inplace=True) removes rows with any missing values in the specified columns.

Imputing Missing Values

• planes["column_name"].fillna(planes["column_name"].mean(), inplace=True) fills missing values in a column with the mean of that column.
• planes["column_name"].fillna(planes["column_name"].median(), inplace=True) fills missing values with the median.
• planes["column_name"].fillna(planes["column_name"].mode()[0], inplace=True) fills missing values with the mode.

Detecting and Removing Outliers

Outlier Detection

• Box plots and scatter plots are used to visually identify outliers
• IQR (Interquartile Range) Method:
  • Calculate Q1 (25th percentile) and Q3 (75th percentile).
  • Compute IQR = Q3 - Q1.
  • Define lower bound = Q1 - (1.5 * IQR) and upper bound = Q3 + (1.5 * IQR).
  • Data points outside these bounds are considered outliers.
• Z-Score Method:
  • Calculate the Z-score for each data point.
  • Data points with a Z-score greater than 3 or less than -3 are considered outliers.

Outlier Removal

• Removing outliers involves several strategies depending on the context
  • Remove only if an error causes them
  • Keep them if they provide insights such as high revenue

IQR Outlier Removal Code

• Calculates thresholds to subset data based if price is lower or higher than the calculated values

Handling Date Formats

• divorce = pd.read_csv("divorce.csv", parse_dates=["divorce_date", "dob_man", "dob_woman", "marriage_date"]) automatically parses specified columns as dates when reading the CSV file.
• divorce["marriage_date"] = pd.to_datetime(divorce["marriage_date"], errors="coerce") converts a column to DateTime format, setting unparseable dates to NaT.

Visualizing Time Series Data

• divorce["marriage_year"] = divorce["marriage_date"].dt.year extracts the year from the "marriage_date" column to create a "marriage_year" column.
• sns.lineplot(data=divorce, x="marriage_year", y="num_kids") creates a line plot to visualize the relationship between "marriage_year" and "num_kids".

Correlation and Heatmaps

• corr_matrix = divorce.corr() computes the correlation matrix of the DataFrame.
• sns.heatmap(corr_matrix, annot=True, cmap="coolwarm", fmt=".2f", linewidths=0.5) generates a heatmap of the correlation matrix.

Scatterplots and KDE Plots

• sns.scatterplot(data=divorce, x="marriage_duration", y="num_kids") creates a scatter plot to visualize the relationship between two variables.
• Color effects improve the plot with hue = "", and palette = "".
• sns.kdeplot(...) generates a Kernel Density Estimate plot, showing the distribution of one variable, can accept hue to visualise two variables

Class Imbalance and Cross-Tabulation

• salaries["Job_Category"].value_counts(normalize=True) calculates the relative frequency of each category in the "Job_Category" column, useful for identifying class imbalance.
• pd.crosstab(salaries["Company_Size"], salaries["Experience"]) performs cross-tabulation to show the relationship between company size and experience.

Feature Extraction and Salary Categorization

• salaries["month"] = salaries["date_of_response"].dt.month extracts the month from a datetime column.
• salaries["weekday"] = salaries["date_of_response"].dt.weekday extracts the weekday.
• pd.cut(...) categorizes salaries into different levels based on specified bins and labels.

Hypothesis Creation and Visual Comparison

• Barplots are used to compare salaries based on location, company size, or employment status using sns.barplot().
• usa_and_gb = salaries[salaries["Employee_Location"].isin(["US", "GB"])] filters data to include only employees in the US or GB.

Cloud Computing Basics

• Cloud computing delivers on-demand computing resources over the internet.
• IaaS provides virtualized computing resources.
• PaaS provides a platform for developing and deploying applications.
• SaaS delivers software applications over the internet.
• Cloud deployment models include public, private, hybrid, multi-cloud, and community clouds.
• Cloud roles include cloud architect, DevOps engineer, and cloud security engineer.
• GDPR is a major regulation affecting cloud services, ensuring data protection and privacy.
• Major cloud providers include AWS, Azure, and GCP. AWS is the market leader.

Key-data Structures

• Lists are ordered, mutable collections
• Tuples are ordered, immutable collections
• Sets are unordered collections with unique values
• Dictionaries are unordered collections of key-value pairs.
• To cast a variable is to change from one type to another.