Statistical Analysis in Python

Python Modules – Introduction

• Modules are used to categorize Python code into smaller, manageable parts.
• A module is a Python file containing statements, classes, objects, functions, constants, and variables.
• Grouping similar code into modules makes code easier to access and use.
• Modules help organize code logically to improve readability and maintainability.

Python Import From Module

• Python's from statement imports specific attributes (e.g., functions, classes) from a module without importing the whole module.
• This allows you to use the attributes directly without the module prefix.

Example

• Demonstrates importing sqrt() and factorial() functions from the math module. Allows direct use of the functions without the math. prefix.

Locating Python Modules

• The interpreter searches for modules in several locations.
  • First, it checks the current directory.
  • Then, it searches the directories listed in the PYTHONPATH environment variable.
  • Finally, it checks the installation-dependent directories configured when Python was installed.

NumPy Module Introduction

• NumPy is a Python library designed for efficient numerical computations using arrays.
• NumPy arrays are significantly faster than Python lists for numerical operations.
• They are stored contiguously in memory. This enables efficient access and manipulation of elements.

NumPy Module - Arrays

• NumPy arrays (ndarrays) offer data homogeneity (all elements are of the same data type) for efficiency.
• They use a fixed data type and store elements in contiguous memory for fast access.
• NumPy arrays are optimized for latest CPU architectures.

NumPy Arrays vs Inbuilt Python Sequences

• NumPy arrays have fixed size, and resizing leads to the creation of a new array.
• All elements in a NumPy array are of the same data type.
• NumPy arrays are faster, require less syntax, and more efficient than Python lists.

Data Allocation in NumPy Array

• NumPy stores data contiguously in memory for optimized access and operations.
• It uses data buffer, shape, and strides for efficient data access and compatibility with low-level libraries.
  • Data buffer: flat block of memory holding array elements.
  • Shape: defines dimensions along each axis.
  • Strides: defines the number of bytes to step to reach the next element in each dimension.

Creating NumPy Array from a List

• NumPy arrays can be created from Python lists using the array() method.
• The user should import the NumPy module using import numpy as np.

NumPy Indexing

• NumPy indexing allows access to elements by their index values (starting from 0).
• Slicing extracts elements within a specific range.
• Index arrays can be used to index arrays with arrays or other sequences.

Types of Indexing

• Basic slicing uses slice objects, integers, or a tuple of slice objects and integers.
• Advanced indexing uses NumPy arrays or tuples with at least one sequence object, or a non-tuple sequence object, of an integer or Boolean type.

NumPy Basic Array Operations

• ndim: Returns the dimensions of the array.
• itemsize: Calculates the byte size of each array element.
• dtype: Determines the data type of the array elements
• reshape: Provides a new view of an array.
• slicing: Extracts a specific set of elements.
• linspace: Returns evenly spaced elements.

NumPy Array Operations - Examples

• Examples demonstrating addition, subtraction, multiplication, division, power, and remainder operations on NumPy arrays.

Python Modules- SciPy and Matplotlib(Introduction)

• SciPy is a library of numerical routines adding fundamental building blocks for modelling and solving scientific problems.
• Includes algorithms for solving optimization, integration, and interpolation problems; matrices, and special functions.
• Matplotlib is a library that visualizes data via graphical displays.

Key Features/Modules of SciPy:

• Linear Algebra
• Optimization
• Differentiation
• Integration
• Interpolation
• Signal
• Fourier
• Image Processing
• Statistics

Basic Plotting in Python - Matplotlib(Introduction)

• Matplotlib is a comprehensive library for visualizing data, including static, animated, and interactive visualizations.
• It helps in better understanding of data through graphical and pictorial representations.
• Plotting functions (e.g., plot()) draw points or lines connecting points on a diagram
• The plot() function takes the x and y axis coordinates as parameters.
• plt.xlabel(), plt.ylabel(), and plt.title() are used for labeling axes and adding titles to plots.
• plt.show() displays the plotted data.

Data Visualization using Pandas

• Pandas DataFrame plots enable visual representations of statistical data present in data frames
• Basic types of plots: Area plot, Bar plot, Histogram plot, Line plot, Scatter plot, and Box plot etc.
• The user can generate these visualizations using pandas .plot method

Python Pandas - Sorting

• Pandas DataFrame sorting orders the DataFrame based on one or more columns, either ascending or descending.
• sort_values() is used for sorting Pandas DataFrames.
• ascending = False specifies descending order.
• na_position determines the position of missing values.

Pandas Data Structures - Series and DataFrames

• Pandas Series is one-dimensional array with labels.
• Pandas DataFrame is two-dimensional tabular data structure with row and column labels.

Pandas Methods:

• sum(), count(), max(), min(), mean(), median(), std(), describe() provide summary statistics for columns.

Handling Missing Data in Pandas

• Missing data can be handled using isnull(), notnull(), dropna(), fillna(), interpolate() methods.
• .fillna() method replaces missing values. It accepts method='ffill' to propagate the last valid observation forward or method ='bfill' to fill with the next valid
  observation backward.
• dropna() is used to remove rows or columns with missing values.

Python Exceptions

• Errors occur during the runtime of a program.
• Exceptions are a type of runtime error and are specific events that change the program's normal flow.
• Handling exceptions protects the program from unexpected behavior in code . These are identified using .try/except blocks.

Types of Exceptions:

• Examples of common exceptions: SyntaxError, ZeroDivisionError, ValueError, IndexError, and ImportError are mentioned.

• Different exception handling methods using try/except/finally blocks. try blocks enclose potentially risky code. The except block contains code to handle particular exceptions (e.g. TypeError, ValueError). The finally block ensures execution of certain code regardless of exceptions being raised.

Test your knowledge on statistical modeling and data analysis using Python. This quiz covers essential functions, libraries, and techniques used for data interpretation and regression analysis. Ideal for students and professionals looking to reinforce their understanding of statistical concepts in Python.