Introduction to Statistics

Introduction to Statistics

• Statistics involves the collection, classification, analysis, and interpretation of numerical data.
• It helps draw inferences based on quantifiable likelihood or probability.
• Statistics interprets large data sets that are otherwise unintelligible through ordinary observation.
• Data tends to behave in predictable ways, leading to regular patterns.
• Statistics is subdivided into descriptive and inferential statistics.

Inferential Statistics

• It uses mathematical methods and probability theory.
• Used for deducing population properties from a sample data analysis.
• Concerned with the precision and reliability of the inferences drawn.

Descriptive Statistics

• It uses mathematical methods like mean, median, and standard deviation.
• Summarizes and interprets data set properties without inferring population properties.

Descriptive Statistics

• Arithmetic mean is the sum of values divided by the number of values.
• If the data is in a frequency distribution, each x-value is multiplied by its frequency f, and the products are summed
• The denominator is the sum of the frequencies (Σf) in this case.
• The median is the middle value in an ordered set of values (ascending or descending).
• The median formula calculates the position of the middle value.
• The mode is the value that appears most often in a data set.

Data Collection and Presentation

• Data comprises raw, unorganized facts needing processing.
• Processed, organized, and structured data in a given context is called information.
• Data is classified into primary and secondary data.
• Primary data is collected specifically for a problem and is more reliable because it's directly obtained.
• Collecting primary data is time-consuming and costly, with delays before information is ready.
• Secondary data is collected for other purposes, which is quicker and less expensive to obtain.
• Secondary data may not always match specific requirements or be as reliable as primary data.

Presentation of Data: Linear Graphs

• A linear graph pictorially represents x values with associated y values.
• Pairs of x and y values are plotted on a graph paper.

Frequency and Frequency Distributions

• Frequency: The count of units associated with each value of a variable.

Frequency Distribution

• Systematic presentation of variable values along with corresponding frequencies.
• Presented in a tabular form called a frequency table.
• Discrete frequency distribution: Class intervals are absent.
• Continuous frequency distribution: Class intervals are present.
• Steps to convert raw data into a frequency distribution:
  • Identify the range of given values.
  • Determine how often each value occurs within that range.
  • Use a tallying procedure for greater accuracy.

Histogram

• Graphs grouped frequency distributions.
• A summary graph shows data points falling in various ranges.

Frequency Polygon

• Another graphical way to present frequency distribution.
• Useful to compare two or more data sets.
• To construct it:
  • Plot frequency density against the class midpoint of an interval.
  • Join the points with straight lines.

Cumulative Frequency Polygon (Ogive)

• Uses cumulative frequencies of the frequency distribution.
• Cumulative frequencies plotted on the Y-axis against upper-class boundaries.

Pie Chart

• Easily understood way to depict percentage or proportional breakdowns of a total.
• Each category's percentage is calculated and represented as a sector of a circle.
• Sector area is proportional to the percentage, aiding in comparing totals.

Data Analytics

• It is the process of examining data sets to draw conclusions, often using specialized systems and software.
• Data analytics technologies and techniques are widely used to enable more-informed business decisions.
• Data analytics refers to applications ranging from basic business intelligence to advanced analytics.

Types of Data Analytics

• Descriptive: Answers what happened using tools to summarize large datasets.
• Diagnostic: Answers why things happened, supplementing descriptive analytics to find the root cause.
• Predictive: Answers what will happen in the future, using historical data to identify trends.
• Prescriptive: Answers what should be done, using insights from predictive analytics for informed decisions.

Introduction to Business Intelligence (BI)

• Business intelligence involves methods, processes, architectures, applications, and technologies.
• Transforms raw data into useful information.
• Enables more effective strategic, tactical, and operational insights and decision-making.

Data

• Raw value elements or facts
• Types of data include numeric vs. textual, structured vs. unstructured, standard vs. proprietary formats.

Information

• Result of collecting and organizing data, providing context and meaning.

Knowledge

• Understanding information, providing insight and actionable intelligence.

Common Data Problems

• Lacking necessary data, information overload, data scattered across systems, and difficulty in data access.

BI as a Decision Process

• Decisions can be made based on facts, simulation, intuition, and group negotiation.
• Traditionally, BI is understood as a Decision Support System (DSS) which contributes to decisions using data.
• Perspectives of BI include a generic decision-making process and an information system.

BI Platforms

• Enable scaling the platform, optimizing performance, and ensuring high availability and disaster recovery.
• Cloud BI capabilities for building, deploying, and managing analytics in the cloud and on-premises.

Data Management

• Includes governance & metadata management, & self contained extraction, transformation & loading & data storage.

Analysis and Content Creation

• Embedded advanced analytics enable easy access to advanced analytics capabilities either internal or integrated.
• Analytical dashboards create highly interactive dashboards and content.
• Mobile exploration allows content delivery to mobile devices.
• Embedding analytic content involves a software developer's kit with APIs and support.
• Publishing analytic content involves publishing, deploying, and operationalizing content.

BI System (Components) at a Glance

• The value of BI Systems is to provide an integrated data processing platform, enable data access at all levels, and streamline data driven decision making.

Data Management limitations

• Transaction oriented i.e. optimized for data insert, update, move, etc.
• Not optimized for complex data analysis
• Individual databases manage data differently.

Data Gathering and Integration

• Enterprise level data comes from multiple sources like operational databases, spreadsheets and must be associated in.
• Reasons that data has to be collected: autonomous, distributed or different.
• Generic data processing steps are extraction, transformation and loading.

Analysis tools used in Business Intelligence processes

• Descriptive reporting is used.
• It is structured and fixed format reports, based on simple queries.

OLAP (Online Analytical Processing)

• Multi-dimensional analysis and reporting application for aggregated data. Great for discovering details from large quantities of data.
• Dimension example:
  • What is the total sales amount grouped by product line (dimension 1), location (dimension 2), time (dimension 3) and ... (other dimensions)?
  • Which segment of business provides the most revenue growth?

Business Analytics (BA)

• Iterative, methodical exploration of an organization's data with emphasis on statistical analysis.

Data Mining Techniques

• Processes and techniques for seeking non-trivial, non-obvious knowledge from extremely large datasets.

Data Presentation/Visualization Tools for BI

• Reports present detailed data in defined layouts and formats
• Dashboards visually display the most important information needed to achieve objectives on a single screen.
• Scorecards use tabular visualization to see how the performance is against targets at a glance.

BI Reporting and Delivery

• BI reporting is about managing and delivering analysis results to users.
• Data analytics and data visualization tools are used.

BI Application Areas

• Can be applied in private and public sectors across various functions.

Business Intelligence Trends

• Due to data changes and analytics, trends are increasingly important.

BI and New Terms

• Big data covers non-structure and various data formats including text, blob and multimedia.
• Data Science, focuses on analysis and presentation models and methods.