Data, Information, Knowledge and Wisdom

• This module will discuss AI, ML, and Data Science
• It will cover data and database concepts
• It will also cover Data Analytics, Data-Driven Decision Making and Data Science Life Cycle
• It will cover the principles of Data Science

Data vs. Information

• Data consists of raw, unprocessed facts
• It lacks inherent meaning
• It forms the building blocks of information
• Data management involves generation, storage, and retrieval
• Information is produced by processing data to reveal its meaning
• Context is required to reveal the meaning of information
• Knowledge creation is enabled through information
• Information should be accurate, relevant, and timely for effective decision-making

Data Repositories

• Data is kept in repositories
• Repositories are for machine-processable, understandable formats
• Repositories use database language (SQL) for searching/retrieving
• Repositories support a human-understandable format
• Data is stored in either a File System or a Database Management System (DBMS), or both

Progression of Data

• Raw facts are the most basic
• Processed data comes next
• Actionable information is after that
• Applied knowledge follows
• Wisdom is the pinnacle

Understanding Data

• Data is a collection of facts in raw form
• Data a base building block
• Information is easier to measure and visualize
• Information is a derived second building block
• Knowledge links the data to apply information to achieve a goal and the relevant third building block
• Wisdom is the top of the DIKW hierarchy and is about guiding

Data Object (Database Design Perspective)

• A Data Object consists of Data Instance and Data Schema
• Data Instance are Raw facts or Raw data
• Data Instance Examples are Mr.Somboon Sae-tae, 6288000, Male
• Data Schema is a skeleton structure of data, its characteristics and properties
• Data Schema is defined as an Attribute of data
• The Data Schema for names are Student_Name, Student_ID, Gender

Introducing Databases

• A database is a shared, integrated computer structure
• It stores a collection of raw facts of interest to the end user
• Metadata provides descriptions of data characteristics and relationships
• DBMS manages the database structure
• DBMS allows to store the actual data in the database
• DBMS secures and controls access to the database

Connecting Data and Extracting Knowledge

• Extracting useful knowledge from data are key for data science

Defining Data Science

• Data science is an interdisciplinary field
• It uses scientific methods, processes, algorithms, and systems
• It extracts knowledge and insights from noisy, structured, and unstructured data
• It applies knowledge and actionable insights from it to a range of application domains
• Data science is concerned with collection, cleaning, and anonymizing large quantities of data
• Data science solves real-life problems and analyzing them to initiate meaningful actions
• Data science studies principles, processes, and techniques for understanding phenomena
• Data science includes automated analysis of data

Data-Driven Decision Making

• Data-Driven Decision Making (DDD) refers to decisions based on the analysis of data
• DDD prioritizes data over intuition

Relationship of Data Engineering and Data Science

• Data engineering and processing are critical to support data science
• Data science benefits from sophisticated data engineering

"Big Data"

• Big data means datasets too large for traditional data processing systems
• Big data requires new processing technologies
• Big Data consists of technologies like Hadoop, HDFS, NoSQL, MapReduce, MongoDB, Cassandra, PIG, HIVE, and HBASE
• "They work together to achieve the goal like extracting value from data"

Defining Data Analytics

• Analytics is the communication and discovery or meaningful patterns within data

Analyzing Data

• Analytics relies on Statistics, Programming and Operations Research

Data Mining

• Data analytics is concerned with extraction of actionable knowledge and insights from big data
• Hypothesis formulation that is based often on conjectures gathered from experience is used
• Data mining also utilizes discovering correlations among variables

Level of Analytics

• There are four main levels
• Descriptive Analytics answer "What happened?" questions
• Diagnostic Analytics answer "Why did that happen?" questions
• Predictive Analytics answer "What will happen?" questions
• Prescriptive Analytics answer "Best course of action?" questions

Business Questions

• "Who Are the Most Profitable Customers" is a simple descriptive stats question
• "Is there a difference in value to the company of these customers?" is a testing question
• "What are the common characteristics of these customers?" is a Segmentation/Classification question
• "Will this new customer become a profitable customer? If so, how profitable?" is a prediction

Business Questions: Techniques

• Most business questions are causal: what would happen if?
• Other easier questions are: "what happened in the past?"

Supervised and Unsupervised Learning

• Supervised Learning: is used for Classification and Regression
• Unsupervised Learning: is used for Clustering and Dimension Reduction
• Unsupervised Learning is often used inside a larger, Supervised learning problem such as auto-encoders for image recognition

Supervised Learning Algorithms

• KNN (k Nearest Neighbors)
• Naive Bayes
• Logistic Regression
• Support Vector Machines
• Random Forests

Unsupervised Learning:

• Clustering
• Factor analysis
• Latent Dirichlet Allocation

Data Science Life Cycle

• Includes analysis
• Includes experiments

Stages of cycle

• Data Collection
• Data Management
• Data Analytics
• Presentation/Visualization
• Archiving/Preservation

Main Challenges in Data Science

• include Privacy
• include Security
• include Data Governance
• include Data & Information Sharing
• include Cost/Operational Expenditures
• include Data Ownership
• Includes volume, velocity and variety

Explore the data processing pipeline. Learn about data storage, the progression from raw data to wisdom, and differentiate between information and data. Understand how knowledge builds upon information and the role of analytics.