Data Science Chapter 2

Study Notes

Introduction to Data Science

Data science is a multidisciplinary field that uses scientific methods, processes, algorithms, and systems to extract knowledge and insights from data.
The main purpose of data science is to find patterns within data and use several techniques to analyze and draw perceptions from data.
Data scientist is responsible for making predictions from data and providing insights to decision-makers.

Data Processing Cycle

Data processing is the re-structuring or re-ordering of data by people or machines to increase their usefulness and add value for a particular purpose.
Three steps constitute the data processing cycle: input, processing, and output.
Input: preparing data in a convenient form for processing.
Processing: changing input data to produce data in a more useful form.
Output: collecting the result of the processing step.

Data Types

Unstructured data: data that is not organized in a pre-defined manner or does not have a pre-defined data structure (e.g., audio, video files, music).
Metadata: data about data, which provides additional information about a specific set of data (e.g., when and where photos were taken).

Data Value Chain

The data value chain describes the information flow within a big data system as a series of steps needed to generate value and useful insights from data.
Key high-level activities in the data value chain include data acquisition, data processing, data analysis, and data visualization.

Big Data

Big Data refers to a collection of data that is huge in volume, velocity, variety, and veracity.
4Vs of Big Data: volume, velocity, variety, and veracity.
Big Data is used for analysis to get insights that help with business decisions.

Big Data Life Cycle with Hadoop

Ingesting data into the system: transferring data from various sources to Hadoop.
Processing: storing and processing data in HDFS and HBase using Spark and MapReduce.
Analyzing: analyzing data using processing frameworks such as Pig, Hive, and Impala.
Access: accessing analyzed data using tools such as Hue and Cloudera Search.

Hadoop Ecosystem

Sqoop: transferring data from RDBMS to HDFS.
Flume: transferring event data such as social media data, clickstreams, etc.
Oozie: job scheduling.
Zookeeper: managing clusters.
Lucene: searching and indexing.
Solar: searching and indexing.
Mahout: providing various functionalities such as collaborative filtering, clustering, and classification.

Description

Test your understanding of data science fundamentals, including the role of data scientists, data processing life cycle, data types, and the basics of Big Data and Hadoop ecosystem components. This quiz covers the key concepts introduced in Chapter 2 of a data science course.