Big Data Analysis and Data Science Quiz

The goal of big data analysis is to use data to obtain insight and understanding.

The DIKW pyramid represents the relationship between Data, Information, Knowledge, and Wisdom. It illustrates the transformation of raw data into meaningful knowledge and wisdom through interpretation and understanding by humans.

The activities in the data science process include Decision Making, Machine Learning, Data Exploration, Information Visualisation, Data Aggregation, Preprocessing, and Warehousing Data.

T. S. Eliot's quote reflects the idea that wisdom can be lost in the abundance of knowledge and information, highlighting the challenge of deriving true wisdom from data analysis amidst the sea of information.

An example of an application that involves all the activities mentioned in the data science process is an Online Store, which requires Decision Making, Machine Learning, Data Exploration, Visualisation, Aggregation, Preprocessing, and Warehousing of Data from various sources.

To answer the query using brute force, all the data for the given URL within the specified time period is processed. Duplicate users are eliminated, and the remaining records are summed up to determine the number of different users who visited the web page within the given time period.

The goal of a (Big) Data System is to answer questions based on the information obtained from the data accumulated over a period of time. Examples of questions that can be answered include: determining the show a user is most likely to want to watch based on their watching habits and the history of shows watched, identifying the number of friends a person has on Facebook, and retrieving a user's transaction history on Amazon within a specific time frame.

Lambda Architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods. It involves three layers: the batch layer for managing historical data, the speed layer for real-time data processing, and the serving layer for querying and accessing the processed data.

The running example in the text is a server that tracks the number of times each tracked web page has been visited within a specific time period by different users.

The material mentioned in the text is mostly taken from the book 'Big Data: Principles and Best Practices for Scalable Real-Time Data Analysis' by N. Marz and J. Warren, specifically from Chapter 1 of the book.