Data Science Machine Learning

Big Data Analytics

• Big data analytics is a field focused on analyzing large datasets.
• Machine learning and data mining are techniques used for big data analytics.

Machine Learning vs. Data Mining

• There is no single, universally agreed-upon definition of machine learning versus data mining.
• Machine learning focuses on creating algorithms that learn from historical data to make predictions.
• Data mining aims to discover properties and useful information within datasets.
• Machine learning can be used as a method in data mining.

Machine Learning Example Applications

• Self-driving cars
• Spam detection
• Fraud detection
• Voice recognition
• Face recognition
• Anomaly detection
• Sales forecasting
• Robotics

What is Machine Learning?

• Machine learning is a data science technique where computers learn from existing data to anticipate future behaviors, outcomes, and trends.
• Machine learning involves learning from historical data, recognizing patterns and trends, and making predictions.

How Machine Learning Works

• Data is divided into training, validation, and test sets.
• The training set is used to build the model.
• The validation set is used to assess the model's performance.
• The test set is used to evaluate the final model's performance.
• The model is tuned using more data, different features, or adjusted parameters.
• Trained models are used to predict new data.

An Example of a Machine Learning Task (Car Rental)

• The task is to forecast car rental demand.
• Steps include: getting data, preparing data, training the model, evaluating the model, and predicting future demand.

Difference Between Traditional and Machine Learning

• Rule-based approach
  • Explicitly programmed to solve problems
  • Decision rules are clearly defined by humans
• Machine learning approach
  • Trained from examples
  • Decision rules are complex and fuzzy
  • Rules are learned by machines from data

Summary

• Machine learning uses historical data for predictions.
• Similar to data mining, but focuses on applying prior knowledge to make decisions.
• Machines approximate complex functions and learn rules from data.

The Data Science Process

• Ask an interesting question: Understand the scientific goal, what to predict.
• Get the data: How was data sampled? Are there privacy issues?
• Explore the data: Visualize, look for anomalies, find patterns.
• Model the data: Build and fit the model. Validate the model.
• Communicate and visualize the results: What was learned? Were the results useful?

How to Start a Data Science Project

• Identify business goals
• Assess the current situation
• Identify data mining goals
• Create a project plan

Sharp vs. Vague Questions

• Sharp questions can be answered with data (e.g., stock price).
• Vague questions can't (e.g., how to increase profits).

The 5 Questions Data Science Can Answer

• Is this A or B? (Classification)
• Is this weird? (Anomaly Detection)
• How much or how many? (Regression)
• How is this organized? (Clustering)
• What should I do now? (Reinforcement Learning)

Q1: Is This A or B?

• Use Classification algorithms
• Example: Will this tire fail in the next 1000 miles? (Yes/No)
• Another Example: Which brings in more customers? ($5 coupon or 25% discount?)

Q2: Is this Weird?

• Use Anomaly Detection algorithms
• Example: Your credit card company identifying unusual transactions.

Q3: How Much? or How Many?

• Use Regression algorithms
• Example: Predicting the temperature next Tuesday.
• Example: Predicting fourth quarter sales.

Q4: How is This Organized?

• Use Clustering algorithms
• Examples: Clustering viewers with similar movie tastes.
• Examples: Clustering printer models that fail the same way.

Q5: What Should I Do Now?

• Use Reinforcement Learning algorithms
• Examples: Self-driving car deciding to brake or accelerate at a yellow light.
• Examples: Robot vacuum deciding whether to keep cleaning or return to charging station.

So, What Do You Want to Find Out?

• Regression: Forecast future outcomes by estimating the relationship between variables.
• Anomaly Detection: Identify and predict unusual data points.
• Clustering: Separate similar data points into groups.
• Classification: Assign new data points to categories or classes.

When to Use Machine Learning

• To automate tasks.
• To deal with high-volume tasks involving complex rules and unstructured data.
• When sufficient examples are available to train a model.
• If the problem has a discernible pattern that can be recognized by the model.
• When you can create meaningful representations of the data.
• Define what success means for the outcome

Summary (Machine Learning)

• Use machine learning when there's a complex task involving large amounts of data and no existing formula, for cases such as speech recognition.

Machine Learning Types

• Supervised learning (classification, regression)
• Unsupervised learning (clustering)
• Semi-supervised learning
• Reinforcement learning

Growth of Machine Learning

• Increasing use for natural language processing, computer vision, medical analysis, and robotics.
• Improved algorithms and increased computing power.

Supervised Learning

• Goal: Map input variables with output variables.
• Learning method using labeled data.
• Example categories include Risk Assessment, Fraud Detection, Spam filtering, etc.

Supervised Learning Applications

• Classification
• Regression
• Forecasting

Unsupervised Learning

• Learning with unlabeled data.
• Goal: Classifies data points based on similarities, differences, and patterns.
• Clustering is a common unsupervised learning technique

Semi-Supervised Learning

• Combines labeled and unlabeled data for learning.

Reinforcement Learning

• Agent learns from experiences (without labeled data), with reward mechanisms.
• Common examples include game theory, operation research, and multi-agent systems

Hadoop-Related Apache Projects

• Ambari
• Avro
• Cassandra
• Chukwa
• HBase
• Hive
• Mahout
• Pig
• Spark
• Tez
• ZooKeeper

Key Components of Mahout

• Collaborative filtering
• Classification
• Clustering

Mahout Reference Book

• Chapter content in the Mahout reference book by Owen, Anil, Dunning, and Friedman.

Mahout Overview

• Mahout's move away from MapReduce to a DSL for linear algebraic operations.

Clustering

• Given a dataset, find clusters of similar data points.
• Similarity (distance) measures (like Euclidean distance) are used to group data points (in 2D,3D, or higher dimensional space)
• Clustering needs an algorithm, a notion of similarity and a stop condition to identify clusters.

k-means Clustering

• Algorithm for partitioning datasets into clusters.
• Iterative process of assigning data points to the nearest centroid.
• Steps involved in k-means clustering: selecting the number of clusters, randomly selecting initial centroids, measuring distance, and assigning each point to the nearest centroid.
• Steps involved (continuation): recalculating centroids, repeating steps 2 and 3 until there's no change in centroids, or a maximum number of iterations is reached.
• evaluating the result by comparing initial and final centroids locations.

Questions

• Determining a good value for k.
• Handling data in various dimensions

The Elbow Method for Determining k

• Plot of F vs k, looking for an elbow in the graph identifying a good value for k.

Question 2: What if the Data is 2-Dimensional, 3 Dimensional...?

• Methods for calculating distances in multi-dimensional space are needed in addition to calculating distances in 2D or 3D.

Hadoop k-means Clustering Jobs

• In Mahout, the MapReduce version of the k-means algorithm runs using the KMeansDriver class.

K-means Clustering Running as MapReduce Job

• Parallelization of tasks to speed up clustering on large datasets using MapReduce.

HelloWorld Clustering Scenario

HelloWorld Clustering Scenario (Part II)

• Detailed code for setting up k-means clustering using Hadoop in Mahout.

HelloWorld Clustering Scenario (Part III)

• Executing k-means clustering on Hadoop using the Java KMeansDriver framework.

HelloWorld Clustering Scenario Result

• Output generated from running the KMeansDriver using the defined method.

Testing Distance Measures

• Different ways to measure the distance between data points.

Manhattan Distances

• Weighted distance is part of Mahout.

Results Comparison

• Comparing different methods for measuring distance and the number of iterations needed.

Classification

• Definition and an example using a classification table.

How Does a Classification System Work?

• Diagram outlining the process used to classify data by training the model.

Process 1: Model Construction

Process 2: Using the Model in Prediction

When to Use Mahout for Classification

• Guidelines for choosing Mahout for classification based on the size of the data.

Advantage of Using Mahout for Classification

• Diagram showing the improved performance of Mahout with large data sets.

Key Terminology for Classification

• Definitions for different classification terms that are needed for learning about machine learning.

Workflow in a Typical Classification Project

• Typical stages of a classification project.

Choosing Algorithms via Mahout

• Algorithm choice guidelines based on dataset size.

Decision Tree

• Basic Classification algorithm, using a divide-and-conquer method of splitting on training data by attributes until the final outcome is assigned to a tree leaf.

Regression

• Predicting a continuous variable based on other variables.

Regression - Example

• Worked examples involving linear plots, polynomial fits and calculations and visualizations.