Decision Trees Overview and Case Study

Decision Trees Overview

• Decision trees are a popular supervised classification technique
• They are a simple way to make decisions
• Decisions represented by hierarchical structures
• Questions asked in a hierarchical order
• Best trees are short and answer questions with the most relevant information
• Decision trees can be built from small datasets and applied to larger populations
• Suitable for simple binary decisions

Learning Objectives

• Understanding decision trees
• Identifying key parts of decision tree building
• Using a basic dataset to create a decision tree
• Recognizing common decision tree algorithms

Case Study: Predicting Heart Attacks

• This case study used data mining to predict heart attacks
• Data focused on patients with previous heart attacks
• Goal to predict which patients were at risk for a second heart attack in the next 30 days
• Predicting risk to determine treatment plan
• The CART algorithm was employed
• More than 100 variables (factors) were included
• Data transformation and cleaning occurred before analysis
• Factors like age, blood pressure, and sinus problems were considered by the decision tree
• The decision tree was 86.5% accurate in predicting heart attack risk

Results

• Low blood pressure (<90) strongly suggests a high risk of subsequent heart attacks (70%)
• If blood pressure was normal, age was considered as the next factor
• Patients under 62 years had almost guaranteed survival (98%)
• Older patients' sinus condition was considered if their age was over 62
• When sinus was okay, survival chance was 89% ; otherwise, 50% chance
• Overall accuracy of the decision tree was 86.5%

Disease Diagnosis

• Decision tree logic applies to many disease diagnoses
• Medical diagnosis involves similar question-and-answer processes
• A physician's thought process mirrors decision tree structure when evaluating patient symptoms and ordering tests
• Decision trees and decision rules aid in diagnosing diseases
• Every question leads to potential answers creating separate branches for further questions
• The process continues until a conclusion is reached (leaf node)
• Medical professionals and experts in general fields employ similar methods to solve problems

Machine Learning and Decision Trees

• Machine learning uses past data to train and extract knowledge and rules
• Decision trees use algorithms to form knowledge from data
• Accuracy is measured by the frequency of correct predictions (predictive accuracy)

Decision Trees Points to Consider

• More training data generally results in better accuracy
• More input variables can lead to greater accuracy
• A good decision tree is efficient (gets the right answer with the least amount of steps)
• Needs a small number of questions

Decision Tree Construction

• Determine the root node of a decision tree
• Splitting the tree
• Finding the next nodes in the branches

Determine the Root Node of a Tree

• Identify the most important question to ask in order to solve a problem
• Evaluate the importance of the questions asked
• Determine the root node for the decision tree
• How many choices/variables are there?
• How many variable choices are there for the problem?
• Evaluate the best choices (using criteria like least error criterion)
• Identify the question that best clarifies the situation
• Find the question that leads to the shortest decision tree

Error and Rules

• Error measures the decision tree's performance and incorrect predictions
• Balance complexity of the tree to avoid overly complex (overfitting) or overly simplified (underfitting) models
• Rules in a tree show the logical connections between choices/variables, conditions and predictions. These are the rules leading to branches and nodes

Select Splitting Criterion

• A metric is chosen to evaluate each variable's importance (information gain, Gini impurity, Chi-square)

Determine Root Node Examples (Outlook, Temp, Humidity, Windy)

• Outlook evaluation using error probabilities
• Temperature evaluation using error probabilities
• Humidity evaluation using error probabilities
• Windy evaluation using error probabilities

Determine the Root Node of a Tree

• Choose the variable with the least errors as the root node of the tree
• In cases of a tie, choose the variable that has the purest sub-trees
• Outlook variable usually gives the best possible root node. Variable chosen is often the best variable when making a decision for the problem
• The first question asked becomes "What is the value of Outlook?" to begin the analysis process

Splitting the Tree

• The data is divided into subsets based on the root node values (e.g., sunny, overcast, rainy)
• Those subsets are further broken down to form sub-trees

Determining Next Nodes (Sunny, Rainy)

• Apply the same procedure used for the root node in each branch (subtree)
• Select the next best question (e.g., humidity or wind)

Decision Tree Algorithm

• Employs divide and conquer method
• Steps for building a decision tree:
  • Create a root node and assign data
  • Identify best splitting variable
  • Add branches based on splitting variable values (mutually exclusive)
  • Repeat for every leaf node until specified stopping criteria is met

Decision Tree Algorithms: Key Elements

• Splitting criteria (which variables to use, how to make bins, for continuous variables)
• Stopping criteria (when to stop adding branches)

Key Elements of Pruning

• Trimming the tree to be more balanced and usable (in complex scenarios)
• Pruning occurs after construction of the full tree
• Tree pruning can help solve imbalances caused by noise, outliers, or overfitting

Popular Decision Tree Algorithms

• C 4.5 (Iterative Dichotomiser ID3)
• CART (Classification and Regression Trees)
• CHAID (Chi-square Automatic Interaction Detector)

Summary of Learning

• Decision trees are a popular tool for data mining
• They are accurate and easy to use, especially when dealing with limited datasets
• Trees are well-suited for communication and explanation of results

Data Analysis Observations

• Zero errors and 100% accuracy are mostly unrealistic except in specific simple datasets
• In real-world data analysis, perfect accuracy is unusual
• It is important that the tree is well-balanced
• Decision trees help with clear business and practical problems rather than complex ones that defy easy classification