Machine Learning Challenges in AIMLB

Here are the study notes for the provided text:

Machine Learning Challenges

• Prediction error consists of two components: bias and variance • Overfitting occurs when a model captures the noise and outliers in the data, resulting in high variance and low bias • Underfitting occurs when a model is unable to capture the underlying pattern of the data, resulting in low variance and high bias • The bias-variance tradeoff is a critical challenge in machine learning • Model selection and tuning are crucial to achieve the right balance between bias and variance

Credit Risk Assessment

• Credit risk assessment is a critical application of machine learning in finance • The goal is to determine whether a loan applicant is likely to default on a loan • Factors considered in credit risk assessment include: + Credit history + Income + Loan terms + Personal information • Machine learning algorithms can be used to develop credit risk models that predict the likelihood of default

Classification

• Classification is a type of supervised learning where the target variable is categorical • The goal of classification is to predict the class label of a new instance based on the attributes of the instance • Examples of classification tasks include: + Spam vs. non-spam emails + Tumor cells as benign or malignant + Credit card transactions as legitimate or fraudulent + Sentiment analysis • Decision trees are a popular classification algorithm

Decision Trees

• Decision trees represent rules that can be understood by humans • Decision trees are useful for knowledge representation and can be used in databases • The goal of decision tree induction is to learn a model that maps each attribute set to one of the predefined class labels • The process of decision tree induction involves: + Selecting the most informative attribute + Partitioning the data according to the attribute's values + Recursively constructing subtrees for the subsets of the data • Conditions for stopping partitioning include: + All samples for a given node belong to the same class + There are no remaining attributes for further partitioning + There are no samples left

Entropy and Information Gain

• Entropy measures the degree of randomness in data • Information gain is the expected reduction in entropy due to splitting on values of an attribute • The best attribute is the one with the highest information gain • Information gain is used to select the most informative attribute in decision tree induction

Best Attribute Selection

• Best attribute selection is critical in decision tree induction • Information gain is used to select the best attribute • Gain ratio is used to overcome the limitation of information gain, which is biased towards multivalued attributes • Gini impurity is an alternative to entropy for selecting attributes

Gini Impurity

• Gini impurity measures how often a randomly chosen example would be incorrectly labeled • Gini impurity is used to select the best attribute in decision tree induction • The best attribute is the one with the highest impurity decrease### Decision Trees and Metrics

• Pruning: a technique to reduce the size of a decision tree by removing branches that provide little predictive power, reducing overfitting.
• Types of Pruning:
  • Pre-pruning: stops the tree building algorithm before it fully classifies the data.
  • Post-pruning: builds the complete tree, then replaces some non-leaf nodes with leaf nodes if it improves validation error.

Computing Information-Gain for Continuous-Valued Attributes

• Sorting: sort the values of the continuous attribute in increasing order.
• Midpoint: consider the midpoint between each pair of adjacent values as a possible split point.
• Split: select the point with the minimum expected information requirement for the attribute as the split-point.

Handling Missing Values

• Handling missing values at training time:
  • Set them to the most common value.
  • Set them to the most probable value given the label.
  • Add a new instance for each possible value.
• Handling missing values at inference time: explore all possibilities and take the final prediction based on a weighted vote of the corresponding leaf nodes.

Decision Boundaries

• Decision trees produce non-linear decision boundaries.

Model Evaluation and Selection

• Evaluation metrics:
  • Accuracy: measures how correctly classified the test set tuples are.
  • Other metrics: consider precision, recall, F-measure, etc.
• Methods for estimating a classifier's accuracy:
  • Holdout method.
  • Random subsampling.
  • Cross-validation.
  • Bootstrap.

Classifier Evaluation Metrics

• Confusion Matrix:
  • A table used to evaluate the performance of a classifier.
  • Contains true positives, false negatives, false positives, and true negatives.
• Accuracy: percentage of test set tuples that are correctly classified.
• Error Rate: 1 - accuracy, or the percentage of misclassified tuples.
• Sensitivity: true positive recognition rate.
• Specificity: true negative recognition rate.
• Precision: exactness - what percentage of tuples that the classifier labeled as positive are actually positive.
• Recall: completeness - what percentage of positive tuples did the classifier label as positive.
• F-measure: harmonic mean of precision and recall.