ml ..pdf

Feature Engineering IFT6758 - Data Science Sources: https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114 https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba https://www.slideshare.net/0xdata/feature-engineering-83511751 Announcements Midterm: October 31 at 16:30 – 18:30 (those who cannot make it at 16:30, send us an email asap. We will arrange a room for them to do the same exam at 14:30 – 16:30) Project evaluation: The teams with the results same as baseline on the scoreboard for Evaluation1 got the full grade for the first project assignment, i.e., User05, User06, User08, User09, User14, User17, User22, User25, User26, User29, User30. The other teams will be graded this week based on their scores for Evaluation 2. We have two invited talks next week on Tuesday (Geospatial and Time series data analysis) and Thursday (Privacy and Transparency in Machine Learning). Project Presentation will take place on November 5 !2 Feature Engineering Data scientists usually spend the most time on feature engineering! !3 Feature engineering cycle Not Feature Engineering data collection Creating the target variable Removing duplicates Fixing mislabeled classes. !4 Feature engineering cycle !5 Feature engineering cycle !6 Feature engineering is hard Powerful feature transformations (like target encoding) can introduce leakage when applied wrong Usually requires domain knowledge about how features interact with each other Time-consuming, need to run thousand of experiments Why Feature Engineering matters Extract more new gold features, remove irrelevant or noisy features Simpler models with better results !7 Key Elements of Feature Engineering Target Transformation Feature Extraction Feature Encoding !8 Target Transformation Predictor/Response Variable Transformation Use it when variable shows a skewed distribution make the residuals more close to “normal distribution” (bell curve) Can improve model fit e.g., log(x), log(x+1), sqrt(x), sqrt(x+1), etc. !9 Target Transformation !10 Key Elements of Feature Engineering Target Transformation Feature Extraction Feature Encoding !11 Imputation Common problem in preparing the data: Missing Values Why we have missing values? Human errors Interruptions in the data flow Privacy concerns What to do? !12 Imputation Common problem in preparing the data: Missing Values Why we have missing values? Human errors Interruptions in the data flow Privacy concerns What to do? Simple solution: drop the row/column Preferable option: Imputation !13 Imputation Numerical Imputation Assing zero Assing NA Default value or medians of the columns (Note that averages are sensitive to outlier values) Categorical Imputation maximum occurred value There is not a dominant value, imputing a category like “Other” !14 Outliers Outliers may introduce to the population during data collections mistake ? variance ? !15 Outlier detection Outlier: A data object that deviates significantly from the normal objects as if it were generated by a diﬀerent mechanism Ex.: Unusual credit card purchase, sports: Michael Jordon, Wayne Gretzky, … Outliers are diﬀerent from the noise data Noise is random error or variance in a measured variable Noise should be removed before outlier detection Outliers are interesting: It violates the mechanism that generates the normal data Outlier detection vs. novelty detection: early stage, outlier; but later merged into the model Applications: Credit card fraud detection Telecom fraud detection Customer segmentation Medical analysis !16 Types of Outliers Three kinds: global, contextual and collective outliers Global outlier (or point anomaly) Object is O if it significantly deviates from the rest of the data set g Ex. Intrusion detection in computer networks Issue: Find an appropriate measurement of deviation Contextual outlier (or conditional outlier) Object is O if it deviates significantly based on a selected context c Ex. 80o F in Urbana: outlier? (depending on summer or winter?) Attributes of data objects should be divided into two groups Contextual attributes: defines the context, e.g., time & location Behavioral attributes: characteristics of the object, used in outlier evaluation, e.g., temperature Can be viewed as a generalization of local outliers—whose density significantly deviates from its local area Issue: How to define or formulate meaningful context? !17 Types of Outliers Collective Outliers A subset of data objects collectively deviate significantly from the whole data set, even if the individual data objects may not be outliers Applications: E.g., intrusion detection: When a number of computers keep sending denial-of-service packages to each other Detection of collective outliers Consider not only behavior of individual objects, but also that of groups of objects Need to have the background knowledge on the relationship among data objects, such as a distance or similarity measure on objects. A data set may have multiple types of outlier One object may belong to more than one type of outlier !18 Finding Outliers To ease the discovery of outliers, we have plenty of methods in statistics: Discover outliers with visualization tools or statistical methodologies Box plot Scatter plot Z-score IQR score We will cover more advanced techniques for anomaly detection (after the mid-term) !19 Finding Outliers Box plot In descriptive statistics, a box plot is a method for graphically depicting groups of numerical data through their quartiles. Box plots may also have lines extending vertically from the boxes (whiskers) indicating variability outside the upper and lower quartiles, hence the terms box-and-whisker plot and box-and-whisker diagram. Outliers may be plotted as individual points. Wikipedia Definition This definition suggests, that if there is an outlier it will plotted as point in boxplot but other population will be grouped together and display as boxes. !20 Finding Outliers Box plot !21 Finding Outliers Above plot shows three points between 10 to 12, these are outliers as there are not included in the box of other observation i.e no where near the quartiles. Here we analysed Uni-variate outlier i.e. we used DIS column only to check the outlier. We can do multivariate outlier analysis too. Can we do the multivariate analysis with Box plot? Well it depends, if you have a categorical values then you can use that with any continuous variable and do multivariate outlier analysis. !22 Finding Outliers Scatter plot A scatter plot , is a type of plot or mathematical diagram using Cartesian coordinates to display values for typically two variables for a set of data. The data are displayed as a collection of points, each having the value of one variable determining the position on the horizontal axis and the value of the other variable determining the position on the vertical axis. Wikipedia Definition This definition suggests, the scatter plot is the collection of points that shows values for two variables. !23 Finding Outliers We can try and draw scatter plot for two variables from our housing dataset. !24 Finding Outliers Looking at the plot above, we can most of data points are lying bottom left side but there are points which are far from the population like top right corner. !25 Finding Outliers Standard deviation In statistics, the standard deviation (SD, also represented by the lower case Greek letter sigma σ for the population standard deviation or the Latin letter s for the sample standard deviation) is a measure of the amount of variation or dispersion of a set of values. Wikipedia Definition !26 Finding Outliers If a value has a distance to the average higher than x * standard deviation,it can be assumed as an outlier. Then what x should be? Usually, a value between 2 and 4 seems practical. !27 Finding Outliers Z-score The Z-score is the signed number of standard deviations by which the value of an observation or data point is above the mean value of what is being observed or measured. Wikipedia Definition The intuition behind Z-score is to describe any data point by finding their relationship with the Standard Deviation and Mean of the group of data points. Z-score is finding the distribution of data where mean is 0 and standard deviation is 1 i.e. normal distribution. !28 Finding Outliers Z-score: While calculating the Z-score we re-scale and center the data and look for data points which are too far from zero. These data points which are way too far from zero will be treated as the outliers. In most of the cases a threshold of 3 or -3 is used i.e if the Z- score value is greater than or less than 3 or -3 respectively, that data point will be identified as outliers. We will use Z-score function defined in scipy library to detect the outliers. !29 Finding Outliers Looking the code and the output above, it is diﬃcult to say which data point is an outlier. Let’s try and define a threshold to identify an outlier. List of arrow numbers List of Column numbers !30 Finding Outliers IQR score The interquartile range (IQR), also called the midspread or middle 50%, or technically H-spread, is a measure of statistical dispersion, being equal to the diﬀerence between 75th and 25th percentiles, or between upper and lower quartiles, IQR = Q3 − Q1. Wikipedia Definition It is a measure of the dispersion similar to standard deviation or variance, but is much more robust against outliers. !31 Finding Outliers Let’s find out we can box plot uses IQR and how we can use it to find the list of outliers as we did using Z-score calculation. First we will calculate IQR !32 Finding Outliers The data point where we have False that means these values are valid whereas True indicates presence of an outlier. !33 Finding Outliers Percentiles Another mathematical method to detect outliers is to use percentiles. You can assume a certain percent of the value from the top or the bottom as an outlier. A common mistake is using the percentiles according to the range of the data, e.g., if your data ranges from 0 to 100, your top 5% is not the values between 96 and 100. Top 5% means the values that are out of the 95th percentile of data. !34 Handling Outliers An Outlier Dilemma: Drop or Cap Correcting Removing Z-score: IQR score: !35 Binning Binning can be applied on both categorical and numerical data: Example !36 Binning The main motivation of binning is to make the model more robust and prevent overfitting, however, it has a cost to the performance. The trade-oﬀ between performance and overfitting is the key point of the binning process. Numerical binning: binning might be redundant due to its eﬀect on model performance. Categorical binning: the labels with low frequencies probably aﬀect the robustness of statistical models negatively. Thus, assigning a general category to these less frequent values helps to keep the robustness of the model. !37 Log Transformation Logarithm transformation (or log transform): It helps to handle skewed data and after transformation, the distribution becomes more approximate to normal. In most of the cases the magnitude order of the data changes within the range of the data. It also decreases the eﬀect of the outliers, due to the normalization of magnitude diﬀerences and the model become more robust. !38 Log Transformation The data you apply log transform must have only positive values, otherwise you receive an error. Also, you can add 1 to your data before transform it. Thus, you ensure the output of the transformation to be positive. !39 Grouping Tidy data where each row represents an instance and each column represent a feature. Datasets such as transactions rarely fit the definition of tidy data -> we use grouping. The key point of group by operations is to decide the aggregation functions of the features. !40 Grouping Aggregating categorical columns: Highest frequency: the max operation for categorical columns Make a Pivot table: This would be a good option if you aim to go beyond binary flag columns and merge multiple features into aggregated features, which are more informative. Apply one-hot encoding !41 Grouping Numerical columns are mostly grouped using: Sum Mean !42 Splitting Most of the time the dataset contains string columns that violates tidy data principles. Split function is a good option, however, there is no one way of splitting features !43 Scaling In most cases, the numerical features of the dataset do not have a certain range and they diﬀer from each other. In real life, it is nonsense to expect age and income columns to have the same range. But from the machine learning point of view, how these two columns can be compared? It is important for algorithms that work based on distance: such as k- NN or k-Means Basically, there are two common ways of scaling: Normalization, and Standardization !44 Normalization Normalization (or min-max normalization) scale all values in a fixed range between 0 and 1. This transformation does not change the distribution of the feature. But due to the decreased standard deviations, the eﬀects of the outliers increases. So before normalization, it is recommended to handle the outliers. !45 Normalization Example: !46 Standardization Standardization (or z-score normalization) scales the values while taking into account standard deviation. In the following formula of standardization, the mean is shown as μ and the standard deviation is shown as σ. If the standard deviation of features is diﬀerent, their range also would diﬀer from each other. This reduces the eﬀect of the outliers in the features. !47 Standardization Example: !48 Key Elements of Feature Engineering Target Transformation Feature Extraction Feature Encoding !49 Feature Encoding Turn categorical features into numeric features to provide more fine-grained information Help explicitly capture non-linear relationships and interactions between the values of features Most of machine learning tools only accept numbers as their input e.g., xgboost, gbm, glmnet, libsvm, liblinear, etc. !50 Feature Encoding Labeled Encoding Interpret the categories as ordered integers (mostly wrong) Python scikit-learn: LabelEncoder Ok for tree-based methods One Hot Encoding Transform categories into individual binary (0 or 1) features Python scikit-learn: DictVectorizer, OneHotEncoder Ok for K-means, Linear, NNs, etc. !51 One Hot Encoding One-hot-encoding: is one of the most common encoding methods in machine learning. This method spreads the values in a column to multiple flag columns and assigns 0 or 1 to them. These binary values express the relationship between grouped and encoded column. !52 Frequency encoding Encoding of categorical levels of feature to values between 0 and 1 based on their relative frequency !53 Target mean encoding Instead of dummy encoding of categorical variables and increasing the number of features we can encode each level as the mean of the response. !54 Target mean encoding It is better to calculate weighted average of the overall mean of the training set and the mean of the level: The weights are based on the frequency of the levels i.e. if a category only appears a few times in the dataset then its encoded value will be close to the overall mean instead of the mean of that level. !55 Target mean encoding Smoothing !56 Target mean encoding Smoothing !57 Target mean encoding leave-one-out schema To avoid overfitting we could use leave-one-out schema !58 Target mean encoding leave-one-out schema To avoid overfitting we could use leave-one-out schema !59 Target mean encoding leave-one-out schema To avoid overfitting we could use leave-one-out schema !60 Target mean encoding leave-one-out schema To avoid overfitting we could use leave-one-out schema !61 Target mean encoding leave-one-out schema To avoid overfitting we could use leave-one-out schema !62 Weight of Evidence and Information Value Weight of evidence: To avoid division by zero: Information Value: !63 Weight of Evidence and Information Value 0.221 !64 Weight of Evidence and Information Value !65 More of Feature Engineerings … Feature Extraction: Numerical data Dimensionality reduction techniques – SVD and PCA (Week 3) Clustering and using cluster IDs or/and distances to cluster centers as new features (Week 3) Feature selection (Week 6) Feature Extraction: Textual data (Week 10) e.g., Bag-of-Words: extract tokens from text and use their occurrences (or TF/IDF weights) as features Feature Extraction: Time series and GEO location (Week 7) Feature Extraction: Image data (Week 11) Feature Extraction: Relational data (Week 12) Anomaly detection (advanced): (Week 13) !66

Document Details

Related

Full Transcript

Upgrade to continue