DM868-DM870-DS804-2024-1.pdf

DM868 DM870 DS804 Arthur Zimek Introduction Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Data Mining and Machine Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References...

DM868 DM870 DS804 Arthur Zimek Introduction Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Data Mining and Machine Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References Arthur Zimek University of Southern Denmark DM868/DM870/DS804, spring 2024 1  Outline DM868 DM870 DS804 Introduction Arthur Zimek Frequent Pattern Mining Introduction Freq. Pattern Mining Feature Spaces Feature Spaces Clustering – Basics Clustering – Basics and k-means Classification – Basics Classification – Basics and a Basic Classifier Bayesian Learning Learning with Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions Entropy, Purity, and Distributions and Learning with Distributions (Non-)Linear Sep. Ensemble Learning Entropy, Purity, and Separation: Linear vs. Non-Linear Separation References Ensemble Learning 2  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 3  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 4  Learning from Data DM868 DM870 DS804 Arthur Zimek Tabulæ Rudolphinæ Introduction (observations by Tycho Brahe and Johannes Kepler) Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Kepler’s laws Summary Freq. Pattern Mining of planetary motion Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning f1 (sun) a1 f2 References =⇒ A2 A1 planet 2 planet 1 a2 f3 5  Data Science DM868 DM870 DS804 Arthur Zimek https://hbr.org/2012/10/ Introduction Data Science data-scientist-the-sexiest-job-of-the-21st-century Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 6  Data Science DM868 DM870 DS804 Arthur Zimek https://hbr.org/2012/10/ Introduction Data Science data-scientist-the-sexiest-job-of-the-21st-century Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References ▶ “data science” is a vaguely defined concept 6  Data Science DM868 DM870 DS804 Arthur Zimek https://hbr.org/2012/10/ Introduction Data Science data-scientist-the-sexiest-job-of-the-21st-century Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References ▶ “data science” is a vaguely defined concept ▶ meaning A: doing science based on (big) data 6  Data Science DM868 DM870 DS804 Arthur Zimek https://hbr.org/2012/10/ Introduction Data Science data-scientist-the-sexiest-job-of-the-21st-century Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References ▶ “data science” is a vaguely defined concept ▶ meaning A: doing science based on (big) data ▶ meaning B: the science of working with data 6  Data Science in Venn Diagrams DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 7  Data Science in Venn Diagrams DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 7  Data Science in Venn Diagrams DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 7  Data Science in Venn Diagrams DM868 DM870 DS804 Arthur Zimek Introduction Recommended Reading: Data Science Knowledge Entertaining post on attempts to describe data science by Discovery from Data Data Mining Methods Venn diagrams: Preprocessing Transformation Evaluation ▶ http://www.kdnuggets.com/2016/10/ Material Summary battle-data-science-venn-diagrams.html Freq. Pattern Mining Feature Spaces (The diagrams shown on the previous slide were taken from Clustering – Basics Classification – Basics there.) Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 8  Data Mining and Related Areas DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Mathematics Databases Data Mining Methods Preprocessing Statistics Data Management Transformation Evaluation Material Summary Freq. Pattern Mining Machine Learning/ Feature Spaces Clustering – Basics Data Mining Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Domain Programming Ensemble Learning References Knowledge Implementation Data Mining tries to combine the strengths of many domains. 9  Data Mining and Related Areas DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Mathematics Databases Data Mining Methods Preprocessing Statistics Data Management Transformation Evaluation Precision Scalability Material Summary Freq. Pattern Mining Machine Learning/ Feature Spaces Clustering – Basics Data Mining Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Domain Programming Ensemble Learning References Knowledge Implementation Flexibility Performance Data Mining tries to combine the strengths of many domains. 9  Understand Methods for Analyzing Data DM868 DM870 DS804 Arthur Zimek ▶ data science: learning from Introduction Data Science data, finding patterns in data, Knowledge Discovery from Data Data Mining Methods understanding databases Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References Illustration by David Parkins (detail). Source: Silberzahn and Uhlmann. 10  Understand Methods for Analyzing Data DM868 DM870 DS804 Arthur Zimek ▶ data science: learning from Introduction Data Science data, finding patterns in data, Knowledge Discovery from Data Data Mining Methods understanding databases Preprocessing Transformation ▶ data mining/machine learning: Evaluation Material computational methods for Summary Freq. Pattern Mining learning from data Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References Illustration by David Parkins (detail). Source: Silberzahn and Uhlmann. 10  Understand Methods for Analyzing Data DM868 DM870 DS804 Arthur Zimek Introduction ▶ data science: learning from Data Science Knowledge data, finding patterns in data, Discovery from Data Data Mining Methods understanding databases Preprocessing Transformation ▶ data mining/machine learning: Evaluation Material computational methods for Summary Freq. Pattern Mining learning from data Feature Spaces Clustering – Basics Classification – Basics ▶ different methods deliver Bayesian Learning Learning with different pictures of the data Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References Illustration by David Parkins. Source: Silberzahn and Uhlmann. 10  Understand Methods for Analyzing Data DM868 DM870 DS804 Arthur Zimek Introduction ▶ data science: learning from Data Science Knowledge data, finding patterns in data, Discovery from Data Data Mining Methods understanding databases Preprocessing Transformation ▶ data mining/machine learning: Evaluation Material computational methods for Summary Freq. Pattern Mining learning from data Feature Spaces Clustering – Basics Classification – Basics ▶ different methods deliver Bayesian Learning Learning with different pictures of the data Distributions Entropy, Purity, and ▶ this course should help: (Non-)Linear Sep. Ensemble Learning ▶ to learn about data mining/ References machine learning methods ▶ to understand their characteristics Illustration by David Parkins. ▶ to apply them correctly Source: Silberzahn and Uhlmann. ▶ to derive meaningful results 10  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 11  Motivation: Knowledge Discovery from Data DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References ▶ huge amount of data collected in various application domains ▶ manual analysis? 12  Definition: Knowledge Discovery from Data DM868 DM870 DS804 Arthur Zimek “KDD is the nontrivial process of identifying valid, novel, Introduction Data Science potentially useful, and ultimately understandable patterns in Knowledge Discovery from Data data.” [Fayyad et al., 1996] Data Mining Methods Preprocessing ▶ data: set of facts (e.g., entries in a database) Transformation ▶ pattern: expression in some language to describe a data subset (e.g., Evaluation Material mathematical model) Summary Freq. Pattern Mining ▶ process: can involve several steps or iterations Feature Spaces ▶ nontrivial: more complex than search, inference, simple aggregations Clustering – Basics Classification – Basics ▶ valid: applicable to new data with a certain degree of reliability Bayesian Learning Learning with ▶ novel: for the system, better: for the user Distributions Entropy, Purity, and ▶ potentially useful: beneficial for user of application (Non-)Linear Sep. Ensemble Learning ▶ ultimately understandable: if not immediately then given some post References processing understandability ⇔ simplicity? (validity, novelty, usefulness, simplicity) ⇔ “interestingness” 13 14 Material Summary References Evaluation Introduction Knowledge Distributions Learning with DS804 Data Science Preprocessing Feature Spaces Transformation Arthur Zimek (Non-)Linear Sep. DM868 DM870 Bayesian Learning Entropy, Purity, and Clustering – Basics Ensemble Learning Freq. Pattern Mining Discovery from Data Classification – Basics Data Mining Methods ▶ focusing ▶ Fo getkussieren: the data KDD ▶ Beschaffun B organize ng der data Datten (file/database) f database ▶ Verwaltung V select g (File/DB relevant dataB) i Selektion rrelevanterr Daten Prozessmodell focusing Databases I: Einführung Knowledge Discovery in process Vo orverarb beitung: ▶ preprocessing In ntegration n von Dateen aus ▶ integrate heterogeneous unterschiedlichen Q Quellen data Vervollstän V ndigung nach model ▶ check for completeness i Konsistenzzprüfung ▶ check for consistency preprocessing Traansform mation D Diskretisierung num meri‐ ▶ transformation sccher Merkkmale 3 trans Fayyad, Piatetsky‐Shapiro trans‐ The KDD process model ▶ discretize Ableitung numeric A attributes neuer Meerkmale n formation Selektion ▶ infer rrelevanterr Merkm. new attributes (cf. Fayyad&et ▶ select relevant attributes Smyth Daata Miniing Data G Generierun ng der Mu uster Das KDD‐Prozessmodell KDD Prozessmodell Mining bzw. Modeelle ▶ data mining al. ) ▶ generate patterns or models Evaaluation n pattern B Bewertungg der Interressant‐ heit durch den Benu ▶ evaluation utzer Validierungg: Statistissche V l i ▶ assess “interestingness” for Prüfung deer Modellee evaluation the user ▶ validate models statistically knowledge  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 15  Example: Clustering DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 16  Example: Outlier DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 17  Example: Classification DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 18  Example: Regression DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 19  Example: Frequent Patterns DM868 DM870 DS804 a,b,c,d,e a,b,c,d,e In 5 cases we have b,c, In 5 out of 10 Arthur Zimek Introduction b,c,d b,c,d and in all those 5 cases (50%) cases, we also have d: a,b,c,d a,b,c,d Data Science Knowledge Discovery from Data a,b,c,d,e c,d,f Data Mining Methods b,c,d Rule with 100% confidence: Preprocessing a,c,e,f a,b,c,d,e Transformation d,c,e,f a,c,d Evaluation occur together. If b,c are in the set, Material a,b,c,d,f a,c,e,f Summary c,d,e,f then also d is in the set. Freq. Pattern Mining Feature Spaces a,b,c,d,f Clustering – Basics Classification – Basics a,b,e,f Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 20  Data Mining Techniques: From Data to Knowledge DM868 DM870 DS804 Arthur Zimek Data Method Knowledge Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation clustering customer profile Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics outlier detection fraud Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning frequent pattern mining product groups References classification star type 21  Patterns and Models DM868 DM870 DS804 Arthur Zimek ▶ predictive models: Introduction Data Science A predictive model should describe (known) data in a Knowledge Discovery from Data Data Mining Methods way suitable to make predictions on unknown data. Preprocessing Transformation ▶ descriptive models: Evaluation Material A model should provide insight in the data, help to Summary Freq. Pattern Mining understand the underlying structure, correlations, Feature Spaces Clustering – Basics properties etc. Classification – Basics Bayesian Learning ▶ A data mining model can be both, descriptive and Learning with Distributions Entropy, Purity, and predictive. (Non-)Linear Sep. Ensemble Learning ▶ Based on both could be a prescriptive model: References Learn from the data for the future: a model to tell you what to change, or how to implement details (typically related to optimization problems). 22  Data Mining Techniques: Categories of Learning DM868 DM870 DS804 Arthur Zimek ▶ supervised (e.g., classification, regression, outlier detection): Introduction Data Science An in general unknown attribute is learned, based on examples (training data) Knowledge Discovery from Data where the attribute is known. Emphasis on predictive modeling. Data Mining Methods Preprocessing ▶ unsupervised (e.g., clustering, outlier detection, association rules) Transformation Evaluation The data are distinguished/organized in different groups without previously Material Summary known examples. Emphasis on descriptive modeling. Freq. Pattern Mining Feature Spaces Clustering – Basics ▶ semi-supervised (e.g., clustering, outlier detection) Classification – Basics Bayesian Learning Semi-supervised techniques are guided by using some information (e.g., only Learning with Distributions one class is known, or some constraints restrict clustering results) Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References Note that: KDD is targeted from very different perspectives, such as researchers in statistics, in machine learning, in databases, and in applied areas such as bioinformatics or economy. Thus, terminology is not always used in the same way. 23  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 24 25 Material Summary References Evaluation Introduction Knowledge Distributions Learning with DS804 Data Science Preprocessing Feature Spaces Transformation Arthur Zimek (Non-)Linear Sep. DM868 DM870 Bayesian Learning Entropy, Purity, and Clustering – Basics Ensemble Learning Freq. Pattern Mining Discovery from Data Classification – Basics Data Mining Methods ▶ focusing ▶ Fo getkussieren: the data KDD ▶ Beschaffun B organize ng der data Datten (file/database) f database ▶ Verwaltung V select g (File/DB relevant dataB) i Selektion rrelevanterr Daten Prozessmodell focusing Databases I: Einführung Knowledge Discovery in process Vo orverarb beitung: ▶ preprocessing In ntegration n von Dateen aus ▶ integrate heterogeneous unterschiedlichen Q Quellen data Vervollstän V ndigung nach model ▶ check for completeness i Konsistenzzprüfung ▶ check for consistency preprocessing Traansform mation D Diskretisierung num meri‐ ▶ transformation sccher Merkkmale 3 trans Fayyad, Piatetsky‐Shapiro trans‐ The KDD process model ▶ discretize Ableitung numeric A attributes neuer Meerkmale n formation Selektion ▶ infer rrelevanterr Merkm. new attributes (cf. Fayyad&et ▶ select relevant attributes Smyth Daata Miniing Data G Generierun ng der Mu uster Das KDD‐Prozessmodell KDD Prozessmodell Mining bzw. Modeelle ▶ data mining al. ) ▶ generate patterns or models Evaaluation n pattern B Bewertungg der Interressant‐ heit durch den Benu ▶ evaluation utzer Validierungg: Statistissche V l i ▶ assess “interestingness” for Prüfung deer Modellee evaluation the user ▶ validate models statistically knowledge  Preprocessing DM868 DM870 DS804 Arthur Zimek Data may be Introduction Data Science ▶ noisy in the sense of containing errors, outliers Knowledge Discovery from Data Data Mining Methods ▶ noisy in the sense of containing lots of irrelevant Preprocessing Transformation information Evaluation Material ▶ incomplete (e.g., missing values, missing attributes, that Summary Freq. Pattern Mining would have been particularly interesting for some given Feature Spaces Clustering – Basics task) Classification – Basics Bayesian Learning Learning with ▶ inconsistent (e.g., different scaling in student evaluation Distributions Entropy, Purity, and sheets for different cohorts, different questions in (Non-)Linear Sep. Ensemble Learning questionaires for different universities) References 26  Typical Preprocessing Tasks DM868 DM870 DS804 Arthur Zimek ▶ data cleaning: impute missing values, smoothing of Introduction Data Science noisy values, identify or remove outliers, resolve Knowledge Discovery from Data Data Mining Methods inconsistencies Preprocessing Transformation ▶ data integration: combination from different data Evaluation Material sources: entity identification, value resolution Summary Freq. Pattern Mining ▶ data reduction: elimination of duplicates Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Note that: Learning with Distributions Entropy, Purity, and Many of these operations do actually change the data, based (Non-)Linear Sep. Ensemble Learning on some assumptions – handle with care if these References assumptions are not explicit! 27  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 28 29 Material Summary References Evaluation Introduction Knowledge Distributions Learning with DS804 Data Science Preprocessing Feature Spaces Transformation Arthur Zimek (Non-)Linear Sep. DM868 DM870 Bayesian Learning Entropy, Purity, and Clustering – Basics Ensemble Learning Freq. Pattern Mining Discovery from Data Classification – Basics Data Mining Methods ▶ focusing ▶ Fo getkussieren: the data KDD ▶ Beschaffun B organize ng der data Datten (file/database) f database ▶ Verwaltung V select g (File/DB relevant dataB) i Selektion rrelevanterr Daten Prozessmodell focusing Databases I: Einführung Knowledge Discovery in process Vo orverarb beitung: ▶ preprocessing In ntegration n von Dateen aus ▶ integrate heterogeneous unterschiedlichen Q Quellen data Vervollstän V ndigung nach model ▶ check for completeness i Konsistenzzprüfung ▶ check for consistency preprocessing Traansform mation D Diskretisierung num meri‐ ▶ transformation sccher Merkkmale 3 trans Fayyad, Piatetsky‐Shapiro trans‐ The KDD process model ▶ discretize Ableitung numeric A attributes neuer Meerkmale n formation Selektion ▶ infer rrelevanterr Merkm. new attributes (cf. Fayyad&et ▶ select relevant attributes Smyth Daata Miniing Data G Generierun ng der Mu uster Das KDD‐Prozessmodell KDD Prozessmodell Mining bzw. Modeelle ▶ data mining al. ) ▶ generate patterns or models Evaaluation n pattern B Bewertungg der Interressant‐ heit durch den Benu ▶ evaluation utzer Validierungg: Statistissche V l i ▶ assess “interestingness” for Prüfung deer Modellee evaluation the user ▶ validate models statistically knowledge  Data Descriptions DM868 DM870 DS804 Arthur Zimek ▶ Many techniques work on feature attributes (feature Introduction Data Science vectors). Knowledge Discovery from Data Data Mining Methods ▶ Other techniques work directly on complex data such as Preprocessing Transformation text, sets, graphs. Evaluation Material ▶ If we are to perform data mining on some complex Summary Freq. Pattern Mining objects, it is an important preprocessing step to derive Feature Spaces Clustering – Basics meaningful features to describe these objects. Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 30  Similarity DM868 DM870 DS804 Arthur Zimek ▶ Similarity (as given by some distance measure) is a Introduction Data Science central concept in data mining, e.g.: Knowledge Discovery from Data ▶ clustering: group similar objects in the same cluster, Data Mining Methods Preprocessing separate dissimilar objects to different clusters Transformation ▶ outlier detection: identify objects that are dissimilar (by Evaluation Material some characteristic) from most other objects Summary Freq. Pattern Mining Feature Spaces ▶ definition of a suitable distance measure is often crucial Clustering – Basics Classification – Basics for deriving a meaningful solution in the data mining task Bayesian Learning Learning with Distributions ▶ images Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning ▶ CAD objects References ▶ proteins ▶ texts ▶... 31  Deriving Features from Complex Objects DM868 DM870 DS804 Arthur Zimek Introduction Data Science ▶ image database Knowledge Discovery from Data ↓ Data Mining Methods Preprocessing color histograms Transformation Evaluation ▶ gene database Material Summary ↓ Freq. Pattern Mining Feature Spaces expression levels Clustering – Basics Classification – Basics Bayesian Learning ▶ text database Learning with Distributions ↓ Entropy, Purity, and (Non-)Linear Sep. word counts Ensemble Learning References Note that: Data mining methods work on the derived feature space no matter the original nature of the object – thus the mapping to 32 a meaningful feature space is of paramount importance.  Typical Transformation Tasks DM868 DM870 DS804 Arthur Zimek ▶ scale, normalize, generalize (e.g., by concept hierarchy) Introduction Data Science Knowledge ▶ data reduction: aggregation, feature combination, Discovery from Data Data Mining Methods dimensionality reduction Preprocessing Transformation ▶ derive new features Evaluation Material Summary Freq. Pattern Mining Note that: Feature Spaces Clustering – Basics Many of these operations do actually change the data, based Classification – Basics Bayesian Learning on some assumptions – handle with care if these Learning with Distributions assumptions are not explicit! Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References 33  Representation Matters DM868 DM870 DS804 Arthur Zimek Introduction Data Science Knowledge Discovery from Data Data Mining Methods Preprocessing Transformation Evaluation Material Summary Freq. Pattern Mining Feature Spaces Clustering – Basics Classification – Basics Bayesian Learning Learning with Distributions Entropy, Purity, and (Non-)Linear Sep. Ensemble Learning References Figure adapted from Figure 1.1, Goodfellow et al.. 34  Outline DM868 DM870 DS804 Introduction Arthur Zimek Data Science Introduction Data Science Knowledge Discovery from Data Knowledge Discovery from Data Data Mining Methods Data Mining Methods Preprocessing Preprocessing Transformation Transformation Evaluation Material Evaluation Summary Material Freq. Pattern Mining Feature Spaces Summary Clustering – Basics Frequent Pattern Mining Classification – Basics Bayesian Learning Feature Spaces Learning with Distributions Clustering – Basics and k-means Entropy, Purity, and (Non-)Linear Sep. Classification – Basics and a Basic Classifier Ensemble Learning References Basic Probability Theory, Bayes’ Rule, and Bayesian Learning Distributions and Learning with Distributions Entropy, Purity, and Separation: Linear vs. Non-Linear Separation Ensemble Learning 35 36 Material Summary References Evaluation Introduction Knowledge Distributions Learning with DS804 Data Science Preprocessing Feature Spaces Transformation Arthur Zimek (Non-)Linear Sep. DM868 DM870 Bayesian Learning Entropy, Purity, and Clustering – Basics Ensemble Learning Freq. Pattern Mining Discovery from Data Classification – Basics Data Mining Methods ▶ focusing ▶ Fo getkussieren: the data KDD ▶ Beschaffun B organize ng der data Datten (file/database) f database ▶ Verwaltung V select g (File/DB relevant dataB) i Selektion rrelevanterr Daten Prozessmodell focusing Databases I: Einführung Knowledge Discovery in process Vo orverarb beitung: ▶ preprocessing In