The Data Explosion

The Data Explosion

• Modern computer systems are accumulating data at an unimaginable rate from a wide variety of sources, including point-of-sale machines, machines logging cheque clearance, bank cash withdrawals, credit card transactions, and Earth observation satellites.
• The volume of data is enormous, with examples including:
  • NASA Earth observation satellites generating a terabyte (10^9 bytes) of data every day.
  • The Human Genome project storing thousands of bytes for each of several billion genetic bases.
  • Data warehouses containing over a hundred million customer transactions.
  • Automatic recording devices, such as credit card transaction files and web logs, as well as non-symbolic data such as CCTV recordings.
  • Over 650 million websites, with some extremely large sites.
  • Over 900 million Facebook users, with an estimated 3 billion postings a day, and 150 million Twitter users, sending 350 million tweets a day.

Knowledge Discovery

• Knowledge Discovery is the non-trivial extraction of implicit, previously unknown, and potentially useful information from data.
• It involves a process of data mining, which is a central part of the Knowledge Discovery process.
• The Knowledge Discovery process involves:
  • Data coming in from many sources.
  • Data integration and storage in a common data store.
  • Pre-processing of data into a standard format.
  • Applying a data mining algorithm to produce rules or patterns.
  • Interpreting the output to gain new and potentially useful knowledge.

Types of Data and Data Mining

• There are two types of data: labelled and unlabelled data.
• Labelled data is used for supervised learning, where the aim is to predict the value of a designated attribute for unseen instances.
• Unlabelled data is used for unsupervised learning, where the aim is to extract the most information possible from the available data.
• Data mining applications can be divided into four main types:
  • Classification: predicting a categorical value, such as classifying medical patients into high, medium, or low risk of acquiring an illness.
  • Numerical Prediction: predicting a numerical value, such as the expected sale price of a house.
  • Association: finding relationships amongst variables, such as in market basket analysis.
  • Clustering: grouping items that are similar, such as customers according to income, age, and types of policy purchased.

Classification

• Classification is a common application of data mining, involving predicting a categorical value.
• Examples include:
  • Classifying medical patients into high, medium, or low risk of acquiring an illness.
  • Classifying people into those likely to vote for different political parties.
  • Classifying student projects into distinction, merit, pass, or fail.

Association Rules

• Association rules involve finding relationships amongst variables, such as in market basket analysis.
• An example of an association rule is: IF cheese AND milk THEN bread (probability = 0.7), indicating that 70% of customers who buy cheese and milk also buy bread.

Clustering

• Clustering algorithms examine data to find groups of items that are similar.
• Examples include:
  • Grouping customers according to income, age, and types of policy purchased.
  • Grouping electrical faults according to the values of certain key variables.