Data Mining Concepts and Process

Data Mining Concepts

• Data mining is a process of discovering patterns and knowledge from large data amounts.
• Data sources include databases, data warehouses, and the web and any data streamed into the system dynamically.
• Ogunleye et al (2021) defines data mining as a process used by companies to transform raw data into useful information.

Data Mining Extraction Process

• Involves examining raw data to build a model describing information and generating reports for business use.
• Building a model from various data sources and formats is iterative due to the diverse availability of raw data.
• The continuous increase in data means that new data sources can change results.

Data Mining Process Steps

• The data mining process contains data preprocessing and data mining.
• Data preprocessing includes data cleaning, integration, reduction, and transformation.
• The data mining phase includes data mining, pattern evaluation, and knowledge representation.

Data Mining Process

• The data mining process involves understanding the business, the data, preparing the data, building the model, evaluating the results, and implementing change and monitoring.
• To be effective, data analysts follow a certain flow of tasks along the data mining process.

Data Preprocessing

• There are different data mining processing models will have different steps, though the general process is usually pretty similar.
• Knowledge Discovery Databases model has nine steps
• CRISP-DM model has six steps
• SEMMA process model has five steps.
• Preprocessing is crucial to ensure data accuracy, completeness, consistency, and timeliness, ensuring it meets the intended purpose.

Major Steps in Data Preprocessing

• Data Cleaning is the first step to ensure dirty data does not confuse procedures and produce inaccurate results.
• Data cleaning involves the removal of noisy or incomplete data.
  • Filling missing data can be done by:
    • Ignoring the tuple
    • Filling the value manually, using central tendency measures, or using the most probable value.
  • Noisy data removal methods include:
    • Binning (sorting values into buckets/bins with smoothening via neighboring values)
    • Smoothing by replacing each value with the bin's mean or median
    • Smoothing by bin boundaries, replacing values with the closest boundary value
    • Identifying outliers and resolving inconsistencies

Data Integration

• Data Integration combines data from multiple heterogeneous sources like databases, data cubes, or files to improve accuracy and speed.
• Different naming conventions in databases can cause redundancies
• Redundancies and inconsistencies can be removed from the integrated data without compromising reliability through additional cleaning.
• Data migration tools like Oracle Data Service Integrator and Microsoft SQL facilitate data integration.

Data Reduction

• Data Reduction obtains relevant data for analysis from the collection of data and maintains integrity while reducing volume, using methods like Naive Bayes, Decision Trees, and Neural Networks.
• Strategies for data reduction:
  • Dimensionality Reduction (reducing the number of attributes)
  • Numerosity Reduction (replacing original data with smaller representations)
  • Data Compression (creating a compressed representation).

Data Transformation

• Data Transformation involves transforming data into a suitable format for efficient mining and easier pattern understanding, including Data Mapping and code generation.
• Strategies include:
  • Smoothing (removing noise using clustering or regression)
  • Aggregation (applying summary operations)
  • Normalization (scaling data to a smaller range)
  • Discretization (replacing numeric values with intervals).

Data Mining and Pattern Evaluation

• Data Mining identifies interesting patterns and knowledge from large data amounts, with intelligent patterns applied to extract data patterns.
• Data is represented through patterns structured using classification and clustering techniques.
• Pattern Evaluation involves identifying interesting patterns based on interestingness measures.
• Data summarization and visualization methods are used to enhance user understanding.

Knowledge Representation

• Focuses on visualizing mined data through reports, tables using representation tools.

Data Mining in Oracle DBMS

• RDBMS represents data in tables with rows and columns accessible through database queries.
• Oracle supports data mining using CRISP-DM, with facilities useful in data preparation and understanding.
• Includes Java, PL/SQL interfaces, automated data mining, SQL functions, and graphical user interfaces.

Data Mining in Datawarehouse

• Modeled for multidimensional structure.
• Each cell in a data cube stores the value of some aggregate measures
• In multidimensional space carried out in OLAP style offers exploration of multiple combinations of dimensions at varying levels of granularity.

Data Mining Challenges

• Data Mining requires managing of difficult large databases and data collection.
• Domain experts are required and difficult to find, complex integration of data from heterogeneous databases.
• Organizational level practices need modification to for data mining results, restructuring requires effort and cost.

Applications of Data Extraction

Areas where data mining is widely used:

• Financial Data Analysis: Used in banking, investment, credit services, and insurance for systematic data analysis.
• Retail and Telecommunication Industries: Helps identify customer behaviors and shopping patterns to improve service and satisfaction.
• Science and Engineering: Monitors system status, isolates software bugs, and detects plagiarism.
• Intrusion Detection and Prevention: Enhances intrusion detection systems by identifying threats.
• Recommender Systems: Recommends products of interest to consumers.

Explore the fundamental concepts of data mining, including its definition, data sources, and extraction process. Understand the iterative nature of building models from diverse data and the key steps involved, such as preprocessing and pattern evaluation. Learn how raw data transforms into valuable information.