Data Engineering Overview

Data Engineering: What? Why?

• Data engineering is a crucial component of real-world data science projects.
• It involves a wide range of activities, such as collecting, collating, extracting, moving, transforming, cleaning, integrating, organizing, representing, storing, and processing data.
• Data engineers handle large, often messy datasets across various teams and organizations, frequently with unclear or ill-defined objectives.
• Data engineering tasks are often underappreciated compared to machine learning activities in data science projects, despite the vital role they play.
• The role of data engineers is expanding rapidly with more roles now available than data scientists.

Data Science: The Conventional View

• A data scientist traditionally works alone, using one static, rectangular dataset in main memory.
• Statistical and machine learning algorithms are applied to predefined objectives.
• While valuable, this approach often ignores the full picture, particularly when large, dynamic datasets are involved.

Data Science Today with Data Engineering

• Modern data science often involves data engineering.
• Data engineering's activities encompass collecting, collating, extracting, moving, transforming, cleaning, integrating, organizing, representing, storing, and processing data.
• Data engineering happens across teams and requires working with large, messy datasets that are often non-rectangular.
• The objectives are frequently unclear. and not well-defined.

Why Learn Data Engineering?

• Most time in real-world data science projects is spent on data-related tasks, such as cleaning, moving, and processing data.
• Data engineering lays the groundwork for machine learning and AI models.
• There is significant demand for data engineers.
• Data engineering requires fundamental skills needed for effective data-driven decision-making.

Modern Data Ecosystem

• The constant acceleration of data processing speeds and bandwidth leads to the constant creation and improvement of tools for creating, sharing, and consuming data.
• The increase of data creators and consumers worldwide continuously contributes to the expansion of data.
• Data's value increases with more data.
• Organizations leverage data to understand and capitalize on opportunities to distinguish themselves from competitors.

Key Players in the Modern Data Ecosystem

• Data Engineers
• Data Analysts
• Data Scientists
• Business Analysts
• Business Intelligence Analysts

Data Engineering Lifecycle

• Data preparation process involves moving raw data into usable formats, often in a process called ETL (Extract, Transform, Load).
• Data must integrate into existing systems (e.g., warehousing), then be verified and audited.

ETL vs ELT

• ETL (Extract, Transform, Load) is a traditional approach.
• Data transformations are done in SQL, handling structured and unstructured data but requiring deep knowledge of the warehousing tools.
• ELT (Extract, Load, Transform) is a newer method.
  • Transformations are more flexible, faster, and scalable.

Metadata

• Metadata is critical for managing the storage of data and includes information about data entities and their relationships, constraints, lineage, and usage trails.

Operationalization and Feedback

• A core aspect of real-world data science is its long-term development and use.
• Feedback loops provide continuous updates to processes based on the results of experiments, tests, jobs and predictions.
• Data “products” are crucial, as they often generate insights needed for decision-making.

Modern Data Solutions

• Most modern data solutions combine data warehousing and data lakes with many-to-many ETLT transformations.
• Data flows move between and into these systems as necessary.