Data Engineering Concepts and Practices Quiz

Data Engineering Definition

• Data engineering is the movement, manipulation, and management of data.
• It involves creating interfaces and mechanisms for data flow and access.
• Data engineers are specialists dedicated to maintaining data availability and usability.
• Data engineering can be considered a superset of business intelligence and data warehousing, incorporating elements from software engineering.
• It's the development, implementation, and maintenance of systems that take raw data and produce high-quality, consistent information supporting downstream use cases (e.g., analysis, machine learning).
• Data engineering involves the intersection of security, data management, DataOps, data architecture, and software engineering.
• A data engineer manages the entire data engineering lifecycle, beginning with extracting data from source systems and ending with serving data for usage like analysis or machine learning.

The Data Engineering Lifecycle

• The lifecycle includes stages: generation, storage, ingestion, transformation, and serving data.
• These are interrelated and interdependent
• Undercurrents (e.g., security, data management, DataOps, data architecture, and software engineering) cut across multiple lifecycle stages.

Data Engineering History

• Early days (1980s-2000s): Focused on data warehousing and business intelligence.
• Early 2000s marked the start of contemporary data engineering and the rise of decentralization and breaking apart monolithic services with the emergence of "big data" technologies, like Apache Hadoop, AWS, Amazon S3, and DynamoDB.
• 2000s/2010s: Saw the simplification of open-source big data tools.
• 2020s: Trend towards decentralized, modularized, managed, and highly abstracted tools in data lifecycle engineering.
• Modern data stack: Collection of off-the-shelf open-source and third-party tools assembled to simplify data analysis.

Data Maturity and the Data Engineer

• Data engineering complexity depends heavily on company data maturity.
• Data maturity is the progression toward higher data utilization, capabilities, and organization-wide integration.
• Data maturity models (e.g., DMM) have different versions.
• Data maturity stages exist, such as Starting, Scaling, and Leading with data

Data Engineer Background and Skills

• Business Responsibilities: Communication (with technical and non-technical people), understanding business/product requirements, understanding agile/devops/dataops, cost control and continuous learning.
• Technical Responsibilities: Building optimized performance and cost effective architectures using premade or in-house components. Possessing all skills related to the data engineering lifecycle.

Data Engineers and Other Technical Roles

• Data engineers are positioned between upstream data producers (like software engineers and data architects) and downstream consumers like data analysts, data scientists, and machine learning engineers.

The Data Engineering Lifecycle Stages (Details)

• Generation: Originating data from source systems (relational databases, NoSQL, IoT), and data streams (essential characteristics—persistence, frequency of generation, errors, schema presence).
• Storage: Data storage choices like warehouses, lakehouses, and databases, with considerations for access frequency (hot, lukewarm, cold data) and storage systems suitability.
• Ingestion: Data movement from sources with challenges like batch vs streaming and pull vs push models for optimal data flow.
• Transformation: Processing data to meet downstream needs through mapping, cleaning, normalization, and creating new features. Often used in other lifecycle phases.
• Serving Data (Analytics): Data value for practical purposes. Analytics core to many enterprises, including business intelligence. Operational analytics focuses on real-time operations in contrast to BI focusing entirely on past and present. Multitenancy commonly used to house customer and analytical data with logical views.
• Serving Data (Machine Learning): Feature store is a tool that combines data engineering.
• Reverse ETL: A process that takes processed data and feeds this back into source systems, allowing data to be fed back into production systems or SaaS platforms.

Major Undercurrents Across the Data Engineering Lifecycle

• Undercurrents:
• Security: access control, data governance.
• Data Management: discoverability, definitions, accountability.
• DataOps: observability, monitoring, incident response.
• Data Architecture: analyzing tradeoffs, agility design.
• Orchestration: coordinating workflows, managing tasks.
• Software Engineering: programming skills, design patterns.

Data Management and Governance

• Data management encompasses processes for planning, execution, and supervision related to data throughout its lifecycle.
• Data governance focuses on data modeling and design.
• Key aspects: data lineage, storage, operations, integration, lifecycle management, advanced analytics systems, ethics, and privacy

Orchestration in Data Engineering

• Coordinating various data processing jobs efficiently on a schedule, different from simple schedulers.
• Builds in metadata about job dependency relationships and utilizes DAGs (directed acyclic graphs).
• Provides job history, visualization, and alerting for improved workflow management. Also includes coordinating workflows.

DataOps

• Applying Agile, DevOps, and SPC (statistical process control) principles and practices to data by streamlining data flows.
• Enables rapid, high-quality data delivery and collaboration for faster insights. Includes collaboration; measuring, monitoring and transparency of results; and automation, incident response, and monitoring as core practices.

Software Engineering in Data Engineering

• Core data processing code (SQL) and development of open-source frameworks (like Hadoop ecosystem).
• Infrastructure as Code (IaC) for configuring and managing infrastructure, pipelines as code for data orchestration, and general-purpose problem-solving skills.

Enterprise Architecture

• Holistic view of the entire enterprise information and technology, including services, processes, and infrastructure. This creates a roadmap for success.
• Enterprise Architecture (EA) is a model aligned with strategy, operations, and technology. It's about designing systems to handle change flexibly with reversible decisions.

Data Architecture

• Reflects the current and future state supporting organizational long-term data needs and strategy.
• Defines structure and interaction among data types, sources, logical assets, and management resources.
• Flexible and easily maintainable data architecture serves business needs, adaptable to change, and considered an ongoing process

Principles of Good Data Architecture

• Principles for data architecture, such as AWS (Operational excellence, Security, Reliability, and Cost Optimization) and Google (Automation, State design, and Managed Services).

Examples & Types of Data Architectures

• Data warehouse and data marts
• Data lakes
• Data lakehouses
• Modern data stacks
• Lambda Architecture
• Kappa Architecture
• IoT Architecture
• Data Mesh

Data Warehouse

• Central hub for data used in reporting and analysis, often separates analytics from production tasks, and centrally stores data.
• Cloud-based warehouses are increasingly accessible. Two primary characteristics are separating analytics from production tasks and centralizing data.

ETL vs ELT

• Data movement using ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) architecture, which differentiates based on where data transformation happens. ETL transforms first, then loads. ELT loads and transforms data later.

Data Marts

• A subset of a data warehouse tailored to a specific department or business unit, enhancing data accessibility to analysts. Data marts extend transformations beyond the main ETL pipeline.

Data Lake vs Data Lakehouse

• Data lake stores structured and unstructured data centrally, while the lakehouse introduces data management controls from data warehouses to manage structured and unstructured data in object storage while offering query and transformation engines.
• Lakehouse maintains ACID compliance.

Modern Data Stack

• Cloud-based, modular, cost-effective data architectures using off-the-shelf components including pipelines, storage, transformation, governance, and monitoring, all with visualization and exploration capabilities.

Lambda Architecture

• A system employing batch, stream, and serving processes independently in a response to analyzing streamed data.
• This method sends data to two destinations (stream and batch processing), creating independent processes for each. Includes several shortcomings.

Kappa Architecture

• A data architecture aiming to use a stream processing platform as the backbone for ingestion, storage, and serving that represents a true event-based system.
• Real-time and batch processing seamlessly handled using a live stream, simplifying transformation for both types of processing.

Architecture for IoT

• IoT data architecture for handling distributed devices and collecting data from various internet-connected devices. A distributed environment.

Data Mesh

• Represents a recent approach that decentralizes data architecture addressing the challenges of centralized data systems, taking domain-driven design ideas.

Unified Data Infrastructure (2.0)

• Describes Unified Data Infrastructure (UDI) which contains data sources, ingestion, transport, storage, query and processing, transformation, and analysis stages in more detail on data structures, tools, and responsibilities.

Machine Learning Infrastructure (2.0)

• A diagram describing end-to-end machine learning, showing interconnected processes like data sources, data transformation, model training and development, model inference, and integration stages.

Blueprint 1 - Modern Business Intelligence

• Shows a complete data architecture for business intelligence including data sources, ingestion, storage, query and processing, transformation and output stages.

Resources

• Joe Reis and Matt Housley provide a book called Fundamentals of Data Engineering, offering practical guidance for constructing data systems that are robust.
• Matt Bornstein et al. have written about Emerging Architectures for Modern Data Infrastructure.