Data Engineering CH01: Introduction

Data Engineering

• Data engineering involves creating interfaces and mechanisms to manage the flow and access of information.
• Data engineers are responsible for maintaining data to ensure it remains accessible and usable for others.
• Data engineers establish and manage an organization's data infrastructure, readying it for analysis by data analysts and scientists.

Data Engineering Lifecycle

• The data engineering lifecycle comprises stages that turn raw data ingredients into a useful end product, ready for consumption by analysts, data scientists, ML engineers, and others.
• The stages include:
  • Generation
  • Storage
  • Ingestion
  • Transformation
  • Serving

Role of a Data Engineer

• A data engineer converts raw data into usable data, providing analytics-ready data to data consumers.
• A data engineer extracts, organizes, and integrates data from disparate sources.
• A data engineer prepares data for analysis and reporting by transforming and cleaning it.
• A data engineer designs and manages data pipelines that encompass the journey of data from source to destination systems.
• A data engineer sets up and manages the infrastructure required for the ingestion, processing, and storage of data.

Stakeholders

Upstream Stakeholders

• Data architects design the blueprint for organizational data management, mapping out processes and overall data architecture and systems.
• Software engineers build the software and systems that run a business, generating internal data that data engineers will consume and process.
• DevOps engineers and site-reliability engineers produce data through operational monitoring.

Downstream Stakeholders

• Data scientists use data analytics and data engineering to make predictions and recommendations using data from the past.
• Data analysts use data scientists' insights and predictions to drive decisions that benefit and grow their business.
• Machine learning engineers develop advanced ML techniques, train models, and design and maintain the infrastructure running ML processes in a scaled production environment.

Data Engineering vs Data Science

• Data engineering sits upstream from data science, providing inputs used by data scientists, who convert these inputs into something useful.

Data Science Hierarchy of Needs

• Collect: Determine what data is needed and what is currently available.
• Move/Store: Secure movement, organization, and storage of the data.
• Explore/Transform: Focus on data exploration and analysis, including anomaly detection and data cleaning.
• Aggregate/Label: Classify information and execute basic analytics.
• Implement/Optimize: Test, learn, and optimize data usage.
• AI and Deep Learning: Delve into experimentation and scale up the use of machine learning models.