Understanding Data Engineering Concepts

• Data consists of facts, observations, and perceptions.
• Data is essential in science, business, healthcare, and tech
• Data is processed to make it meaningful.
• Organizations use data to gain a competitive edge

Data Value

• Accuracy and accessibility are key components
• The job of a Data Engineer is to ensure accuracy and accessibility

Data Engineering

• It involves creating interfaces and mechanisms to manage data flow
• Data Engineers maintain data to ensure usability
• Data Engineers establish and manage an organization's data infrastructure
• Analysts and scientists use the data after it is prepared by Data Engineers

Data Engineering Lifecycle

• Generation is the start point in the data engineering lifecycle
• It involves storage, Ingestion, and transformation
• Serving data is another stage in data engineering lifecycle
• The lifecycle is supported by security, data management, DataOps, and data architecture

Data Engineer

• Transforms raw data into usable analytics-ready data
• Ensures data is accurate, reliable, and follows regulations
• Accessible data is a key result
• A data engineer's responsibilities include extracting, organizing, and integrating data
• Data engineers prepare data for analysis and reporting by transforming it
• They design and manage data pipelines and set up the necessary infrastructure
• They make data accessible for business uses and to stakeholders
• "Big data engineers" are now simply called "data engineers"

Essential Skills

• Technical skills
• Functional skills
• Soft skills

Technical Prowess

• Knowledge of operating systems like UNIX, Linux, and Windows
• Understanding infrastructure components: virtual machines, networking, etc
• Cloud-based services from Amazon, Google, IBM, and Microsoft
• RDBMS knowledge, such as IBM DB2, MySQL, Oracle and PostgreSQL
• Understanding of NoSQL databases: Redis, MongoDB, Cassandra, and Neo4J
• Data warehouses: Oracle Exadata, IBM Db2 Warehouse on Cloud, Amazon RedShift
• Data Lakes: Azure Data Lake Storage, AWS Lake Formation, Alder Lake, Google BigLake.
• Working with data pipelines is useful
• Data pipeline solutions include Apache Beam, AirFlow, and DataFLow
• ETL tools such as IBM Infosphere, AWS Glue, and Improvado are important
• Proficiency in languages for data querying/manipulation/processing is needed
• SQL for relational databases and SQL-like query languages for NoSQL databases
• Programming languages like Python, R, and Java
• Shell and Scripting languages, such as Unix/Linux Shell, PowerShell
• Familiarity with BigData processing tools like Hadoop, Hive, and Spark

Functional Skills

• Convert business requirements into technical specifications: a core functional skill
• Working with complete software development lifecycle stages is important
• Software development lifecycle stages: Ideation, architecture, design, testing, etc
• A data engineer must Understand data potential application in business
• Must understand risks of poor data management covering data quality, privacy, security, compliance

Soft Skills

• Interpersonal skills
• Teamwork
• Collaboration
• Effective communication

Technical Roles

• The Data Engineer is a hub connecting data producers and consumers
• Data producers: software engineers, data architects, and DevOps/SREs
• Data Consumers: data analysts, data scientists, and ML engineers
• Data engineers interact with those in operational roles like DevOps engineers

Upstream Stakeholders

• Data architects design the blueprint
• They map processes for organizational data management
• Act as a bridge between technical and nontechnical sides
• Software engineers build software; are responsible for generating data that data engineers use
• DevOps/SREs produce data through operational monitoring
• They may be downstream too, consuming data through dashboards

Downstream Stakeholders

• Data Scientists use Data Analytics and Data Engineering to make predictions and recommendations
• Data Analysts (Business Analysts) use those predictions to drive decisions
• ML engineers overlap with Data Engineers and Data Scientists
• They develop advanced techniques, train models, and maintain infrastructure

Data Engineering vs Data Science

• Data engineering sits upstream from data science
• Data engineers provide the inputs that data scientists convert

Data Science Hierarchy

• Data collection is the first step for a data scientist
• The next is movement, securing organization, and storage of data
• Data exploration and analysis, including data cleaning are then performed
• Data classification and and basic analytics occur during the data aggregation stage
• Analytics, metrics, and training data allow testing, learning, and optimization
• AI and deep learning, with the right resources and data, are at the top

Types of Data:

• Structured data: facts and values (5-10%)
• Unstructured data: contains information (80%)
• Semi-structured data: tags with elements (10 - 20%)

Structured Data

• It has a well-defined structure, can be stored in schemas, can be tabular
• Represents only 5% to 10% of all data
• Supports objective numbers and facts
• Is the simplest way to manage information
• Can be collected, exported, stored, and organized in typical databases

Sources of Structured Data

• SQL Databases
• OLTP systems
• Spreadsheets like Excel and Google Sheets
• Online forms with GPS and RFID tags
• Network and web server logs

Pros of Structured Data

• Structured data allows Machine learning and algorithmic usage
• It is easier to use for business users who can understand the subject matter
• More tools are offered since it predates unstructured data.

Cons of Structured Data

• Data can only be used for its intended purpose
• Commonly stored with rigid schemas in "data warehouses"
• Any adjustments will lead to massive time expenditure.

Use Cases for Structured Data

• CRM software analyzes customer behavior with analytical tools
• Hotel and ticket bookings
• Accounting firms and departments
• HR employee data
• Admissions/Enrollment

Tools for Structured Data

• PostgreSQL
• SQLite
• MySQL
• Oracle Database
• Microsoft SQL Server

Unstructured Data

• Does not have an identifiable structure
• Cannot be easily organized
• Is increasing rapidly representing over 80% of all data
• Does not follow format, sequences, semantics, and rules
• Contains sources with analytics applications

Sources of Unstructured Data

• Web pages
• Social Media feeds
• Images, Videos, and Audio files
• Documents
• Surveys
• Media Logs
• Powerpoint

Pros of Unstructured Data

• Retains Adaptability, increasing file formats.
• Can be collected very quickly
• Is pay-as-you-use

Cons of Unstructured Data

• Requires experience in data science
• Unique tools are required, increasing time and resources
• Complexity adds time to usage with algorithms

Use Cases for Unstructured Data

• Uses text within social media posts for sentiment
• Uses processing within videos for facial recognition
• Voice recordings for speech sentiment
• Big data usage with volume data needing insights
• Mining data for consumer habits
• Analytics for alert patterns/shifts
• Chatbots for text analysis

Tools for Unstructured Data

• MongoDB
• Hadoop
• Data Lake
• Amazon DynamoDB

Semi-Structured Data

• Has organizational traits, but lacks a rigid schema
• Contains tags and elements to organize it correctly
• Can't be stored with rows and columns

Sources of semi-structured data

• XML files
• Binary executables
• Zips
• Email

Pros of Semi-Structured Data

• Modifications that change data are flexible
• Scalability to handle various formats
• Easy capture due to no predefined data
• Performance with nested databases

Cons of Semi-Structured Data

• Complexity when dealing with nested databases
• Challenges when indexing efficiently
• Integrate with difficult data through customs
• Relational and Non-relational data can have defined relationships
• Non SQL data is for not only SQL within the data
• Both databases are different through the types of data presented

Data Engineering Skills

• ETL processes move data into warehouses
• Data Storage must be proper to design proper data solutions
• In ETL (extract, transform, and load) systems, data moves from databases/sources to a repository like a data warehouse.

Data Extraction, Transformation, Loading

• Data is copied from multiple source locations.
• SQL and NoSQL servers, CRMs, Text and document files and websites are common sources
• The transformation involves manipulating data, filtering it, and authenticating it
• Load can be done around transformations based on machine
• Cloud computating is an efficient way to process all data at once

Data Warehouses vs Data Lake

• Data warehouses use designed schemas for transactional, operational databases
• Fast query results are needed due to higher cost storage
• Warehouse requires highly curated Data as the truth
• Lakes use non-relational, IOT devices, mobile apps, etc