Data Science vs Data Engineering

Data Science vs Data Engineering

• Data Science combines domain expertise, coding skill, and knowledge of mathematics and statistics skills to extract meaningful insights from data
• Data Engineering focuses on data formats, storage, extraction, and transformation
• Data analysis translates a business into a question and make accuracy-cost trade-offs
• Data Modeling includes classification, regression, and anomaly detection
• Data Engineering includes: data management, production and software engineering

Data Scientist vs Data Engineer

• Data engineers usually have more technical expertise and solid data warehousing and programming backgrounds
• Data scientists tend to be more mathematical
• There is crossover between the roles
• Machine learning models require writing small applications and heavy data manipulation

Data Maturity and the Data Engineer

• Data engineering complexity depends on a company's data maturity
• Data maturity is the progression toward data utilization, capabilities, and integration
• Data maturity depends how data is leveraged as a competitive advantage

Data Engineer Responsibilities

• Analyzing and organizing raw data
• Building data systems and pipelines
• Evaluating business needs and objectives
• Interpreting trends and patterns
• Preparing data for prescriptive and predictive modeling
• Building algorithms and prototypes
• Developing analytical tools and programs

Data Engineering Skills

• Coding proficiency is essential in languages like SQL, NoSQL, Python, Java, R, and Scala
• Should be familiar with relational and non-relational databases and how they work
• Needs ETL (extract, transform, and load) systems knowledge which is the process of moving data from databases and other sources into a single repository, like a data warehouse
• Requires big data tools and various technologies
• Should comprehend cloud computing and data security

Why Python

• Python is easy and simple
• Python is efficient and performs bulky tasks using fewer lines of code
• Python has diverse libraries and frameworks
• Python is versatile where one can implement Python on almost all software, actions, and infrastructures
• Python has a vast community that supports Python learners
• Python is portable and extensible as it can be used on any other platform without making any significant changes
• Python is flexible, developers can choose a programming style between OOPs and scripting
• Python has attractive documentation, lessons, and tutorials

Relational and Non-relational Databases

• A relational database is a collection of data items with pre-defined relationships between them organized as a set of tables with rows and columns
• Non-relational database does not use the tabular schema of rows and columns
• NoSQL stands for "not only SQL"
• Examples of this model are: Documents, Semi-structured data, and Large and unstructured data which come results from the Internet of Things (IoT), social networks, and the rise of Al

ETL Process

• ETL is the process of moving data from databases and other sources into a single repository like a data warehouse
• The components of the ETL process are: extraction, transformation, and load

Data Extraction

• Extracting data gets data
• The data is copied or exported from source locations to a staging area
• The data comes from structured or unstructured sources like from SQL or NoSQL servers, CRM and ERP systems, text and document files, emails, web pages, and more

Data Transformation

• 80% of data science work is data preparation, with 75% of data scientists finding this the most boring
• Raw data is transformed to be useful for analysis and to fit the schema of the eventual target data warehouse
• Data engineers bring their skill in manipulating data to a project, which includes:
  • Filtering, cleansing, de-duplicating, validating, and authenticating the data
  • Performing calculations, translations, or summaries based on the raw data
  • Formatting the data into tables or joined tables to match the target data warehouse schema

Data Loading

• Load is the movement and storage of data or storing data to the target location
• Data engineers sometimes swap the load and transform steps around to be (ELT) when dealing with big data technologies such as Hadoop/Spark
• Extraction process is cheaper
• Spreads the processing burden across multiple machines/clusters
• Cloud computing separates storage and computational machines, where one can scale down expensive machines used to process data without affecting the stored data
• A data lake is a centralized repository that allows storing structured and unstructured data at any scale

Data Warehouses vs Data Lake

• Data warehouse hold relational data coming from transactional systems, operational databases, and line of business applications, while Data lakes hold non-relational and relational data from IoT devices, web sites, mobile apps, social media, and corporate applications
• Warehouses are designed prior to the DW implementation (schema-on-write) but Lakes are Written at the time of analysis (schema-on-read)
• Data warehouses have faster query results using higher cost storage but Data lakes have query results getting faster using low-cost storage
• Data warehouses have highly curated data that serves as the central version of the truth while Data Lakes have any data that may or may not be curated (ie. raw data)
• Warehouses are analyzed by business analysts while Lakes are by data scientists, data developers, and business analysts (using curated data)

Data Engineering Skills

• Data engineers need to grasp the basic concepts and understand the needs of data scientists on the work team with machine learning
• Data engineers are tasked with managing big data with technologies that include Hadoop, MongoDB, and Kafka
• Data engineers need to understand cloud storage and computing
• Data engineers are tasked with securely managing and storing data to protect it from loss or theft