Securing and Scaling Data Pipelines

• The module is about securing and scaling data pipelines
• The module highlights how cloud security best practices apply to analytics and machine learning (ML) data pipelines
• It lists AWS services for securing a data pipeline
• It describes how infrastructure as code (IaC) supports security and scalability of a data pipeline infrastructure
• The module identifies the function of common AWS CloudFormation template sections

Cloud Security Review

• The AWS Well-Architected Framework includes security as a key pillar
• The Shared Responsibility Model describes the division of security responsibilities between AWS and the customer
• The customer is responsible for security in the cloud
• AWS is responsible for security of the cloud
• Key design principles for data security include implementing a strong identity foundation, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from data, and preparing for security events

Access Management

• Authentication uses credentials to establish the identity of the requestor; it grants or denies access to resources based on identity
• Authentication utilizes usernames, passwords, and multi-factor authentication (MFA) among other methods
• Authorization takes place only after authentication and determines the level of access that an identity has to a resource
• Common authorization methods include attribute-based access control (ABAC) and role-based access control (RBAC)
• The Principle of Least Privilege involves granting only the necessary permissions to perform a task
• Start with a minimum set of permissions, grant additional permissions as necessary, and revoke unnecessary permissions

AWS Identity and Access Management (IAM)

• IAM helps securely share and control access to AWS resources for individuals and groups
• IAM integrates with most AWS services, supports federated identity management, granular permissions, and MFA
• IAM provides identity information for information assurance and compliance audits

Data Security

• Data at rest is any data that persists in nonvolatile storage for any duration
• Data in transit is any data that is sent from one system to another

Data at rest protections:

• Implement secure key management
• Enforce encryption at rest and access control
• Audit the use of encryption keys and data access logs
• Use mechanisms to keep people away from data
• Automate data-at-rest protection

Data in transit protections:

• Implement secure key and certificate management
• Enforce encryption in transit
• Authenticate network communications
• Automate detection of unintended data access
• Secure data from between VPC or on-premises locations
• AWS Key Management Service (AWS KMS) provides the ability to create and manage cryptographic keys
• KMS uses hardware security modules (HSMs) to protect keys, which are integrated with other AWS services
• KMS provides the ability to set usage policies to determine which users can use which keys

Logging and Monitoring

• Logging is the collection and recording of activity and event data
• The logged information varies based on the service
• Common log elements include date and time of event, origin of event, and identity of resources that were accessed
• Monitoring is the continuous verification of the security and performance of your resources, applications, and data
• AWS provides services that give you the visibility to spot issues before they impact operations
• AWS CloudTrail is the primary AWS solution for logging
• CloudTrail assists in enabling governance and compliance and records actions taken by users, roles, or AWS services as events
• CloudTrail can be used to view, search, download, archive, analyze, and respond to account activity across an AWS infrastructure
• Amazon CloudWatch is a monitoring and observability service
• CloudWatch provides a unified view of the operational health of AWS resources, applications, and services
• CloudWatch collects metrics in the AWS Cloud and on premises and can be used to monitor and troubleshoot infrastructure
• CloudWatch customizes logs and events

Key Takeaways for Cloud Security Review

• Access management includes authentication and authorization; adhere to the principle of least privilege with both
• IAM integrates with most AWS services and helps to securely share and control individual and group access to AWS resources
• Securing data at rest and data in transit is a key aspect of a data security plan
• Logging and monitoring can assist in maintaining compliance with local laws and regulations

Security of analytics workloads

• Classify and protect data by understanding data classifications and policies, identify the source data owners,and record data classifications into the Data Catalog
• Implement data encryption and retention policies and honor classifications downstream
• Control data access by allowing data owners to determine access, build user identity solutions, implement data access authorization models, and establish an emergency access process
• Control the access to workload infrastructure by preventing unintended access,implement least privilege policies,monitor the infrastructure changes and user activities and secure infrastructure audit logs
• Securing the Stream Processing Pipeline includes data sources, ingestion and producers, stream storage, stream processing and consumers, and downstream destinations.

Key Takeaways for Security of Analytics Workloads

• Honor data classifications and protection policies set by source data owners
• Secure access to the data in the analytics workload
• Share data downstream in compliance with the source system's classification policies
• Ensure the environment is accessible with the least permissions necessary
• Automate auditing of environment changes and alert in case of abnormal environment access