Cloud Security for Data Pipelines

Module Objectives

• Cloud security best practices apply to analytics and machine learning data pipelines.
• AWS services play key roles in securing a data pipeline.
• Infrastructure as code (IaC) supports the security and scalability of a data pipeline infrastructure.
• It is important to identify the function of common AWS CloudFormation template sections.

AWS Well-Architected Framework

• The AWS Well-Architected Framework includes security as a core pillar alongside operational excellence, reliability, performance efficiency, cost optimization, and sustainability.

Shared Responsibility Model

• In the cloud, customers are responsible for security "in" the cloud, which includes customer data, platform, applications, identity and access management, operating system, network, and firewall configuration.
• Client-side and server-side data encryption, along with networking traffic protection, also fall under the customer’s responsibilities.
• AWS is responsible for the security "of" the cloud, which includes AWS foundation services like compute, storage, databases, and networking, as well as the AWS Global Infrastructure.
• The AWS Global Infrastructure includes Regions, Availability Zones, and Edge Locations.

Design Principles for Data Security

• Implement a strong identity foundation is a key first step
• Enable Traceability ensures actions can be tracked and audited
• Security should be applied at all layers
• Automate Security best practices to reduce human error and improve consistency.
• Protect data in transit and at rest through encryption and access controls.
• Keep people away from data by restricting direct access and using automated processes.
• It is important to Prepare for security events by having incident response plans in place.

Access Management - Authentication

• Authentication uses credentials to establish the identity of the requestor.
• Authentication grants or denies access to resources based on identity.
• Authentication utilizes usernames, passwords, and multi-factor authentication (MFA) among other methods.

Access Management - Authorization

• Authorization takes place only after authentication.
• Authorization determines the level of access that an identity has to a resource.
• Authorization methods include attribute-based access control (ABAC) and role-based access control (RBAC).

Access Management - Principle of Least Privilege

• Operate under the principle of least privilege which grants only the permissions that are required to perform a task.
• Start with a minimum set of permissions.
• Only gran additional permissions only as necessary.
• Revoke unnecessary permissions as circumstances change.

AWS Identity and Access Management (IAM)

• AWS IAM helps you to securely share and control access to AWS resources for individuals and groups.
• IAM integrates with most AWS services.
• IAM supports federated identity management and granular permissions.
• Supports MFA
• IAM offers identity information for information assurance and compliance audits.

Data Security - Data At Rest

• Data at rest is any data that persists in nonvolatile storage for any duration with implement secure key management.
• Enforce encryption at rest, enforce access control, audit the use of encryption keys, and use mechanisms to keep people away from data.
• Automate data-at-rest protection.
• Audit data access logs.

Data Security - Data In Transit

• Data in transit is any data that is sent from one system to another with the use of Implement secure key and certificate management.
• Enforce encryption in transit and authenticate network communications.
• Automate detection of unintended data access.
• Secure data from between VPC or on-premises locations.

AWS Key Management Service (AWS KMS)

• AWS KMS provides the ability to create and manage cryptographic keys.
• KMS uses hardware security modules (HSMs) to protect keys.
• KMS is integrated with other AWS services, and provides the ability to set usage policies to determine which users can use which keys.

Logging

• Logging is the collection and recording of activity and event data.
• The information logged varies based on the service.
• Common log elements include date and time of event, origin of event, and identity of resources that were accessed.

Monitoring

• Monitoring is the continuous verification of the security and performance of your resources, applications, and data.
• AWS provides several services that give you the visibility to spot issues before they impact operations.

AWS CloudTrail

• CloudTrail is the primary AWS solution for logging.
• CloudTrail assists to enable governance and compliance as well as operational and risk auditing of your AWS account.
• Records actions taken by a user, role, or AWS service as events.
• CloudTrail can be used to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure.

Amazon CloudWatch

• CloudWatch is a monitoring and observability service
• Provides a unified view of the operational health of your AWS resources, applications, and services.
• CloudWatch collects metrics in the AWS Cloud and on premises and monitors and troubleshoot infrastructure
• CloudWatch can be used to customize logs and events.

Key Takeaways Regarding Cloud Security:

• Manage access through authentication and authorization while prioritizing least privilege.
• IAM integrates with most AWS services and assists to securely share and control resource access.
• Securing data both at rest and in transit is crucial for a comprehensive data security plan.
• Logging and monitoring are essential for maintaining compliance with laws and regulations.

Data Classification

• It is required to understand data classifications and policies by identifying source owner, record data classification in the data catalog and implement encryption and retention policies for data.
• It is recommended to honor classifications downstream.

Data Access Controls

• It is required to build user identity solutions to Allow data owners to determine access, and implement data access authorization models.
• It is recommended to Establish an emergency access process.

Workload Infrastructure Access Control

• It is required to prevent unintended access and implement least privilege polices.
• Recommended to Monitor infrastructure changes and user activities and maintain infrastructure audit logs.

Securing the Stream Processing Pipeline

• Sensitive data goes through the data soruces, ingestion and producers, stream storage, and the Stream processing and consumers, and downstream destinations of Amazon S3 and Amazon Redshift.
• AWS Activities that emit CloudWatch events go through the Cloudwatch events, Kinesis Data Streams, and Amazon Managed Service for Apache Flink.
• The data classification is maintained throughout the pipeline.

Key Takeaways Regarding Security of Analytics Workloads:

• Enforce the data classifications and protection policies that the owners have assigned to the source data.
• Protect access to the data within the analytics workload.
• Share data downstream in accordance with the classification policies of the source system.
• Ensure the environment is accessible with the least permissions necessary; automate auditing of environment changes, and alert in case of abnormal environment access.