Glue

• Glue is a service used for transform, extract, and load (ETL) jobs

Glue Cost

Crawlers

• Crawlers are billed hourly based on the number of Data Processing Units (DPUs) used, with charges calculated by the second and a 10-minute minimum.

Data Catalog

• The Glue Data Catalog offers up to one million objects for free. Additional objects are charged at $1 per 100,000 objects over the million object limit per month.

ETL Jobs

• ETL jobs are billed hourly based on the number of DPUs, with charges calculated by the second and a 10-minute minimum, though versions 2.0 and later have a 1-minute minimum.

Data Processing Units (DPUs)

• A single DPU provides 4 vCPUs and 16 GB of memory.

Number of DPUs used

• Apache Spark jobs need a minimum of 2 DPUs and default to 10.
• Spark Streaming requires a minimum of 2 DPUs and defaults to 10.
• Ray jobs (ML/AI) need a minimum of 2 MDPUs (higher memory) and default to 6.

DPU Cost

• DPUs cost $0.44 per DPU hour, although this can vary by region.

Glue Notebooks/Interactive Sessions

• Glue notebooks are used for interactively developing ELT code.
• Billing is based on the session's active time and the number of DPUs used.
• Configurable idle timeouts are available.
• There is a 1-minute minimum billing period.
• A minimum of 2 DPUs are required, with a default of 5.

ETL Job Example

• An Apache Spark job running for 15 minutes and using 6 DPUs costs $0.44 per DPU hour.

Stateful vs. Stateless

• Stateful systems remember past interactions, influencing future ones. For example, a crawler will reload everything when new files are added to a bucket
• Stateless systems process each request independently, without relying on past interactions.

Data Ingestion in AWS

• Amazon Kinesis supports both stateful and stateless data processing.
• AWS Data Pipeline orchestrates workflows for both stateful and stateless data ingestion.
• Glue offers ETL jobs with features like job bookmarks for tracking progress.

Glue - Transform, Extract, and Load

Extract

• RDS, Aurora, DynamoDB
• Redshift
• S3, Kinesis

Transform

• Filtering: removing unnecessary data
• Joining: combining data
• Aggregation: summarizing data
• Find Matches (ML): identifying records that refer to the same entity (duplicates)
• Detect PII (Personal Identity Information): identifying and managing sensitive information

Data Format Hierarchy

• CSV > PARQUET > JSON > XML

Glue Workflows

• Used to orchestrate multi-step data processing jobs, managing executions and monitoring of jobs/crawlers.
• Good for managing AWS Glue operations.
• Provides a visual interface.
• Workflows can be created manually.

Triggers

• Initiate jobs and crawlers. -Scheduled triggers: start the workflow at regular intervals. -On-demand triggers: starts manually the workflow from the AWS console. -Event Bridge triggers: launches the workflow based on specific events captured by EventBridge.

Glue Job Types

• Spark ETL jobs: large-scale data processing (2 DPU to 100 DPU).
• Spark Streaming ETL jobs: analyze data in real time (2 DPU to 100 DPU).
• Python Shell jobs: suitable for lightweight tasks (0.06 to 1 DPU).
• Ray jobs: suitable for parallel processing tasks.

Execution Types

• Standard: Designed for predictable ETL jobs which start immediately and guarantees consistent job execution times.
• Flex execution: A cost-effective option for less time-sensitive ETL jobs that may start with some delay.

Glue Partitioning

• Glue automatically partitions data if it's properly organized.
• Improves query performance and reduces I/O operations.
• Glue can skip over large segments within partitioned data and process each partition independently (parallel processing).
• Provides cost efficiency by reducing query efforts.
• Partitioning can be defined as part of the ETL job script or within the Glue Data Catalog.

Glue DataBrew

• A data preparation tool with a visual interface for cleaning and formatting data
• Offers 250+ pre-built transformations and no-code data preparation capabilities.
• Automates and schedules data preparations.

Integration

• Amazon S3 -> AWS Glue DataBrew -> Amazon Redshift

DataBrew Components

• Project: Used to configure transformation tasks.
• Step: An applied transformation to the dataset.
• Recipe: A set of transformation steps that can be saved and reused.
• Job: The execution of a recipe on a dataset that outputs to locations such as S3
• Schedule: Schedules jobs to automate transformation.
• Data profiling: Used to understand the quality and characteristics of your data.

Transformations

• Nest to Map: Converts columns into a map.
• Nest to Array: Converts columns into an array.
• Nest to Struct: Similar to Nest to Map but retains exact data and order.
• Unnest Array: Converts an array to columns.
• Pivot: Pivots columns and values to rotate data from rows into columns.
• Unpivot: Converts columns into rows.
• Transpose: Switches columns and rows.
• Additionally, supports join, split, filter, sort, and date/time conversions, count distinct

Cost

• Interactive sessions cost $1 per session.
• DataBrew jobs cost $0.48 per node-hour.