AWS Academy: Analyzing and Visualizing Data

• Analyzing and Visualizing Data with AWS Academy Data Engineering

Module Objectives

• List factors for analysis and visualization tools
• Compare AWS tools and services for data analysis and visualization
• Determine appropriate AWS tools and services to analyze and visualize data based on influencing factors like business needs, data characteristics and access to data.

Factors Influencing Tool Selection

• The simplified data pipeline includes data sources, ingestion, storage, processing, analysis, and visualization.
• Understand the business needs to determine necessary analyses and visualizations for insight development.
• Assess data type and quality, plus update frequency to determine data characteristics
• Data characteristics are the nature and format of data (structured, unstructured, semi-structured)
• Data characteristics inform selection criteria (volume, velocity, variety, veracity, and value)
• Data velocity is the speed that data is updated
• Data veracity is the accuracy and reliability of data
• Data value is the utility and importance of the data
• Data volume is the amount of data produced
• Consider the data pipeline stage and required data access levels.
• Evaluate roles of who needs access to the data, and the access required (ie data analysts)
• Granularity of insight refers to the level of detail needed (from detailed to high-level aggregate)

Factors Influencing Tool Selection Details

• Determine needed analyses to develop insights for business needs
• Determine insights pulled from data for business needs
• Determine visualizations that illustrate the insights for business needs
• Determine if consumers need to generate reports or interact with dashboards for business needs
• Granularity of insight demanded depends on role
• Consider the required visualizations for business needs, like KPIs, relationships, comparisons, distributions, and compositions.

Industry Specific Granularity of Insight

• Finance: finance managers want details like revenue, costs, and profit margins for each line of business; CFOs want similar at an aggregate level across all lines of business
• Marketing: marketing managers want leads, opportunities, and closed deals in each area (e.g., postal code or city); CMOs are interested in the same metrics at state or regional level
• Sales: sales managers focus on sales pipeline and time to close an opportunity; VPs of sale wants similar data at an aggregate level

Data Characteristics

• Consider volume, speed, and update frequency of the data.

Examples of Data Types

• Volume and velocity: historical analysis and streaming IoT data.
• Variety and veracity: structured and unstructured data.
• Value: periodic reports and self-service dashboards.

Two Fraud Detection Use Cases

• Rule Based (Batch Pipeline): Data volume is kilobytes to terabytes, arriving in predefined intervals, processed in minutes to hours; provides historical reporting of fraud (reactive approach)
• ML in Real-Time (Streaming Pipeline): Data volume is bytes to megabytes, processing time is milliseconds to seconds providing ability to detect fraud in real time (proactive approach)

Data Access Considerations

• Understand the data source, combined data necessity, and access requirements.
• Consider authorization based on roles (least privilege)
• Structure data access based on various roles (e.g., Amazon Athena for data analysts, Amazon QuickSight for domain experts).

AWS Services in Data Pipeline

• Store raw data: Amazon Simple Storage Service (Amazon S3).
• Collect data: Amazon Kinesis Data Streams.
• Migrate data: AWS DataSync, AWS DMS, Amazon AppFlow, AWS Glue.
• Store aggregated data: Amazon S3, Amazon Redshift, Amazon RDS, Amazon OpenSearch Service, Amazon SageMaker Data Wrangler.
• Process data: Amazon Managed Service for Apache Flink, Amazon EMR, AWS Glue, SageMaker Data Wrangler.
• Visualize and query: Athena, QuickSight, and OpenSearch Service.

Amazon Athena

• An interactive query service that uses SQL to analyze data in Amazon S3.
• Features: serverless, data combining, one-time queries, BI tool compatibility, and Apache Iceberg integration.

One-time Querying for data in Amazon S3

• Data sources run through Amazon S3, EMR, and Redshift prior to using services like Amazon Athena and Amazon QuickSight

Capability to Update Data in Amazon S3

• Inserts, updates, and deletes stored data (with Apache Iceberg)
• Tracks data versions automatically
• Enables continuous ingestion and updates via Apache Iceberg.

Amazon QuickSight

• Cloud-scale BI that delivers easy to understand insights.
• Connects to cloud data and combines sources for data, giving decision makers interactive experience
• Consists of opportunity to explore and interpret information in an interactive visual environment
• includes forecasting visualization capabilities
• natural language questions can be asked using QuickSight Q.

QuickSight Examples

• To visualize sentiments, phrases, and tweets for a specific topic, use donut charts, word clouds, heatmaps, and tabular views.

Ask Questions Using Natural Language

• Use QuickSight Q to get immediate responses instead of waiting in the Bl queue
• It is a BI capability powered by ML, which processes natural language
• No need to build pre-defined data models or dashboards.

Amazon OpenSearch Service

• Deploys, operates, and scales OpenSearch clusters in AWS Cloud.
• Used for log analytics, application monitoring, and clickstream analytics.
• Integrates with OpenSearch Dashboards and Kibana.

OpenSearch Dashboard Example

• Visualize support calls after using services to get full transcripts of calls, keywords from the transcripts, and an overall sentiment of each call.
• Pie charts/bar charts/histograms can be used to analyze and search for the data

Comparison of AWS Services

• Athena: interactive SQL analysis, direct data analysis, instant queries, and serverless.
• QuickSight: dashboards, visualizations, and serverless.
• OpenSearch Service: operational analytics, real-time data visualization, and fully managed service.

Factors for AWS Tool Selection

• Selecting appropriate tools involves assessing business needs, data characteristics, and access requirements.

Gaming Analytics Use Case

• A gaming analytics use case influences selection of analysis tools, AWS tools and services (Athena, QuickSight, OpenSearch), and selecting tools for a gaming analytics use case (Solutions based on the use case and personas)

Three Key Personas and their Tool Use in a Gaming Company

• Analyst: explores and analyzes player data.
• Business user: showcases and reports results to leadership.
• DevOps engineer: monitors and analyzes real-time performance.

Key Factors/Takeaways

• Use cases illustrate granularity of insights
• Daily batch aggregates of client usage patterns (Athena).
• Consolidated aggregate KPIs for leadership (QuickSight).
• Continuous health and performance monitoring (OpenSearch Service).