AWS Academy: Analyzing and Visualizing Data

• Analyzing and Visualizing Data: AWS Academy Data Engineering

Module Objectives

• List factors to consider when selecting analysis and visualization tools
• Compare available AWS tools and services for data analysis and visualization
• Ability to determine the appropriate AWS tools and services to analyze and visualize data based on influencing factors: business needs, data characteristics, and access to data

Factors Influencing Tool Selection

• The simplified data pipeline is Ingestion -> Storage -> Processing -> Analysis & Visualization
• Factors to consider when selecting tools involves business needs, data characteristics, and access to data

Business Needs

• Understand business needs to determine which data analyses and visualizations are needed to help develop insights
• Consider analyses needed to develop insights, insights can be pulled from the data, visualizations illustrate the insights, and whether consumers need to generate a report or interact with a dashboard
• Industry needs detailed level data in finance, marketing, and sales
• Finance managers require revenue, costs, and profit margins about their line of business
• Marketing managers need the number of leads, opportunities, and closed deals within an area, such as a postal code or city
• Sales managers need to know how long it takes to close an opportunity and many opportunities needed to achieve quota targets
• Industry needs aggregate level data in finance, marketing and sales
• CFO’s require similar metrics at an aggregate level across all lines of businesses; the ability to drill down to any line of business
• CMO’s are interested in related metrics, such as a state or region
• VP’s of sales require similar information at an aggregate level, with the ability to drill down to a sales representative or sales territory
• KPIs can show performance in a particular area or function
• Relationships can establish or prove whether a relationship exists between two or more variables
• Comparisons show or examine how different variables change over time or provide a static snapshot of how different variables compare
• Distributions show how data is distributed over certain intervals, which are based on clustering of data
• Compositions highlight elements that make up your data

Data Characteristics

• Consider the amount of data
• Consider the data speed and volume
• Consider how frequently data is updated
• Consider how quickly the data is processed
• Consider the type of data
• Historical analysis can visualize a year's worth of sales data where users can drill down by region and salesperson
• Streaming Internet of Things (IoT) data can visualize the real-time error rates of sensors in a factory
• Structured data involves querying a relational database to report on customer service tickets submitted in a period
• Unstructured data involves sentiment analysis being performed on customer service emails
• Business analysts can use periodic reports to showcase and report results to leadership
• DevOps engineers use self-service dashboards to monitor and analyze performance in real-time

Data Characteristics - Use Cases

• Rule-Based Batch Pipeline: millions of transactions (kilobytes to terabytes), with data arriving in predefined intervals (minutes to multiple days)
• Rule-Based Batch Pipeline: data is processed in minutes to hours and it is structured and semi-structured, delivering insights from historical reporting of fraud cases with a reactive approach to fraud detection
• ML in Real Time: millions of transactions (bytes to megabytes) arriving in real time (milliseconds to seconds)
• ML in Real Time: data is processed in milliseconds to seconds, unstructured and semi-structured, with the ability to detect fraud with a proactive approach to fraud detection

Data Access

• Consider where the data comes from
• Consider whether data needs to be combined from multiple sources
• Consider who needs data access and at what level, or who can access the tools
• A user's authorization to access data depends on their role in the organization
• Business analysts and managers might be authorized to read the output that data engineers or data analysts create, but not delete or update it
• Follow the least privilege principle; give users the least amount of access and responsibility needed to complete their duties

Access, Functions, and Tools

• Data sources can be used by data analysts, data engineers or domain experts using tools like Amazon Athena, to perform data discovery or SQL data query
• Data sources can be used by data analysts or data scientists with Amazon QuickSight to create visualizations

Selection Factors Key Takeaways

• When selecting analysis and visualization tools, consider business needs, data characteristics, and access to data
• Consider the granularity and format of the insights based on business needs
• Consider the volume, velocity, variety, veracity, and value of your data
• Consider the functions of individuals who will access, analyze, and visualize the data

Amazon Web Services Tools

• AWS services in the data pipeline: ingestion, storage, processing, analysis and visualization
• Amazon Simple Storage Service (Amazon S3) is used to store raw data
• Amazon Kinesis Data Streams and Amazon Data Firehose are used to collect data
• AWS DataSync, AWS Database Migration Service (AWS DMS), Amazon AppFlow, and AWS Glue are used to migrate data
• Amazon S3, Amazon Redshift, Amazon Relational Database Service, Amazon OpenSearch Service, and Amazon SageMaker Data Wrangler are used to store aggregated data
• Amazon Managed Service for Apache Flink, Amazon EMR, AWS Glue, and SageMaker Data Wrangler are used to process data
• Athena, QuickSight and OpenSearch Service are used to visualize and query data

Amazon Athena

• Is an interactive query service that uses SQL to analyze data in Amazon S3, and is serverless
• Combines data from multiple data sources and can be used for one-time queries
• Can be used from your favorite business intelligence (BI) tools, such as QuickSight
• Can update data stored in Amazon S3 with Apache Iceberg integration
• Users can use Athena to insert, update, and delete data that is stored in Amazon S3, with the Apache Iceberg integration
• Can track data versions automatically, so Apache Iceberg integration provides a way for continuous ingestion and updates

Amazon QuickSight

• A cloud-scale BI service delivers easy-to-understand insights
• Connects to data in the cloud and combines data from many different sources
• Gives decision-makers the opportunity to explore and interpret information in an interactive visual environment
• Provides forecasting visualization capabilities
• Provides the ability to ask questions using natural language with QuickSight Q
• QuickSight helps visualize the sentiments, phrases, and tweets for a specific topic in QuickSight with donut charts, word clouds of phrases, heat maps of tweets, and tabular views of tweets
• QuickSight Q provides immediate responses when a visualization isn't already in the dashboard
• QuickSight Q is powered by ML, uses natural language processing, and doesn't require building pre-defined data models or dashboards

Amazon OpenSearch Service

• Managed service deploying, operating, and scaling OpenSearch clusters in the AWS Cloud
• Uses open-source search and analytics engine for use cases, such as log analytics, real-time application monitoring, and clickstream analytics
• Amazon OpenSearch Service is integrated with visualization tools, including OpenSearch Dashboards and Kibana
• OpenSearch Dashboards can let you analyze and visualize support calls, pull keywords, and identify sentiments with Amazon S3, Amazon Transcribe, and Amazon Comprehend
• Use OpenSearch Service and OpenSearch Dashboards to search and visualize the data through pie charts (positive, negative, neutral), bar charts of keywords, and a histogram of when and how often the calls were made

AWS Comparison

• Athena: Interactive analysis using SQL, analyzing data directly, starting query data instantly, and serverless
• QuickSight: Dashboards and visualizations, building visualizations and dashboards for business analytics, and serverless
• OpenSearch Service: Operational analytics, searching, exploring, filtering, aggregating, and visualizing data in near real time, and fully managed service

AWS Tools Key Takeaways

• AWS tools and services that are commonly used to query and visualize data: Athena, QuickSight, and OpenSearch Service
• Athena is used for interactive analysis with SQL
• Decision-makers can use QuickSight to interact with data visually and get insight quickly
• OpenSearch Service is used for operational analytics to visualize data in near real time

Gaming Analytics - Applying what was Learned

• Influence selection of factors of analysis and visualization tools like business needs, data characteristics, and access to data
• AWS tools and services: Athena, QuickSight, and OpenSearch Service
• Select using solutions based on a particular use case or personas in the use case
• Three personas in gaming analytics: analysts, business, and DevOps
• Analysts explore and analyze player data
• Business users showcase and report results to leadership
• DevOps engineers monitor and analyze performance in real time

Gaming Analytics Pipeline

• Game clients, servers and backend create data producers
• APIs create solutions for events stream & configuration data
• Amazon Managed Service for Apache Flink, Lambda, integrate data streaming
• Data Firehose and Lambda integrate streaming ingestion
• Data lake integration and ETL with Amazon S3 and AWS Glue
• Metrics include CloudWatch
• For interactive analytics, use Athena, QuickSight, and OpenSearch Service
• Data consumers utilize interactive analytics

Example Gaming Use Cases

• Athena: generate insights by querying daily aggregates of player usage data from batch data for financial and geographical insights that are stored to retrieve player purchase history, play history, and geo info
• QuickSight: visualize KPIs, such as average revenue per user or per paying user, retention and conversion rates and used for forecasting with data combined from multiple sources and aggregated to a high granularity level, retrieve player purchase, play and geo info
• OpenSearch Service: monitor health and performance, and analyze performance for predictive load balancing by processing large volumes of streaming telemetry and server logs, including structured and unstructured data with retrieve access logs and performance data for game servers
• This use case showcased the granularity of visualized insights like daily batch aggregates of client usage patterns, consolidated aggregate KPIs for leadership, as well as continuous health and performance monitoring
• Keep in mind the influencing factors you select the use AWS tools and services with multiple solutions that meet the business needs of data analysis and visualization