Big Data Systems Benchmarking

Big Data Systems - Benchmarking & Measurement

• Big data systems are becoming more complex
• Single transactions can span across many components/nodes
• Consumers expect faster page load times
• Waiting time and outages cost money.
• Big data systems require performance benchmarking and measurement.

Lecture Topics

• Introduction to performance analysis,
• Back of the envelope calculations,
• Measurement,
• Benchmarks, and
• BigBench.

Where Levels of Measurement Fit In

• Measurement is required on all levels
• Concrete level depends on the research question.
• The level of analysis depends on the research questions being asked.

Why Measurement and Benchmarking

• Systems are increasingly complex
• Single transactions span across many components/nodes
• Consumer expectations for page load time are decreasing
• 1999 – 8 sec
• 2009 – 2 sec
• 2018 – 3 sec -> 50% consumers leave the page
• Poor performance and outages cost money.

Prof. Rabl's 7-Step Paper/Thesis Recipe

• Literature search
• Identify a research problem
• Describe a novel solution
• Perform BotEC to show potential
• Conduct experiments to prove feasibility
• Write the paper
• Manage and handle revisions

Benchmark vs. Analysis

• Analysis: focuses on single systems/algorithms, individual optimizations, and micro-benchmarks
• Benchmark: focuses on comparing multiple systems, using standard or real workloads
• A comprehensive study benefits from both analysis and benchmarking.

Understanding System Performance

• Modeling: involves back-of-the-envelope calculations and analytical models
• Measurement: requires experimental design and use of benchmarks
• Simulation: uses emulation and trace-driven methods
• Validation is crucial; one technique's results should be confirmed by another one.

Back of the Envelope Calculation

• Used to estimate system performance to quickly assess feasibility.

How to Get Good (Enough) Performance

• Understand the application
• Perform back-of-the-envelope calculations
• Estimate system performance
• Filter out impractical ideas early
• Benchmark to get a definitive answer

Useful Latency Numbers

• Various latencies and bandwidths for different operations are presented
• Examples: L1 cache reference, branch mispredict, etc.
• Different technologies and devices have different latencies.

Basic Considerations

• Determine if the data size is big enough to be considered big data
• Data that fits in memory is not big data
• Data size affects the performance of tasks such as finding maximum/minimum/average elements in a list.

Simple BotEC Example

• Calculate time to generate image results page (with 30 thumbnails)
• Consider serial reading and parallel reading scenarios
• Key issues include caching strategies and pre-computation of thumbnails

Sorting Example

• Calculating time to sort 1GB of 4-byte numbers
• Discusses concepts like quicksort, memory bandwidth, and time complexity.

Complete Sorting Program (code example)

• Provided C++ code to sort a large dataset and measures the execution time

Results (from the experiment)

• Summary of sorting results: total duration (time taken) and output of profiler data is shown

Measurement & Metrics

• Metrics used to evaluate different aspects of the system, such as throughput, latency, capacity, fault tolerance, efficiency, cost, and scalability.

Basic Terminology

• Data Generator Driver: System under test, deployment, workloads, requests by users, and metrics for evaluating performance
• Workload: requests from users to evaluate system behavior
• System Under Test (SuT): system being tested
• Benchmark Tooling: Tools used to perform benchmarking.
• Metrics: Measurements collected during the benchmark
• Measurements: Values recorded during the experiment

Questions to Be Answered Beforehand

• Identify the scenario for evaluation and the data to use
• Choose the hardware and software to be used
• Determine criteria for evaluating performance

Common Metrics

• Performance: Throughput, Latency, Accuracy, Capacity
• Fault-tolerance: Time to failure, Availability
• Efficiency: Energy, Cost, Fairness
• Scalability: Important when considering efficiency

Throughput / Latency

• Metrics for evaluating system performance (throughput and latency, including 95th/99th percentile latency)

Capacity

• Describes maximum achievable throughput under ideal conditions
• Response time is high under ideal conditions

Usable Capacity

• Achievable throughput without exceeding pre-specified response limits
• Sustainable throughput

Knee Capacity

• Low response time and high throughput

Bottlenecks

• Potential obstacles in evaluating fast systems, including driver issues, network saturation, time vs processing time, and coordinated omissions

Real Application Examples

• Take an application, implement it on the test system, and assess its performance
• Pros: Real-world scenarios, challenges
• Cons: Proprietary datasets, large workloads, scalability

Comparing to Other Systems/Work

• Ensure that other systems/work reproduce or equal results to ensure there are no issues.

BigBench / TPCx-BB

• Big data benchmark (end-to-end benchmark for parallel DBMS and MR engines)

BigBench Data Model

• Structured: (e.g., TPC-DS)
• Semi-structured: (e.g., website click-stream)
• Unstructured: (e.g., customer review)

Workload

• Business functions (e.g., marketing, operations, merchandising)
• Queries covering all business functions.

Query 1

• Query to find frequently sold products together (in the provided SQL/query format)

Benchmark Process – TPCx-BB

• Benchmarking process steps and the processes involved

Summary

• Summary of introduction to performance analysis, back of envelope calculation, measurement, and benchmarks, and BigBench

Questions

• In Moodle
• Per email [email address]
• In Q&A sessions

Explore the critical aspects of performance benchmarking and measurement in big data systems. This quiz covers essential topics, including measurement techniques and the impact of system performance on consumer expectations. Improve your understanding of how complexity in big data requires effective analysis and benchmarks.