Podcast
Questions and Answers
Which of the following is a method used to validate measurements?
Which of the following is a method used to validate measurements?
Experimental design is not a valid technique for validating system performance measurements.
Experimental design is not a valid technique for validating system performance measurements.
False
What is the purpose of a 'back of the envelope' calculation?
What is the purpose of a 'back of the envelope' calculation?
To estimate system performance within an order of magnitude.
A common way to check the performance of a system is by using __________.
A common way to check the performance of a system is by using __________.
Signup and view all the answers
Match the following types of performance validation techniques with their descriptions:
Match the following types of performance validation techniques with their descriptions:
Signup and view all the answers
What is the primary purpose of benchmarking in data engineering systems?
What is the primary purpose of benchmarking in data engineering systems?
Signup and view all the answers
Measurement and benchmarking are critical primarily due to the simplicity of systems and their components.
Measurement and benchmarking are critical primarily due to the simplicity of systems and their components.
Signup and view all the answers
What is the effect of low performance on consumer behavior as noted in the content?
What is the effect of low performance on consumer behavior as noted in the content?
Signup and view all the answers
According to Prof. Rabl's recipe, the first step in writing a research paper is to conduct a literature __________.
According to Prof. Rabl's recipe, the first step in writing a research paper is to conduct a literature __________.
Signup and view all the answers
Match the components of system measurement with their descriptions:
Match the components of system measurement with their descriptions:
Signup and view all the answers
Which of the following steps is NOT part of Prof. Rabl's 7 Step Paper/Thesis Recipe?
Which of the following steps is NOT part of Prof. Rabl's 7 Step Paper/Thesis Recipe?
Signup and view all the answers
Consumer page load time expectations have remained constant from 1999 to 2018.
Consumer page load time expectations have remained constant from 1999 to 2018.
Signup and view all the answers
What is BotEC in the context of Prof. Rabl's recipe?
What is BotEC in the context of Prof. Rabl's recipe?
Signup and view all the answers
What kind of benchmarks evaluate the impact of variable-sized records on system performance?
What kind of benchmarks evaluate the impact of variable-sized records on system performance?
Signup and view all the answers
Fixed-sized records consistently achieve higher throughput than variable-sized records.
Fixed-sized records consistently achieve higher throughput than variable-sized records.
Signup and view all the answers
What does Viper utilize for efficient record retrieval?
What does Viper utilize for efficient record retrieval?
Signup and view all the answers
The benchmark involved preloading _____ million records, each with a normal distribution.
The benchmark involved preloading _____ million records, each with a normal distribution.
Signup and view all the answers
Which of the following best describes the behavior of Viper in terms of operations?
Which of the following best describes the behavior of Viper in terms of operations?
Signup and view all the answers
Match the type of record with its characteristic:
Match the type of record with its characteristic:
Signup and view all the answers
In-place updates achieve higher throughput compared to copy-on-write updates.
In-place updates achieve higher throughput compared to copy-on-write updates.
Signup and view all the answers
What performance metric is significantly lower for variable-sized records compared to fixed-sized records?
What performance metric is significantly lower for variable-sized records compared to fixed-sized records?
Signup and view all the answers
Study Notes
Big Data Systems - Benchmarking & Measurement
- Big data systems are becoming more complex
- Single transactions can span across many components/nodes
- Consumers expect faster page load times
- Waiting time and outages cost money.
- Big data systems require performance benchmarking and measurement.
Lecture Topics
- Introduction to performance analysis,
- Back of the envelope calculations,
- Measurement,
- Benchmarks, and
- BigBench.
Where Levels of Measurement Fit In
- Measurement is required on all levels
- Concrete level depends on the research question.
- The level of analysis depends on the research questions being asked.
Why Measurement and Benchmarking
- Systems are increasingly complex
- Single transactions span across many components/nodes
- Consumer expectations for page load time are decreasing
- 1999 – 8 sec
- 2009 – 2 sec
- 2018 – 3 sec -> 50% consumers leave the page
- Poor performance and outages cost money.
Prof. Rabl's 7-Step Paper/Thesis Recipe
- Literature search
- Identify a research problem
- Describe a novel solution
- Perform BotEC to show potential
- Conduct experiments to prove feasibility
- Write the paper
- Manage and handle revisions
Benchmark vs. Analysis
- Analysis: focuses on single systems/algorithms, individual optimizations, and micro-benchmarks
- Benchmark: focuses on comparing multiple systems, using standard or real workloads
- A comprehensive study benefits from both analysis and benchmarking.
Understanding System Performance
- Modeling: involves back-of-the-envelope calculations and analytical models
- Measurement: requires experimental design and use of benchmarks
- Simulation: uses emulation and trace-driven methods
- Validation is crucial; one technique's results should be confirmed by another one.
Back of the Envelope Calculation
- Used to estimate system performance to quickly assess feasibility.
How to Get Good (Enough) Performance
- Understand the application
- Perform back-of-the-envelope calculations
- Estimate system performance
- Filter out impractical ideas early
- Benchmark to get a definitive answer
Useful Latency Numbers
- Various latencies and bandwidths for different operations are presented
- Examples: L1 cache reference, branch mispredict, etc.
- Different technologies and devices have different latencies.
Basic Considerations
- Determine if the data size is big enough to be considered big data
- Data that fits in memory is not big data
- Data size affects the performance of tasks such as finding maximum/minimum/average elements in a list.
Simple BotEC Example
- Calculate time to generate image results page (with 30 thumbnails)
- Consider serial reading and parallel reading scenarios
- Key issues include caching strategies and pre-computation of thumbnails
Sorting Example
- Calculating time to sort 1GB of 4-byte numbers
- Discusses concepts like quicksort, memory bandwidth, and time complexity.
Complete Sorting Program (code example)
- Provided C++ code to sort a large dataset and measures the execution time
Results (from the experiment)
- Summary of sorting results: total duration (time taken) and output of profiler data is shown
Measurement & Metrics
- Metrics used to evaluate different aspects of the system, such as throughput, latency, capacity, fault tolerance, efficiency, cost, and scalability.
Basic Terminology
- Data Generator Driver: System under test, deployment, workloads, requests by users, and metrics for evaluating performance
- Workload: requests from users to evaluate system behavior
- System Under Test (SuT): system being tested
- Benchmark Tooling: Tools used to perform benchmarking.
- Metrics: Measurements collected during the benchmark
- Measurements: Values recorded during the experiment
Questions to Be Answered Beforehand
- Identify the scenario for evaluation and the data to use
- Choose the hardware and software to be used
- Determine criteria for evaluating performance
Common Metrics
- Performance: Throughput, Latency, Accuracy, Capacity
- Fault-tolerance: Time to failure, Availability
- Efficiency: Energy, Cost, Fairness
- Scalability: Important when considering efficiency
Throughput / Latency
- Metrics for evaluating system performance (throughput and latency, including 95th/99th percentile latency)
Capacity
- Describes maximum achievable throughput under ideal conditions
- Response time is high under ideal conditions
Usable Capacity
- Achievable throughput without exceeding pre-specified response limits
- Sustainable throughput
Knee Capacity
- Low response time and high throughput
Bottlenecks
- Potential obstacles in evaluating fast systems, including driver issues, network saturation, time vs processing time, and coordinated omissions
Real Application Examples
- Take an application, implement it on the test system, and assess its performance
- Pros: Real-world scenarios, challenges
- Cons: Proprietary datasets, large workloads, scalability
Comparing to Other Systems/Work
- Ensure that other systems/work reproduce or equal results to ensure there are no issues.
BigBench / TPCx-BB
- Big data benchmark (end-to-end benchmark for parallel DBMS and MR engines)
BigBench Data Model
- Structured: (e.g., TPC-DS)
- Semi-structured: (e.g., website click-stream)
- Unstructured: (e.g., customer review)
Workload
- Business functions (e.g., marketing, operations, merchandising)
- Queries covering all business functions.
Query 1
- Query to find frequently sold products together (in the provided SQL/query format)
Benchmark Process – TPCx-BB
- Benchmarking process steps and the processes involved
Summary
- Summary of introduction to performance analysis, back of envelope calculation, measurement, and benchmarks, and BigBench
Questions
- In Moodle
- Per email [email address]
- In Q&A sessions
Studying That Suits You
Use AI to generate personalized quizzes and flashcards to suit your learning preferences.
Related Documents
Description
Explore the critical aspects of performance benchmarking and measurement in big data systems. This quiz covers essential topics, including measurement techniques and the impact of system performance on consumer expectations. Improve your understanding of how complexity in big data requires effective analysis and benchmarks.
