Podcast
Questions and Answers
What distinguishes revenue from being a continuous variable?
What distinguishes revenue from being a continuous variable?
- It can only take integer values.
- It is always reported as a discrete count.
- It can take any value within a range. (correct)
- It represents the total sum of transactions.
How is conversion rate calculated?
How is conversion rate calculated?
- Total revenue divided by the number of transactions.
- Count of all transactions divided by pageviews.
- Number of successful transactions divided by total site visits.
- Count of all users who qualified for a condition divided by total users. (correct)
What type of variable are transactions considered?
What type of variable are transactions considered?
- Continuous and can take fractional orders.
- Neither continuous nor discrete.
- Only continuous since they can vary in dollar amount.
- Discrete since they can only take whole number values. (correct)
Which of the following is NOT a metric discussed for evaluating data patterns?
Which of the following is NOT a metric discussed for evaluating data patterns?
Which element is included in the dimensions and metrics within GA4 reporting?
Which element is included in the dimensions and metrics within GA4 reporting?
What is the effect of increasing the sample size on the standard deviation?
What is the effect of increasing the sample size on the standard deviation?
Which of the following represents a discrete metric?
Which of the following represents a discrete metric?
What is a key characteristic of continuous metrics?
What is a key characteristic of continuous metrics?
What is the consequence of the peaking problem in statistical reporting?
What is the consequence of the peaking problem in statistical reporting?
Which method should be considered to convert a continuous metric into a discrete one?
Which method should be considered to convert a continuous metric into a discrete one?
What does a p-value represent in hypothesis testing?
What does a p-value represent in hypothesis testing?
In which scenario would you typically apply a binomial metric?
In which scenario would you typically apply a binomial metric?
Which statement accurately describes a confidence interval?
Which statement accurately describes a confidence interval?
When should you conduct a one-tailed test?
When should you conduct a one-tailed test?
When analyzing conversion rates, what does a lift indicate?
When analyzing conversion rates, what does a lift indicate?
What is a Type I error in hypothesis testing?
What is a Type I error in hypothesis testing?
What does increasing the sample size help to reduce in statistical analysis?
What does increasing the sample size help to reduce in statistical analysis?
What is the relationship between Type I and Type II errors?
What is the relationship between Type I and Type II errors?
What is the best way to reduce the risk of Type I and Type II errors during experiments?
What is the best way to reduce the risk of Type I and Type II errors during experiments?
In the initial observation of an experiment, what could an unexpectedly high conversion rate suggest?
In the initial observation of an experiment, what could an unexpectedly high conversion rate suggest?
What is primarily the purpose of conducting two-tailed experiments?
What is primarily the purpose of conducting two-tailed experiments?
What characterizes the variation in Day 2 observation regarding conversions and conversion rates?
What characterizes the variation in Day 2 observation regarding conversions and conversion rates?
What was the conversion rate for the control group on Day 3?
What was the conversion rate for the control group on Day 3?
What conclusion can be drawn from the data collected on Day 4 about the variation?
What conclusion can be drawn from the data collected on Day 4 about the variation?
What effect did the results from Day 3 have on the perception of the change?
What effect did the results from Day 3 have on the perception of the change?
What is indicated by the 'Lift' percentage when comparing variation to control on Day 5?
What is indicated by the 'Lift' percentage when comparing variation to control on Day 5?
How does collecting more data impact the confidence interval?
How does collecting more data impact the confidence interval?
What does a conversion rate of 44.00% on Day 5 suggest about the variation’s performance?
What does a conversion rate of 44.00% on Day 5 suggest about the variation’s performance?
Flashcards
P-value
P-value
The probability of observing results as extreme as the ones obtained, assuming the null hypothesis is true. Essentially, it measures the strength of evidence against the null hypothesis.
Confidence Interval
Confidence Interval
A range around a measurement that indicates the precision of that measurement. It essentially represents the uncertainty associated with a statistical estimate.
One-tailed test
One-tailed test
A statistical test where the alternative hypothesis specifies a direction of change, either greater than or less than the null hypothesis.
Two-tailed test
Two-tailed test
Signup and view all the flashcards
Type I Error (False Positive)
Type I Error (False Positive)
Signup and view all the flashcards
Type II Error (False Negative)
Type II Error (False Negative)
Signup and view all the flashcards
How A/B Testing Avoids Errors
How A/B Testing Avoids Errors
Signup and view all the flashcards
Focus on Core Concepts
Focus on Core Concepts
Signup and view all the flashcards
Lift
Lift
Signup and view all the flashcards
Control Group
Control Group
Signup and view all the flashcards
Variation Group
Variation Group
Signup and view all the flashcards
Conversion Rate
Conversion Rate
Signup and view all the flashcards
Sample
Sample
Signup and view all the flashcards
Population
Population
Signup and view all the flashcards
Data Collection
Data Collection
Signup and view all the flashcards
A/B testing
A/B testing
Signup and view all the flashcards
Pageviews per session
Pageviews per session
Signup and view all the flashcards
Session duration
Session duration
Signup and view all the flashcards
Session
Session
Signup and view all the flashcards
Regression to the Mean
Regression to the Mean
Signup and view all the flashcards
Bayesian Statistical Method
Bayesian Statistical Method
Signup and view all the flashcards
Frequentist Statistical Method
Frequentist Statistical Method
Signup and view all the flashcards
Binomial (Discrete) Metric
Binomial (Discrete) Metric
Signup and view all the flashcards
Continuous Metric
Continuous Metric
Signup and view all the flashcards
Converting Continuous Metrics
Converting Continuous Metrics
Signup and view all the flashcards
Peaking Problem
Peaking Problem
Signup and view all the flashcards
Increasing Sample Size
Increasing Sample Size
Signup and view all the flashcards
Study Notes
Course Information
- Course name: BUS441 Web Analytics
- Week: 11
- Topic: Experiment Analysis & Interpretation
- Term: Fall 2024
- Institution: SFU
Disclaimer
- Instructor is not a statistician
Traditional Hypothesis Testing
- A 6-step "validate" phase for testing hypotheses
- Step 1: Formulate the null and alternative hypotheses (H₀ and H₁)
- Step 2: Select an appropriate statistical test.
- Step 3: Determine the significance level (α).
- Step 4: Collect data and calculate the test statistic.
- Step 5: Determine the critical value of the test statistic.
- Step 6: Compare the probability with the significance level.
- Step 7: Reject or do not reject the null hypothesis.
- Step 8: Draw a marketing research conclusion
P-values
- P-values represent the probability of obtaining results as extreme or more extreme than observed, given that the null hypothesis is true.
- Difficulty in explaining p-values intuitively
- Understanding involves nuance and complexity.
- Intuitive simplification loses details and can lead to misunderstandings.
Confidence Intervals
- A confidence interval is a range around a measurement which communicates its precision.
- 95% confidence interval: For 100 samples using the same plan, 95 of the confidence intervals will contain the true value (estimated from the sample)
- Wider confidence interval means reduced precision
One vs. Two-Tailed Tests
- One-tailed tests: Suitable for situations with a directional hunch.
- Two-tailed tests: More commonly used for determining if a difference exists without a prior directional assumption.
- Two-tailed tests are a better default.
Type I and Type II Errors
- Type I Error (False Positive): Declaring a variant a winner when it is not.
- Type II Error (False Negative): Declaring a winner as no better than the original alternative
- Errors inversely related - reducing one often increases the other.
Increasing Sample Size
- Crucial for reducing sampling errors.
- A larger sample size leads to a narrower confidence interval and greater precision.
- Insufficient sample size can lead to inaccurate conclusions.
- Statistical significance increases with greater sample amounts.
Bayesian vs Frequentist
- Frequentist: use a pre-set significance rate, longer tests with lower intuitive results.
- Bayesian: flexible testing times, higher intuitive results, but more complex calculations.
Combination Approach
- Stats Engine: statistical significance increases over time. Sequential tests become more confident with each visitor.
- Traditional statistics: peeking at results increases the chance of finding significant results which are not actually significant.
Continuous vs Discrete Metrics
- Continuous: infinite range of values. (e.g., revenue, transaction value)
- Discrete: defined set of possible outcomes (e.g., purchases, page visits)
- Continuous metrics can be converted to discrete metrics using thresholds.
Reports in GA4
- Real-time reporting
- Explore tab versus reports
- Dimensions & Metrics
- Segments
- Traffic from test
- Traffic from different sources
- Device types
- Click data
- Pageview data
- Data scopes in demo store
- Orders & pages
- Orders from homepage
- Unique orders
Weekly Assignment
- AB test results interpretation in Canvas
Up Next
- No class next week (Flex) week
- Final presentations in 2 weeks
- Final exam in ~4 weeks
Studying That Suits You
Use AI to generate personalized quizzes and flashcards to suit your learning preferences.