BUS441 Web Analytics Week 11 Quiz

Choose a study mode

Play Quiz
Study Flashcards
Spaced Repetition
Chat to Lesson

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What distinguishes revenue from being a continuous variable?

  • It can only take integer values.
  • It is always reported as a discrete count.
  • It can take any value within a range. (correct)
  • It represents the total sum of transactions.

How is conversion rate calculated?

  • Total revenue divided by the number of transactions.
  • Count of all transactions divided by pageviews.
  • Number of successful transactions divided by total site visits.
  • Count of all users who qualified for a condition divided by total users. (correct)

What type of variable are transactions considered?

  • Continuous and can take fractional orders.
  • Neither continuous nor discrete.
  • Only continuous since they can vary in dollar amount.
  • Discrete since they can only take whole number values. (correct)

Which of the following is NOT a metric discussed for evaluating data patterns?

<p>Count of total pageviews. (C)</p> Signup and view all the answers

Which element is included in the dimensions and metrics within GA4 reporting?

<p>Traffic segments such as device types. (B)</p> Signup and view all the answers

What is the effect of increasing the sample size on the standard deviation?

<p>It causes the standard deviation to shrink. (A)</p> Signup and view all the answers

Which of the following represents a discrete metric?

<p>Total number of transactions. (D)</p> Signup and view all the answers

What is a key characteristic of continuous metrics?

<p>They can take on an infinite range of values. (B)</p> Signup and view all the answers

What is the consequence of the peaking problem in statistical reporting?

<p>It leads to overconfidence in results. (B)</p> Signup and view all the answers

Which method should be considered to convert a continuous metric into a discrete one?

<p>Count occurrences above or below specific thresholds. (B)</p> Signup and view all the answers

What does a p-value represent in hypothesis testing?

<p>The probability of observing results as extreme as the ones you got, assuming the null hypothesis is true (D)</p> Signup and view all the answers

In which scenario would you typically apply a binomial metric?

<p>Documenting whether a user clicked a button. (D)</p> Signup and view all the answers

Which statement accurately describes a confidence interval?

<p>It is a range that indicates how precise a measurement is (C)</p> Signup and view all the answers

When should you conduct a one-tailed test?

<p>When you expect a specific direction of the effect (B)</p> Signup and view all the answers

When analyzing conversion rates, what does a lift indicate?

<p>An improvement in performance compared to a control group. (A)</p> Signup and view all the answers

What is a Type I error in hypothesis testing?

<p>Declaring the variant a winner when it is not (A)</p> Signup and view all the answers

What does increasing the sample size help to reduce in statistical analysis?

<p>Variability in results. (B)</p> Signup and view all the answers

What is the relationship between Type I and Type II errors?

<p>They are inversely related (A)</p> Signup and view all the answers

What is the best way to reduce the risk of Type I and Type II errors during experiments?

<p>Increase the sample size (C)</p> Signup and view all the answers

In the initial observation of an experiment, what could an unexpectedly high conversion rate suggest?

<p>The results may be a false positive and further testing is needed (A)</p> Signup and view all the answers

What is primarily the purpose of conducting two-tailed experiments?

<p>To assess whether any significant difference exists (D)</p> Signup and view all the answers

What characterizes the variation in Day 2 observation regarding conversions and conversion rates?

<p>Equal conversion rates between control and variation (A)</p> Signup and view all the answers

What was the conversion rate for the control group on Day 3?

<p>53.33% (C)</p> Signup and view all the answers

What conclusion can be drawn from the data collected on Day 4 about the variation?

<p>The variation's conversion rate was lower than the control group's. (A)</p> Signup and view all the answers

What effect did the results from Day 3 have on the perception of the change?

<p>Frustration, suggesting immediate test termination (D)</p> Signup and view all the answers

What is indicated by the 'Lift' percentage when comparing variation to control on Day 5?

<p>The variation performed worse but is still acceptable. (D)</p> Signup and view all the answers

How does collecting more data impact the confidence interval?

<p>It narrows the confidence interval. (A)</p> Signup and view all the answers

What does a conversion rate of 44.00% on Day 5 suggest about the variation’s performance?

<p>It indicates failure compared to higher past rates. (B)</p> Signup and view all the answers

Flashcards

P-value

The probability of observing results as extreme as the ones obtained, assuming the null hypothesis is true. Essentially, it measures the strength of evidence against the null hypothesis.

Confidence Interval

A range around a measurement that indicates the precision of that measurement. It essentially represents the uncertainty associated with a statistical estimate.

One-tailed test

A statistical test where the alternative hypothesis specifies a direction of change, either greater than or less than the null hypothesis.

Two-tailed test

A statistical test where the alternative hypothesis states that there is a difference, but it doesn't specify a particular direction.

Signup and view all the flashcards

Type I Error (False Positive)

The error of rejecting the null hypothesis when it's actually true. This means wrongly concluding there's a significant effect when there isn't.

Signup and view all the flashcards

Type II Error (False Negative)

The error of failing to reject the null hypothesis when it's actually false. This means missing a significant difference when there actually is one.

Signup and view all the flashcards

How A/B Testing Avoids Errors

Increasing the sample size in an experiment can reduce both types of errors.

Signup and view all the flashcards

Focus on Core Concepts

It's important to focus on the core concepts of web analytics rather than just the technical details.

Signup and view all the flashcards

Lift

The difference in performance between a control group and a variation group. A positive lift indicates improvement, while a negative lift indicates a decline.

Signup and view all the flashcards

Control Group

An experimental group that doesn't receive the treatment being tested. It's used as a baseline for comparison.

Signup and view all the flashcards

Variation Group

An experimental group that receives the treatment being tested.

Signup and view all the flashcards

Conversion Rate

The proportion of users who take a desired action, such as making a purchase or signing up for a newsletter.

Signup and view all the flashcards

Sample

A group of individuals or data points chosen to represent a larger population.

Signup and view all the flashcards

Population

The entire group of individuals or data points that you are interested in studying.

Signup and view all the flashcards

Data Collection

The process of collecting data over a period of time to observe changes and trends.

Signup and view all the flashcards

A/B testing

A statistical test to compare two versions of a webpage or feature to determine which performs better. It involves randomly assigning users to different versions and measuring their performance.

Signup and view all the flashcards

Pageviews per session

A measure of how frequently a user interacts with a webpage, often expressed as the average number of pageviews per session.

Signup and view all the flashcards

Session duration

The period of time a user spends on a website from the moment they enter to the moment they leave, often measured in seconds or minutes.

Signup and view all the flashcards

Session

A period of continuous activity on a website by a single user, typically defined by a set time limit of inactivity or a specific action, like clicking a link or navigating to a new page.

Signup and view all the flashcards

Regression to the Mean

The tendency for estimates of a population mean to become more accurate as the size of the sample data increases.

Signup and view all the flashcards

Bayesian Statistical Method

A statistical technique used to estimate the true average of a population based on sample data. It incorporates prior beliefs about the average into the calculation.

Signup and view all the flashcards

Frequentist Statistical Method

A statistical technique that relies solely on observed data to make inferences about a population. It does not use prior beliefs and is based on probability calculations.

Signup and view all the flashcards

Binomial (Discrete) Metric

A metric that has only two possible values, typically represented as 1 or 0 (e.g., 'Yes' or 'No', 'Click' or 'No Click').

Signup and view all the flashcards

Continuous Metric

A metric that can take on any value within a continuous range (e.g., order value, time spent on site).

Signup and view all the flashcards

Converting Continuous Metrics

The process of converting a continuous metric into a discrete metric by defining a threshold and categorizing data points above or below that threshold.

Signup and view all the flashcards

Peaking Problem

The problem of overestimating the true effect of a change when a small sample size is used.

Signup and view all the flashcards

Increasing Sample Size

Increasing the size of the sample data helps reduce the impact of random variability and provides a more accurate estimate of the population mean.

Signup and view all the flashcards

Study Notes

Course Information

  • Course name: BUS441 Web Analytics
  • Week: 11
  • Topic: Experiment Analysis & Interpretation
  • Term: Fall 2024
  • Institution: SFU

Disclaimer

  • Instructor is not a statistician

Traditional Hypothesis Testing

  • A 6-step "validate" phase for testing hypotheses
  • Step 1: Formulate the null and alternative hypotheses (H₀ and H₁)
  • Step 2: Select an appropriate statistical test.
  • Step 3: Determine the significance level (α).
  • Step 4: Collect data and calculate the test statistic.
  • Step 5: Determine the critical value of the test statistic.
  • Step 6: Compare the probability with the significance level.
  • Step 7: Reject or do not reject the null hypothesis.
  • Step 8: Draw a marketing research conclusion

P-values

  • P-values represent the probability of obtaining results as extreme or more extreme than observed, given that the null hypothesis is true.
  • Difficulty in explaining p-values intuitively
  • Understanding involves nuance and complexity.
  • Intuitive simplification loses details and can lead to misunderstandings.

Confidence Intervals

  • A confidence interval is a range around a measurement which communicates its precision.
  • 95% confidence interval: For 100 samples using the same plan, 95 of the confidence intervals will contain the true value (estimated from the sample)
  • Wider confidence interval means reduced precision

One vs. Two-Tailed Tests

  • One-tailed tests: Suitable for situations with a directional hunch.
  • Two-tailed tests: More commonly used for determining if a difference exists without a prior directional assumption.
  • Two-tailed tests are a better default.

Type I and Type II Errors

  • Type I Error (False Positive): Declaring a variant a winner when it is not.
  • Type II Error (False Negative): Declaring a winner as no better than the original alternative
  • Errors inversely related - reducing one often increases the other.

Increasing Sample Size

  • Crucial for reducing sampling errors.
  • A larger sample size leads to a narrower confidence interval and greater precision.
  • Insufficient sample size can lead to inaccurate conclusions.
  • Statistical significance increases with greater sample amounts.

Bayesian vs Frequentist

  • Frequentist: use a pre-set significance rate, longer tests with lower intuitive results.
  • Bayesian: flexible testing times, higher intuitive results, but more complex calculations.

Combination Approach

  • Stats Engine: statistical significance increases over time. Sequential tests become more confident with each visitor.
  • Traditional statistics: peeking at results increases the chance of finding significant results which are not actually significant.

Continuous vs Discrete Metrics

  • Continuous: infinite range of values. (e.g., revenue, transaction value)
  • Discrete: defined set of possible outcomes (e.g., purchases, page visits)
  • Continuous metrics can be converted to discrete metrics using thresholds.

Reports in GA4

  • Real-time reporting
  • Explore tab versus reports
  • Dimensions & Metrics
  • Segments
  • Traffic from test
  • Traffic from different sources
  • Device types
  • Click data
  • Pageview data
  • Data scopes in demo store
  • Orders & pages
  • Orders from homepage
  • Unique orders

Weekly Assignment

  • AB test results interpretation in Canvas

Up Next

  • No class next week (Flex) week
  • Final presentations in 2 weeks
  • Final exam in ~4 weeks

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Quiz Team

Related Documents

More Like This

Experimental Design and ANOVA Quiz
13 questions

Experimental Design and ANOVA Quiz

RazorSharpForesight1304 avatar
RazorSharpForesight1304
Experimental Design Concepts Quiz
5 questions
Chapter 3. Experimental psychology
50 questions
Use Quizgecko on...
Browser
Browser