Evaluation Studies in HCI
64 Questions
0 Views

Choose a study mode

Play Quiz
Study Flashcards
Spaced Repetition
Chat to lesson

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What is the recommended number of participants for usability testing?

  • 15 to 20 participants
  • 1 to 3 participants
  • 5 to 12 participants (correct)
  • More than 20 participants

    • What are the benefits of remote, unmoderated usability testing?

  • Requires less expensive equipment
  • Allows for testing in a controlled laboratory setting
  • Enables large numbers of participants to be tested in real-world settings (correct)
  • Provides more accurate data than traditional usability testing

    • What is experimental design in evaluation studies?

  • Setting up conditions and finding ways to keep other variables constant
  • Testing a product in a natural setting
  • Manipulating independent variables to observe their effects on dependent variables (correct)
  • Collecting data about users' performance on predefined tasks

    • What is a null hypothesis in hypothesis testing?

    <p>A statement that is rejected without necessarily supporting the opposite statement</p> Signup and view all the answers

    What is matched-participant design in experimental design?

    <p>Matches participants in pairs based on certain user characteristics</p> Signup and view all the answers

    What are the primary methods for collecting data in field studies?

    <p>Observing and interviewing people, and may include video, audio, field notes, and photos</p> Signup and view all the answers

    What is an in-the-wild study?

    <p>A study that explores how new technologies are adopted and integrated into participants' cultures and settings</p> Signup and view all the answers

    What is the Painpad?

    <p>A tangible device for patients to report their pain levels</p> Signup and view all the answers

    What is the primary purpose of usability testing?

    <p>To test whether a product is usable by the intended user and whether users are satisfied with their experience</p> Signup and view all the answers

    What is the recommended number of participants for usability testing?

    <p>5 to 12</p> Signup and view all the answers

    What is remote, unmoderated usability testing?

    <p>Allowing users to perform a set of tasks with a product in their own setting, with their interactions logged remotely</p> Signup and view all the answers

    What are some of the usability problems identified in the iPad usability testing case study?

    <p>Difficulty typing on the on-screen keyboard and difficulty interacting with some websites</p> Signup and view all the answers

    What is experimental design?

    <p>Manipulating independent variables to observe their effects on dependent variables</p> Signup and view all the answers

    What is a null hypothesis in hypothesis testing?

    <p>Used to reject a statement without necessarily supporting the opposite statement</p> Signup and view all the answers

    What is a same-participant design in experimental design?

    <p>Involves all participants performing in all conditions to lessen the impact of individual differences</p> Signup and view all the answers

    What is the purpose of field studies?

    <p>To evaluate user experiences in natural settings</p> Signup and view all the answers

    What is the purpose of usability testing?

    <p>To evaluate how usable a product is by the intended user and whether users are satisfied with their experience</p> Signup and view all the answers

    What is the recommended number of participants for usability testing?

    <p>5 to 12 participants</p> Signup and view all the answers

    What is remote, unmoderated usability testing?

    <p>Allowing users to perform a set of tasks with a product in their own setting, with their interactions logged remotely</p> Signup and view all the answers

    What were some of the usability problems encountered during the iPad usability study?

    <p>Difficulty typing on the on-screen keyboard and difficulty interacting with some websites</p> Signup and view all the answers

    What is experimental design in evaluation studies?

    <p>Manipulating independent variables to observe their effects on dependent variables</p> Signup and view all the answers

    What is a null hypothesis in experimental design?

    <p>Used to reject a statement without necessarily supporting the opposite statement</p> Signup and view all the answers

    What is the difference between two-tailed and one-tailed hypotheses?

    <p>Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect</p> Signup and view all the answers

    What is a field study in evaluation studies?

    <p>Testing a product in a natural setting, such as people's homes, schools, work, and leisure environments</p> Signup and view all the answers

    What is the purpose of in-the-wild studies?

    <p>To deploy and evaluate new technologies in natural settings and reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time</p> Signup and view all the answers

    What is Painpad and what was the purpose of the field study conducted with it?

    <p>A pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals</p> Signup and view all the answers

    What are the two types of data collected in the field study conducted with Painpad?

    <p>Pain ratings and user satisfaction questionnaires</p> Signup and view all the answers

    What is the purpose of long evaluations in field studies?

    <p>To evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces</p> Signup and view all the answers

    What is the traditional setting for usability testing?

    <p>Controlled laboratory settings</p> Signup and view all the answers

    What methods are commonly used to collect data in usability testing?

    <p>Video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews</p> Signup and view all the answers

    What is the recommended number of participants for usability testing?

    <p>5 to 12</p> Signup and view all the answers

    What is remote, unmoderated usability testing?

    <p>Users perform a set of tasks with a product in their own setting, and their interactions are logged remotely</p> Signup and view all the answers

    What is experimental design?

    <p>Manipulating independent variables to observe their effects on dependent variables</p> Signup and view all the answers

    What is a null hypothesis?

    <p>Used to reject a statement without necessarily supporting the opposite statement</p> Signup and view all the answers

    What is a two-tailed hypothesis?

    <p>Can be interpreted in two ways</p> Signup and view all the answers

    What is the primary method of data collection in field studies?

    <p>Observing and interviewing people</p> Signup and view all the answers

    What is an in-the-wild study?

    <p>Deploying and evaluating new technologies in natural settings</p> Signup and view all the answers

    What is Painpad?

    <p>A pain-monitoring tangible device for patients to report their pain levels</p> Signup and view all the answers

    What is the purpose of a field study?

    <p>To evaluate how new technologies are adopted and integrated into participants' cultures and settings</p> Signup and view all the answers

    What is a same-participant design?

    <p>Involves all participants performing in all conditions to lessen the impact of individual differences</p> Signup and view all the answers

    What is the key concern when determining the number of users to involve in usability testing?

    <p>The acceptability of the sample size</p> Signup and view all the answers

    What is remote, unmoderated usability testing?

    <p>Usability testing carried out in participants' own settings</p> Signup and view all the answers

    What did a case study of iPad usability testing recommend?

    <p>Pay attention to users' expectations</p> Signup and view all the answers

    What is experimental design in evaluation studies?

    <p>Manipulating independent variables to observe their effects on dependent variables</p> Signup and view all the answers

    What is the difference between two-tailed and one-tailed hypotheses?

    <p>Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect</p> Signup and view all the answers

    What is the purpose of field studies in evaluation studies?

    <p>To evaluate user experiences in natural settings</p> Signup and view all the answers

    What is a matched-participant design in experimental design?

    <p>Matching participants in pairs based on certain user characteristics</p> Signup and view all the answers

    What is the most widely used statistical test in HCI and related fields?

    <p>T-tests</p> Signup and view all the answers

    What is an in-the-wild study in evaluation studies?

    <p>A field study that explores how new technologies are adopted and integrated into participants' cultures and settings</p> Signup and view all the answers

    What is the purpose of a field study with Painpad?

    <p>To evaluate how robust and easy Painpad was to use in hospital environments</p> Signup and view all the answers

    What is the difference between different-participant and same-participant design in experimental design?

    <p>Different-participant design involves all participants performing in all conditions to lessen the impact of individual differences, while same-participant design randomly allocates a single group of participants to each of the experimental conditions</p> Signup and view all the answers

    What is the primary method of collecting data in field studies?

    <p>Observation and interviewing people</p> Signup and view all the answers

    What is the main difference between usability testing and experimental design?

    <p>Usability testing involves testing whether the product being developed is usable by the intended user, while experimental design involves manipulating independent variables to observe their effects on dependent variables.</p> Signup and view all the answers

    What is the recommended number of participants for usability testing?

    <p>Between 5 and 12 participants</p> Signup and view all the answers

    What is the advantage of remote, unmoderated usability testing?

    <p>It allows for large numbers of participants to be tested in real-world settings</p> Signup and view all the answers

    What are the main types of data collected in a usability testing?

    <p>User satisfaction questionnaires and think-aloud techniques</p> Signup and view all the answers

    What is the main disadvantage of usability labs?

    <p>They are too expensive to run and maintain</p> Signup and view all the answers

    What is the purpose of experimental design?

    <p>To manipulate independent variables to observe their effects on dependent variables</p> Signup and view all the answers

    What is the difference between same-participant design and different-participant design?

    <p>Same-participant design involves matching participants in pairs, while different-participant design randomly allocates a single group of participants to each of the experimental conditions.</p> Signup and view all the answers

    What is the main advantage of field studies?

    <p>They allow for testing in natural settings</p> Signup and view all the answers

    What is the most widely used statistical test in HCI and related fields?

    <p>T-tests</p> Signup and view all the answers

    What is the purpose of hypotheses testing?

    <p>To examine the relationship between independent and dependent variables</p> Signup and view all the answers

    What is the main challenge of conducting field studies?

    <p>They tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled</p> Signup and view all the answers

    What is the main purpose of in-the-wild studies?

    <p>To deploy and evaluate new technologies in natural settings</p> Signup and view all the answers

    Study Notes

    Evaluation Studies: Usability Testing, Experiments, and Field Studies

    • The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.

    • Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.

    • Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.

    • The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.

    • Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.

    • Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.

    • A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.

    • The study used two usability evaluation methods: usability testing with think-aloud and an expert review.

    • The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.

    • The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.

    • Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.

    • Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad

    • The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.

    • The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.

    • Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.

    • The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.

    • Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.

    • The findings were written up as a report, providing recommendations for app developers and the general public.

    • A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.

    • Hypotheses testing involves examining the relationship between independent and dependent variables.

    • A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.

    • Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.

    • Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.

    • User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies

    • Null hypothesis is used to reject a statement without necessarily supporting the opposite statement

    • Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings

    • Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex

    • Experimental design determines which participants to involve for which conditions in an experiment

    • Different-participant design randomly allocates a single group of participants to each of the experimental conditions

    • Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences

    • Matched-participant design matches participants in pairs based on certain user characteristics

    • T-tests are the most widely used statistical test in HCI and related fields

    • Field studies are conducted to evaluate user experiences in natural settings

    • Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled

    • Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos

    • Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings

    • Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.

    • In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.

    • In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.

    • In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.

    • Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.

    • Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.

    • The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.

    • Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.

    • Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.

    • User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.

    • Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.

    • Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.

    Evaluation Studies: Usability Testing, Experiments, and Field Studies

    • The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.

    • Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.

    • Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.

    • The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.

    • Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.

    • Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.

    • A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.

    • The study used two usability evaluation methods: usability testing with think-aloud and an expert review.

    • The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.

    • The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.

    • Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.

    • Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad

    • The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.

    • The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.

    • Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.

    • The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.

    • Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.

    • The findings were written up as a report, providing recommendations for app developers and the general public.

    • A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.

    • Hypotheses testing involves examining the relationship between independent and dependent variables.

    • A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.

    • Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.

    • Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.

    • User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies

    • Null hypothesis is used to reject a statement without necessarily supporting the opposite statement

    • Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings

    • Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex

    • Experimental design determines which participants to involve for which conditions in an experiment

    • Different-participant design randomly allocates a single group of participants to each of the experimental conditions

    • Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences

    • Matched-participant design matches participants in pairs based on certain user characteristics

    • T-tests are the most widely used statistical test in HCI and related fields

    • Field studies are conducted to evaluate user experiences in natural settings

    • Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled

    • Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos

    • Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings

    • Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.

    • In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.

    • In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.

    • In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.

    • Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.

    • Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.

    • The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.

    • Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.

    • Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.

    • User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.

    • Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.

    • Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.

    Evaluation Studies: Usability Testing, Experiments, and Field Studies

    • The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.

    • Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.

    • Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.

    • The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.

    • Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.

    • Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.

    • A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.

    • The study used two usability evaluation methods: usability testing with think-aloud and an expert review.

    • The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.

    • The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.

    • Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.

    • Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad

    • The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.

    • The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.

    • Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.

    • The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.

    • Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.

    • The findings were written up as a report, providing recommendations for app developers and the general public.

    • A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.

    • Hypotheses testing involves examining the relationship between independent and dependent variables.

    • A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.

    • Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.

    • Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.

    • User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies

    • Null hypothesis is used to reject a statement without necessarily supporting the opposite statement

    • Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings

    • Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex

    • Experimental design determines which participants to involve for which conditions in an experiment

    • Different-participant design randomly allocates a single group of participants to each of the experimental conditions

    • Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences

    • Matched-participant design matches participants in pairs based on certain user characteristics

    • T-tests are the most widely used statistical test in HCI and related fields

    • Field studies are conducted to evaluate user experiences in natural settings

    • Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled

    • Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos

    • Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings

    • Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.

    • In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.

    • In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.

    • In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.

    • Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.

    • Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.

    • The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.

    • Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.

    • Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.

    • User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.

    • Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.

    • Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.

    Evaluation Studies: Usability Testing, Experiments, and Field Studies

    • The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.

    • Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.

    • Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.

    • The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.

    • Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.

    • Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.

    • A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.

    • The study used two usability evaluation methods: usability testing with think-aloud and an expert review.

    • The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.

    • The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.

    • Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.

    • Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad

    • The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.

    • The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.

    • Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.

    • The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.

    • Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.

    • The findings were written up as a report, providing recommendations for app developers and the general public.

    • A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.

    • Hypotheses testing involves examining the relationship between independent and dependent variables.

    • A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.

    • Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.

    • Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.

    • User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies

    • Null hypothesis is used to reject a statement without necessarily supporting the opposite statement

    • Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings

    • Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex

    • Experimental design determines which participants to involve for which conditions in an experiment

    • Different-participant design randomly allocates a single group of participants to each of the experimental conditions

    • Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences

    • Matched-participant design matches participants in pairs based on certain user characteristics

    • T-tests are the most widely used statistical test in HCI and related fields

    • Field studies are conducted to evaluate user experiences in natural settings

    • Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled

    • Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos

    • Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings

    • Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.

    • In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.

    • In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.

    • In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.

    • Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.

    • Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.

    • The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.

    • Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.

    • Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.

    • User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.

    • Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.

    • Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.

    Evaluation Studies: Usability Testing, Experiments, and Field Studies

    • The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.

    • Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.

    • Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.

    • The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.

    • Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.

    • Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.

    • A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.

    • The study used two usability evaluation methods: usability testing with think-aloud and an expert review.

    • The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.

    • The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.

    • Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.

    • Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad

    • The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.

    • The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.

    • Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.

    • The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.

    • Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.

    • The findings were written up as a report, providing recommendations for app developers and the general public.

    • A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.

    • Hypotheses testing involves examining the relationship between independent and dependent variables.

    • A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.

    • Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.

    • Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.

    • User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies

    • Null hypothesis is used to reject a statement without necessarily supporting the opposite statement

    • Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings

    • Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex

    • Experimental design determines which participants to involve for which conditions in an experiment

    • Different-participant design randomly allocates a single group of participants to each of the experimental conditions

    • Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences

    • Matched-participant design matches participants in pairs based on certain user characteristics

    • T-tests are the most widely used statistical test in HCI and related fields

    • Field studies are conducted to evaluate user experiences in natural settings

    • Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled

    • Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos

    • Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings

    • Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.

    • In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.

    • In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.

    • In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.

    • Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.

    • Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.

    • The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.

    • Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.

    • Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.

    • User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.

    • Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.

    • Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.

    Evaluation Studies: Usability Testing, Experiments, and Field Studies

    • The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.

    • Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.

    • Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.

    • The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.

    • Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.

    • Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.

    • A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.

    • The study used two usability evaluation methods: usability testing with think-aloud and an expert review.

    • The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.

    • The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.

    • Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.

    • Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad

    • The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.

    • The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.

    • Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.

    • The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.

    • Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.

    • The findings were written up as a report, providing recommendations for app developers and the general public.

    • A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.

    • Hypotheses testing involves examining the relationship between independent and dependent variables.

    • A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.

    • Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.

    • Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.

    • User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies

    • Null hypothesis is used to reject a statement without necessarily supporting the opposite statement

    • Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings

    • Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex

    • Experimental design determines which participants to involve for which conditions in an experiment

    • Different-participant design randomly allocates a single group of participants to each of the experimental conditions

    • Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences

    • Matched-participant design matches participants in pairs based on certain user characteristics

    • T-tests are the most widely used statistical test in HCI and related fields

    • Field studies are conducted to evaluate user experiences in natural settings

    • Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled

    • Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos

    • Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings

    • Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.

    • In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.

    • In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.

    • In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.

    • Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.

    • Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.

    • The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.

    • Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.

    • Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.

    • User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.

    • Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.

    • Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.

    Studying That Suits You

    Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

    Quiz Team

    Related Documents

    chap15.pdf

    Description

    Test your knowledge on the different types of evaluation studies used in Human-Computer Interaction (HCI) research. From usability testing to experiments and field studies, this quiz will cover the key concepts, methods, and findings from various evaluation studies. Brush up on your understanding of null and alternative hypotheses, experimental design, data collection methods, and statistical tests commonly used in HCI research. Learn about the benefits and limitations of different evaluation studies, and how they can provide insights into user experiences with technology in controlled and natural

    More Like This

    Use Quizgecko on...
    Browser
    Open
    Browser
    Browser