Podcast Beta
Questions and Answers
What is the recommended number of participants for usability testing?
What are the benefits of remote, unmoderated usability testing?
What is experimental design in evaluation studies?
What is a null hypothesis in hypothesis testing?
Signup and view all the answers
What is matched-participant design in experimental design?
Signup and view all the answers
What are the primary methods for collecting data in field studies?
Signup and view all the answers
What is an in-the-wild study?
Signup and view all the answers
What is the Painpad?
Signup and view all the answers
What is the primary purpose of usability testing?
Signup and view all the answers
What is the recommended number of participants for usability testing?
Signup and view all the answers
What is remote, unmoderated usability testing?
Signup and view all the answers
What are some of the usability problems identified in the iPad usability testing case study?
Signup and view all the answers
What is experimental design?
Signup and view all the answers
What is a null hypothesis in hypothesis testing?
Signup and view all the answers
What is a same-participant design in experimental design?
Signup and view all the answers
What is the purpose of field studies?
Signup and view all the answers
What is the purpose of usability testing?
Signup and view all the answers
What is the recommended number of participants for usability testing?
Signup and view all the answers
What is remote, unmoderated usability testing?
Signup and view all the answers
What were some of the usability problems encountered during the iPad usability study?
Signup and view all the answers
What is experimental design in evaluation studies?
Signup and view all the answers
What is a null hypothesis in experimental design?
Signup and view all the answers
What is the difference between two-tailed and one-tailed hypotheses?
Signup and view all the answers
What is a field study in evaluation studies?
Signup and view all the answers
What is the purpose of in-the-wild studies?
Signup and view all the answers
What is Painpad and what was the purpose of the field study conducted with it?
Signup and view all the answers
What are the two types of data collected in the field study conducted with Painpad?
Signup and view all the answers
What is the purpose of long evaluations in field studies?
Signup and view all the answers
What is the traditional setting for usability testing?
Signup and view all the answers
What methods are commonly used to collect data in usability testing?
Signup and view all the answers
What is the recommended number of participants for usability testing?
Signup and view all the answers
What is remote, unmoderated usability testing?
Signup and view all the answers
What is experimental design?
Signup and view all the answers
What is a null hypothesis?
Signup and view all the answers
What is a two-tailed hypothesis?
Signup and view all the answers
What is the primary method of data collection in field studies?
Signup and view all the answers
What is an in-the-wild study?
Signup and view all the answers
What is Painpad?
Signup and view all the answers
What is the purpose of a field study?
Signup and view all the answers
What is a same-participant design?
Signup and view all the answers
What is the key concern when determining the number of users to involve in usability testing?
Signup and view all the answers
What is remote, unmoderated usability testing?
Signup and view all the answers
What did a case study of iPad usability testing recommend?
Signup and view all the answers
What is experimental design in evaluation studies?
Signup and view all the answers
What is the difference between two-tailed and one-tailed hypotheses?
Signup and view all the answers
What is the purpose of field studies in evaluation studies?
Signup and view all the answers
What is a matched-participant design in experimental design?
Signup and view all the answers
What is the most widely used statistical test in HCI and related fields?
Signup and view all the answers
What is an in-the-wild study in evaluation studies?
Signup and view all the answers
What is the purpose of a field study with Painpad?
Signup and view all the answers
What is the difference between different-participant and same-participant design in experimental design?
Signup and view all the answers
What is the primary method of collecting data in field studies?
Signup and view all the answers
What is the main difference between usability testing and experimental design?
Signup and view all the answers
What is the recommended number of participants for usability testing?
Signup and view all the answers
What is the advantage of remote, unmoderated usability testing?
Signup and view all the answers
What are the main types of data collected in a usability testing?
Signup and view all the answers
What is the main disadvantage of usability labs?
Signup and view all the answers
What is the purpose of experimental design?
Signup and view all the answers
What is the difference between same-participant design and different-participant design?
Signup and view all the answers
What is the main advantage of field studies?
Signup and view all the answers
What is the most widely used statistical test in HCI and related fields?
Signup and view all the answers
What is the purpose of hypotheses testing?
Signup and view all the answers
What is the main challenge of conducting field studies?
Signup and view all the answers
What is the main purpose of in-the-wild studies?
Signup and view all the answers
Study Notes
Evaluation Studies: Usability Testing, Experiments, and Field Studies
-
The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.
-
Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.
-
Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.
-
The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.
-
Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.
-
Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.
-
A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.
-
The study used two usability evaluation methods: usability testing with think-aloud and an expert review.
-
The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.
-
The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.
-
Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.
-
Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad
-
The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.
-
The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.
-
Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.
-
The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.
-
Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.
-
The findings were written up as a report, providing recommendations for app developers and the general public.
-
A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.
-
Hypotheses testing involves examining the relationship between independent and dependent variables.
-
A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.
-
Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.
-
Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.
-
User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies
-
Null hypothesis is used to reject a statement without necessarily supporting the opposite statement
-
Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings
-
Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex
-
Experimental design determines which participants to involve for which conditions in an experiment
-
Different-participant design randomly allocates a single group of participants to each of the experimental conditions
-
Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences
-
Matched-participant design matches participants in pairs based on certain user characteristics
-
T-tests are the most widely used statistical test in HCI and related fields
-
Field studies are conducted to evaluate user experiences in natural settings
-
Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled
-
Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos
-
Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings
-
Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.
-
In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.
-
In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.
-
In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.
-
Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.
-
Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.
-
The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.
-
Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.
-
Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.
-
User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.
-
Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.
-
Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.
Evaluation Studies: Usability Testing, Experiments, and Field Studies
-
The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.
-
Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.
-
Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.
-
The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.
-
Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.
-
Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.
-
A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.
-
The study used two usability evaluation methods: usability testing with think-aloud and an expert review.
-
The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.
-
The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.
-
Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.
-
Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad
-
The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.
-
The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.
-
Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.
-
The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.
-
Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.
-
The findings were written up as a report, providing recommendations for app developers and the general public.
-
A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.
-
Hypotheses testing involves examining the relationship between independent and dependent variables.
-
A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.
-
Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.
-
Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.
-
User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies
-
Null hypothesis is used to reject a statement without necessarily supporting the opposite statement
-
Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings
-
Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex
-
Experimental design determines which participants to involve for which conditions in an experiment
-
Different-participant design randomly allocates a single group of participants to each of the experimental conditions
-
Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences
-
Matched-participant design matches participants in pairs based on certain user characteristics
-
T-tests are the most widely used statistical test in HCI and related fields
-
Field studies are conducted to evaluate user experiences in natural settings
-
Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled
-
Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos
-
Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings
-
Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.
-
In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.
-
In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.
-
In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.
-
Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.
-
Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.
-
The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.
-
Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.
-
Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.
-
User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.
-
Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.
-
Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.
Evaluation Studies: Usability Testing, Experiments, and Field Studies
-
The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.
-
Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.
-
Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.
-
The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.
-
Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.
-
Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.
-
A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.
-
The study used two usability evaluation methods: usability testing with think-aloud and an expert review.
-
The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.
-
The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.
-
Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.
-
Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad
-
The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.
-
The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.
-
Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.
-
The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.
-
Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.
-
The findings were written up as a report, providing recommendations for app developers and the general public.
-
A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.
-
Hypotheses testing involves examining the relationship between independent and dependent variables.
-
A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.
-
Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.
-
Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.
-
User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies
-
Null hypothesis is used to reject a statement without necessarily supporting the opposite statement
-
Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings
-
Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex
-
Experimental design determines which participants to involve for which conditions in an experiment
-
Different-participant design randomly allocates a single group of participants to each of the experimental conditions
-
Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences
-
Matched-participant design matches participants in pairs based on certain user characteristics
-
T-tests are the most widely used statistical test in HCI and related fields
-
Field studies are conducted to evaluate user experiences in natural settings
-
Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled
-
Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos
-
Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings
-
Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.
-
In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.
-
In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.
-
In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.
-
Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.
-
Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.
-
The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.
-
Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.
-
Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.
-
User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.
-
Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.
-
Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.
Evaluation Studies: Usability Testing, Experiments, and Field Studies
-
The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.
-
Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.
-
Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.
-
The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.
-
Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.
-
Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.
-
A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.
-
The study used two usability evaluation methods: usability testing with think-aloud and an expert review.
-
The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.
-
The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.
-
Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.
-
Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad
-
The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.
-
The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.
-
Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.
-
The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.
-
Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.
-
The findings were written up as a report, providing recommendations for app developers and the general public.
-
A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.
-
Hypotheses testing involves examining the relationship between independent and dependent variables.
-
A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.
-
Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.
-
Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.
-
User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies
-
Null hypothesis is used to reject a statement without necessarily supporting the opposite statement
-
Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings
-
Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex
-
Experimental design determines which participants to involve for which conditions in an experiment
-
Different-participant design randomly allocates a single group of participants to each of the experimental conditions
-
Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences
-
Matched-participant design matches participants in pairs based on certain user characteristics
-
T-tests are the most widely used statistical test in HCI and related fields
-
Field studies are conducted to evaluate user experiences in natural settings
-
Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled
-
Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos
-
Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings
-
Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.
-
In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.
-
In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.
-
In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.
-
Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.
-
Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.
-
The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.
-
Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.
-
Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.
-
User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.
-
Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.
-
Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.
Evaluation Studies: Usability Testing, Experiments, and Field Studies
-
The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.
-
Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.
-
Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.
-
The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.
-
Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.
-
Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.
-
A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.
-
The study used two usability evaluation methods: usability testing with think-aloud and an expert review.
-
The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.
-
The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.
-
Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.
-
Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad
-
The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.
-
The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.
-
Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.
-
The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.
-
Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.
-
The findings were written up as a report, providing recommendations for app developers and the general public.
-
A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.
-
Hypotheses testing involves examining the relationship between independent and dependent variables.
-
A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.
-
Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.
-
Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.
-
User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies
-
Null hypothesis is used to reject a statement without necessarily supporting the opposite statement
-
Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings
-
Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex
-
Experimental design determines which participants to involve for which conditions in an experiment
-
Different-participant design randomly allocates a single group of participants to each of the experimental conditions
-
Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences
-
Matched-participant design matches participants in pairs based on certain user characteristics
-
T-tests are the most widely used statistical test in HCI and related fields
-
Field studies are conducted to evaluate user experiences in natural settings
-
Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled
-
Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos
-
Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings
-
Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.
-
In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.
-
In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.
-
In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.
-
Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.
-
Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.
-
The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.
-
Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.
-
Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.
-
User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.
-
Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.
-
Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.
Evaluation Studies: Usability Testing, Experiments, and Field Studies
-
The chapter discusses evaluation studies that take place in a range of settings, from controlled laboratories to natural settings, such as people's homes, schools, work, and leisure environments.
-
Usability testing is traditionally carried out in controlled laboratory settings to emphasize how usable a product is, and it involves testing whether the product being developed is usable by the intended user in order to achieve the tasks for which it was designed and whether users are satisfied with their experience.
-
Usability testing involves collecting data about users' performance on predefined tasks through a combination of methods, including video recordings, think-aloud techniques, user satisfaction questionnaires, and structured or semi-structured interviews.
-
The number of users involved in usability testing is a key concern, and early research suggests that 5 to 12 participants are an acceptable number, but more users can provide a broader selection of the target user population.
-
Usability labs can be expensive and labor-intensive to run and maintain, but less expensive and more versatile alternatives, such as mobile and remote usability testing equipment, are becoming popular.
-
Remote, unmoderated usability testing allows users to perform a set of tasks with a product in their own setting, and their interactions are logged remotely, enabling large numbers of participants to be tested in real-world settings.
-
A case study of testing the usability of the iPad illustrates how usability tests are carried out and the types of modifications that are made to accommodate real-world constraints, such as having a limited amount of time to evaluate the iPad as it came onto the market.
-
The study used two usability evaluation methods: usability testing with think-aloud and an expert review.
-
The study found that users had high expectations for the iPad's usability, and while they found it easy to use, they also encountered some usability problems, such as difficulty typing on the on-screen keyboard and difficulty interacting with some websites.
-
The study recommended that designers pay attention to users' expectations, design for touch interaction, and optimize websites for the iPad's screen size.
-
Experimental design is another evaluation method that involves manipulating independent variables to observe their effects on dependent variables, and it is typically carried out in research labs.
-
Field studies involve testing a product in a natural setting, such as people's homes, schools, work, and leisure environments, and they provide insights into how a product is used in the real world.Usability Testing for the iPad
-
The study aimed to understand typical usability issues when using applications and accessing websites on the iPad, and whether users preferred using apps over browsing the web.
-
The testing was conducted in two US cities with seven experienced iPhone users, with a range of ages and genders.
-
Participants were asked to explore any application they found interesting on the iPad and then open specific apps or websites to carry out tasks.
-
The testing equipment included a camera, webcam, and software called Morae to synchronize data streams.
-
Usability problems identified included small links and fonts, difficulty finding a back button, and getting lost in applications.
-
The findings were written up as a report, providing recommendations for app developers and the general public.
-
A second study was conducted a year later with 16 iPad users, revealing new problems such as boring splash screens and too much information on the screen.
-
Hypotheses testing involves examining the relationship between independent and dependent variables.
-
A null hypothesis and alternative hypothesis are derived, with the latter stating that there is a difference in the dependent variable depending on the independent variable.
-
Two-tailed hypotheses can be interpreted in two ways, while one-tailed hypotheses state one specific effect.
-
Multiple methods can reveal different usability problems, and it's important to adapt testing methods to fit the participants' culture and background.
-
User testing with prototypes can be integrated into the design process, as described in a report by the Nielsen Norman Group.Conducting Evaluation Studies: Experimental Design and Field Studies
-
Null hypothesis is used to reject a statement without necessarily supporting the opposite statement
-
Experimental design involves setting up conditions and finding ways to keep other variables constant to prevent them from influencing the findings
-
Hypothesis testing can be extended to include more variables, but it makes the experimental design more complex
-
Experimental design determines which participants to involve for which conditions in an experiment
-
Different-participant design randomly allocates a single group of participants to each of the experimental conditions
-
Same-participant design involves all participants performing in all conditions to lessen the impact of individual differences
-
Matched-participant design matches participants in pairs based on certain user characteristics
-
T-tests are the most widely used statistical test in HCI and related fields
-
Field studies are conducted to evaluate user experiences in natural settings
-
Field studies tend to be messy as activities often overlap and are constantly interrupted by events that are not predicted or controlled
-
Data for field studies is collected primarily by observing and interviewing people, and may include video, audio, field notes, and photos
-
Privacy and security arrangements need to be made for participants in field studies, especially if expensive or precious equipment is being evaluated in a public place.Field Studies: Understanding Technology Adoption in Natural Settings
-
Field studies explore how new technologies are adopted and integrated into participants' cultures and settings.
-
In-the-wild studies are increasingly popular, where researchers deploy and evaluate new technologies in natural settings.
-
In-the-wild studies reveal how users approach new technology, derive benefits, use it in everyday contexts, and sustain its use over time.
-
In-the-wild studies can uncover different results from lab studies, which may only uncover usability issues.
-
Painpad is a pain-monitoring tangible device for patients to report their pain levels, evaluated in a field study with patients recovering from ambulatory surgery in UK hospitals.
-
Painpad enables patients to monitor their own pain levels by pressing keys on the pad to record their pain rating.
-
The field study aimed to evaluate how robust and easy Painpad was to use in hospital environments and how it compared with nurse-collected pain ratings.
-
Two types of data were collected in the field study: pain ratings and user satisfaction questionnaires.
-
Painpad was well received and easy to use, with patients recording more pain scores than would typically be collected in the hospital by nurses.
-
User satisfaction questionnaires provided nuanced insights into patients' needs, likes, and dislikes.
-
Field studies can also evaluate how experts learn and interact with tools for complex tasks, typically in their own workplaces.
-
Long evaluations of how experts learn and interact with tools for complex tasks typically involve an initial interview, introductory training, novice usage, mature usage, and a semi-structured exit interview.
Studying That Suits You
Use AI to generate personalized quizzes and flashcards to suit your learning preferences.
Related Documents
Description
Test your knowledge on the different types of evaluation studies used in Human-Computer Interaction (HCI) research. From usability testing to experiments and field studies, this quiz will cover the key concepts, methods, and findings from various evaluation studies. Brush up on your understanding of null and alternative hypotheses, experimental design, data collection methods, and statistical tests commonly used in HCI research. Learn about the benefits and limitations of different evaluation studies, and how they can provide insights into user experiences with technology in controlled and natural
