Evaluation Terms Quiz

Defining Evaluation Terms and Other Considerations

• Data analytics is used to draw inferences from large volumes of raw data, while web analytics is used to measure website traffic through analyzing users’ click data.
• Analytical evaluation models and predicts user behavior, and can refer to heuristic evaluation, walk-throughs, modeling, and analytics.
• Controlled experiments are used to test hypotheses about an interface or other dimension, with aspects such as tasks, time, and environment controlled.
• Crowdsourcing involves hundreds, thousands, or even millions of people evaluating a product or taking part in an experiment, either in person or online.
• Ecological validity refers to how the environment in which an evaluation is conducted influences or even distorts the results.
• Expert review or crit involves someone with usability expertise and knowledge of the user population reviewing a product for potential problems.
• Formative evaluation is done during design to check that the product fulfills requirements and continues to meet users’ needs.
• Heuristic evaluation involves applying knowledge of typical users, often guided by heuristics, to identify usability problems.
• Informed consent forms describe the study and participants' rights, and are mandatory in many universities and major organizations.
• Reliability refers to how well a method produces the same results on separate occasions under the same circumstances, while validity concerns whether the evaluation method measures what it is intended to measure.
• Ecological validity concerns how the environment in which an evaluation is conducted influences or even distorts the results.
• Bias occurs when the results are distorted, and can happen when expert evaluators, researchers, or interviewers selectively gather data or influence responses.

Data Analysis, Interpretation, and Presentation

• Qualitative and quantitative data and analysis are different.

• A combination of qualitative and quantitative approaches is common in data analysis.

• Initial reactions or observations from the data involve identifying patterns or calculating simple numerical values such as ratios, averages, or percentages.

• Data cleansing is important to check for any anomalies that might be erroneous.

• Interpretation of the findings often proceeds in parallel with analysis, but it is important to make sure that the data supports any conclusions.

• Making claims that go beyond what the data can support is a common mistake in data analysis, interpretation, and presentation.

• Finding the best way to present findings depends on the goals and audience for whom the study was performed.

• Quantitative data is in the form of numbers, while qualitative data is in the form of words and images.

• All forms of data gathering may result in qualitative and quantitative data.

• Quantitative analysis uses numerical methods, while qualitative analysis focuses on the nature of something and can be represented by themes, patterns, and stories.

• Initial processing of data involves transcription, expansion of notes, entry of answers to close-ended questions into a spreadsheet, clean up data, filter into different data sets, synchronization between data recordings, and records of behavior.

• Basic quantitative analysis techniques include averages and percentages, and there are three types of average: mean, median, and mode.Basic Data Analysis Techniques

• Data analysis is the process of examining data to extract meaningful insights and conclusions.

• There are two basic approaches to data analysis: quantitative analysis and qualitative analysis.

• Quantitative analysis involves numerical data and statistical methods, while qualitative analysis involves non-numerical data and subjective interpretation.

• Quantitative data can be analyzed using spreadsheet software like Excel or Google Sheets, while qualitative data requires manual analysis and interpretation.

• The design of survey questions affects the kind of analyses that can be performed and the conclusions that can be drawn.

• Close-ended questions with fixed answer options can be analyzed quantitatively, while open-ended questions require qualitative analysis.

• Basic quantitative analysis involves finding averages, identifying outliers, and producing graphical representations of the data.

• Data visualization tools like heatmaps can create more sophisticated representations of the data.

• Outliers are usually removed from the main data set because they distort general patterns, but they may also be interesting cases to investigate further.

• Basic qualitative analysis involves identifying themes, categorizing data, and analyzing critical incidents.

• Critical incident analysis is a way to isolate subsets of data for more detailed analysis.

• The first step in qualitative analysis is to gain an overall impression of the data and start looking for interesting features, topics, repeated observations, or things that stand out.Basic Qualitative Data Analysis Techniques

• Qualitative data can be analyzed inductively or deductively, depending on the data obtained and the goal of the study.

• The objective of qualitative data analysis is to classify elements of the data in order to gain insights toward the study’s goal.

• Identifying themes (thematic analysis) takes an inductive approach, while categorizing data takes a deductive approach.

• Analysis is often performed iteratively, and it is common for themes identified inductively then to be applied deductively to new data.

• Reliability of the analysis can be achieved by training another person to do the coding and calculating the inter-rater reliability.

• Using more sophisticated analytical frameworks to structure the analysis of qualitative data can lead to additional insights that go beyond the results of basic techniques.

• Thematic analysis is a widely used analytical technique that aims to identify, analyze, and report patterns in the data.

• After an initial pass through the data, the next step is to look more systematically for themes across participants’ transcripts, seeking further evidence both to confirm and disconfirm initial impressions in all of the data.

• An overall narrative is starting to emerge from the set of themes, and some of the original themes may not seem as relevant and can be removed.

• Affinity diagrams are a common technique for exploring data, identifying themes, and looking for an overall narrative.

• Inductive analysis is appropriate when the study is exploratory, and it is important to let the themes emerge from the data itself.

• Categorization is appropriate when the analysis frame is chosen beforehand, based on the study goal.Analytical Frameworks for Qualitative Data Analysis

• Usability studies can use think-aloud protocols to identify usability problems and categorize them for analysis.

• Critical incident analysis helps to identify significant subsets of data for more detailed analysis by focusing on incidents that are significant or pivotal.

• Six different analytical frameworks can be used for qualitative data analysis, ordered by granularity.

• Conversation analysis focuses on recordings of spoken conversations to provide insights into how conversations are managed and progress.

• Discourse analysis focuses on recordings of speech or writing to understand how words are used to convey meaning.

• Content analysis examines any form of text to determine the frequency of items appearing in a text at varying levels.

• Interaction analysis uses video recordings to analyze verbal and non-verbal interactions between people and artifacts to understand how knowledge and action are used within an activity.

• Grounded theory constructs a theory around the phenomenon of interest using empirical data of any kind.

• Systems-based frameworks focus on large-scale and heterogeneous data involving people and technology to provide insights about organizational effectiveness and efficiency.

• Tuomas Kari et al. used critical incident analysis to identify types of behavior change induced in players of the game Pokémon GO.

• Elise Grison et al. used critical incident analysis to investigate factors influencing travelers' choices of transport mode in Paris.

• In a task of identifying the next available theater performance or movie to attend, critical incidents may include searching social media, discovering a favorite movie is playing, and needing a credit card to purchase tickets.