INFO 3307 Lecture 10_11 - Evaluations Analysis and Interpretations.pdf

Lecture 10 and 11: Understanding Evaluations, AnalysisPeople and Interpretations INFO 3307 HUMAN COMPUTER INTERACTION  Understanding Lecture 10: People Gathering Data INFO 3307 HUMAN COMPUTER INTERACTION  Aims Di...

Lecture 10 and 11: Understanding Evaluations, AnalysisPeople and Interpretations INFO 3307 HUMAN COMPUTER INTERACTION  Understanding Lecture 10: People Gathering Data INFO 3307 HUMAN COMPUTER INTERACTION  Aims Discuss how to plan and run a successful data gathering program. Enable you to plan and run an interview. Enable you to design a simple questionnaire. Enable you to plan and carry out an observation. www.id-book.com 3  Five key issues 1. Setting goals Decide how to analyze data once collected 2. Identifying participants Decide who to gather data from 3. Relationship with participants Clear and professional Informed consent when appropriate 4. Triangulation Look at data from more than one perspective Collect more than one type of data, e.g., qualitative from experiments and qualitative from interviews 5. Pilot studies Small trial of main study www.id-book.com 4  Data Recording Notes, audio, video, photographs can be used individually or in combination: Notes plus photographs Audio plus photographs Video Different challenges and advantages with each combination 5 www.id-book.com  Interviews Unstructured - are not directed by a script. Rich but not replicable. Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness. Reliable Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability. Focus groups – a group interview www.id-book.com 6  Interview questions Two types: ‘closed questions’ have a predetermined answer format, e.g.. ‘yes’ or ‘no’ ‘open questions’ do not have a predetermined format Closed questions are easier to analyze Avoid: Long questions Compound sentences - split them into two Jargon and language that the interviewee may not understand Leading questions that make assumptions e.g.. why do you like …? Unconscious biases e.g.. gender stereotypes www.id-book.com 7  Running the interview Introduction – introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present the informed consent form. Warm-up – make first questions easy and non-threatening. Main body – present questions in a logical order A cool-off period – include a few easy questions to defuse tension at the end Closure – thank interviewee, signal the end, eg. switch recorder off. www.id-book.com 8  Enriching the interview process Props - devices for prompting interviewee, e.g. use a prototype, scenario www.id-book.com 9  Questionnaires Questions can be closed or open Closed questions are easier to analyze, and may be distributed and analyzed by computer Can be administered to large populations Disseminated by paper, email and the web Sampling can be a problem when the size of a population is unknown as is common online evaluation www.id-book.com 10  Questionnaire design The impact of a question can be influenced by question order. You may need different versions of the questionnaire for different populations. Provide clear instructions on how to complete the questionnaire. Strike a balance between using white space and keeping the questionnaire compact. Avoid very long questionnaires Decide on whether phrases will all be positive, all negative or mixed. www.id-book.com 11  Question and response format ‘Yes’ and ‘No’ checkboxes Checkboxes that offer many options Rating scales Likert scales semantic scales (unlikely to buy, slightly to buy, definitely to buy) 3, 5, 7 or more points Open-ended responses www.id-book.com 12  Encouraging a good response Make sure purpose of study is clear Promise anonymity Ensure questionnaire is well designed Offer a short version for those who do not have time to complete a long questionnaire If mailed, include a stamped addressed envelope Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is good, 20% is often acceptable www.id-book.com 13  Advantages of online questionnaires Relatively easy and quick to distribute Responses are usually received quickly No copying and postage costs Data can be collected in database for analysis Time required for data analysis is reduced Errors can be corrected easily www.id-book.com 14 Example of an online questionnaire www.id-book.com 15  Problems with online questionnaires Sampling is problematic if population size is unknown Preventing individuals from responding more than once can be a problem Individuals have also been known to change questions in email questionnaires www.id-book.com 16  Observation Direct observation in the field Structuring frameworks Degree of participation (insider or outsider) Direct observation in controlled environments Indirect observation: tracking users’ activities Diaries Interaction logging Video and photographs collected remotely by drones or other equipment www.id-book.com 17 Observation 18 www.id-book.com  Structuring frameworks to guide observation Three easy-to-remember parts: The person: Who? The place: Where? The thing: What? A more detailed framework (Robson, 2014): Space: What is the physical space like and how is it laid out? Actors: What are the names and relevant details of the people involved? Activities: What are the actors doing and why? Objects: What physical objects are present, such as furniture Acts: What are specific individual actions? Events: Is what you observe part of a special event? Time: What is the sequence of events? Goals: What are the actors trying to accomplish? Feelings: What is the mood of the group and of individuals? www.id-book.com 19  Planning and conducting observation in the field Decide on how involved you will be: passive observer to active participant How to gain acceptance How to handle sensitive topics, eg. culture, private spaces, etc. How to collect the data: What data to collect What equipment to use When to stop observing www.id-book.com 20  Observations and materials that might be collected (Crabtree, 2007) Activity or job descriptions. Rules and procedures that govern particular activities. Descriptions of activities observed. Recordings of the talk taking place between parties. Informal interviews with participants explaining the detail of observed activities. Diagrams of the physical layout, including the position of artifacts. Other information collected when observing activities: Photographs of artifacts (documents, diagrams, forms, computers, etc.) Videos of artifacts. Descriptions of artifacts. Workflow diagrams showing the sequential order of tasks. Process maps showing connections between activities. www.id-book.com 21  Observation in a controlled environment Direct observation Think aloud techniques Indirect observation – tracking users’ activities Diaries Interaction logs Web analytics Video, audio, photos, notes are used to capture data in both types of observations www.id-book.com 22  Web analytics A system of tools and techniques for optimizing web usage by: Measuring, Collecting, Analyzing, and Reporting web data Typically focus on the number of web visitors and page views. www.id-book.com 23 A section of Google analytics dashboard for id-book.com www.id-book.com 24  Choosing and combining techniques Depends on the: Focus of the study Participants involved Nature of the technique(s) Resources available Time available www.id-book.com 25  Summary Data gathering sessions should have clear goals. An informed consent may be needed. Five key issues of data gathering are: goals, choosing participants, triangulation, participant relationship, pilot. Data may be recorded using handwritten notes, audio or video recording, a camera, or any combination of these. Interviews may be structured, semi-structured or unstructured Focus groups are group interviews Questionnaires may be on paper, online or telephone Observation may be direct or indirect, in the field or in controlled settings. Techniques can be combined depending on the study focus, participants, nature of technique, available resources and time. 26 www.id-book.com  Lecture 11:Understanding People Data Analysis, Interpretation and Presentation INFO 3307 HUMAN COMPUTER INTERACTION  Aims Discuss the difference between qualitative and quantitative data and analysis. Enable you to analyze data gathered from: Questionnaires. Interviews. Observation studies. Make you aware of software packages that are available to help your analysis. Identify common pitfalls in data analysis, interpretation, and presentation. Enable you to interpret and present your findings in appropriate ways. www.id-book.com 28  Quantitative and qualitative Quantitative data – expressed as numbers Qualitative data – difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction Quantitative analysis – numerical methods to ascertain size, magnitude, amount Qualitative analysis – expresses the nature of elements and is represented as themes, patterns, stories Be careful how you manipulate data and numbers! www.id-book.com 29  Simple quantitative analysis Averages Mean: add up values and divide by number of data points Median: middle value of data when ranked Mode: figure that appears most often in the data Percentages Be careful not to mislead with numbers! Graphical representations give overview of data Number of errors made Internet use Number of errors made 10 Number of errors made 4.5 Number of errors made < once a day 4 8 3.5 6 once a day 3 2.5 4 2 once a week 1.5 2 1 0 2 or 3 times a week 0.5 0 0 5 10 15 20 User www.id-book.com once a month 1 3 5 7 9 11 13 15 17 30 User  Simple qualitative analysis Recurring patterns or themes Emergent from data, dependent on observation framework if used Categorizing data Categorization scheme may be emergent or pre-specified Looking for critical incidents Helps to focus in on key events www.id-book.com 31  Tools to support data analysis Spreadsheet – simple to use, basic graphs Statistical packages, e.g. SPSS Qualitative data analysis tools Categorization and theme-based analysis Quantitative analysis of text-based data Nvivo and Atlas.ti support qualitative data analysis www.id-book.com 32  Presenting the findings Only make claims that your data can support The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken Graphical representations (as discussed above) may be appropriate for presentation Other techniques are: Rigorous notations, e.g. UML Using stories, e.g. to create scenarios Summarizing the findings www.id-book.com 33  Summary The data analysis that can be done depends on the data gathering that was done Qualitative and quantitative data may be gathered from any of the three main data gathering approaches Percentages and averages are commonly used in Interaction Design Mean, median and mode are different kinds of ‘average’ and can have very different answers for the same set of data Presentation of the findings should not overstate the evidence www.id-book.com 34