Unit 7 Comm and Testing part 2.pdf

Communication and testing Continued ”Show don’t sell” Prototypes help us think through solutions, make design decisions, and answer key questions: What is prototyping? Why is it important: A process to generate, visualize and refine ideas (and assumptions), to more fully understand them, build on th...

Communication and testing Continued ”Show don’t sell” Prototypes help us think through solutions, make design decisions, and answer key questions: What is prototyping? Why is it important: A process to generate, visualize and refine ideas (and assumptions), to more fully understand them, build on them and quickly evaluate them at lower cost. Fast and resourceful way to generate ideas. Fail earlier, with less risk and less cost, while embedding the learning in the next version. Reduces uncertainty common in new ideas. Improves decision making as you’re choosing from more tangible options. Determine when an idea is actually desirable, viable, and feasible. In this famous example from IDEO a physician taped together a marker, film canister, and a clip to show what an ideal surgical device might look like. Use as many “off the shelf” elements that you can. © 2024 ID Academy Process and use the feedback: It is better to build several small prototypes to illustrate key features than to try to create a single, integrated prototype All good prototypes answer a question. Think about what question or questions your prototype(s) is/are answering (for example, how will it be carried / what material should it be) You may… Prototype to BUILD the idea Prototype to EXPERIENCE the idea Prototype to TEST the idea Jeff Hawkins, the inventor of the Palm pilot (the first successful handheld device) walked around with a block of wood for several weeks. He would pull it out whenever he felt like he was in a situation where he might use the device in the future. This helped him understand all the scenarios of use. Making prototypes active does not mean they need to be complex. Students at ID prototyping a service for Mayo clinic. Important elements were high resolution (like the menu) while less important things were low resolution (the counter) This prototype demonstrated how an emergency cart could fit into the galley space on a 777 aircraft. 3 The prototype matrix helps connect project stage to solution type: V1 (rough draft) Assessment of critical functions Visualization Design sketches and list of requirements Cheap materials / origami Components built separately of materials adequate for testing / or components developed and tested digitally Looks like / works like protype with (nearly) accurate materials. Digital 3D model complete. Hand sketch storyboards based on personas Layout in Android studio, Adobe XD, or similar Code developed or adapted for specific functions and features. Final functions assembled into single system and errors checked One or more skits that allow users to experience the service and benefits A/B testing and operations research A Wizard of Oz prototype (users can experience look and feel of service without all operational details functioning). List of requirements for each abstraction plus simple input / process / output diagrams Flow diagrams for each possible abstraction Code developed or adapted for specific functions and features. Components can be tested separately Working algorithm that can be tested in a controlled environment Design sketches and list of requirements Set of off the shelf and existing components stitched together to demonstrate functionality (representing how the robot will work) Code and mechanics sufficiently developed (separately) to allow for testing. Code and mechanics integrated (works- like prototype) Sketch phase Jewelry box (simple product) App on the phone (a digital service) New coffee shop (physical service design) An algorithm A robot © 2024 ID Academy Hand sketch storyboards based on personas 4 ”Create experiences not explanations” The prototype matrix helps connect project stage to solution type: Jewelry box (simple product) App on the phone (a digital service) New coffee shop (physical service design) An algorithm A robot © 2024 ID Academy Assessment of critical functions Sketch phase V1 (rough draft) Design sketches and list of requirements Components built separately of materials adequate for In this section we will focus on how create Cheap materials / origami / or phase. components and test prototypes intesting the sketch developed and tested digitally Looks like / works like protype with (nearly) accurate materials. Digital 3D model complete. There is no better wayCode to accelerate concept developed or adapted for Layout in Android studio, than to show stakeholders development specific functions and Adobe XD, orearly similar versions of your ideas and get their features. feedback. Showing sketch prototypes ensures that you have not over-invested in a One or moreconcept skits that allowknowing if people have before A/B testing and operations users to experience interest,the findservice it generally acceptable, and research and benefitsdon’t have hidden beliefs or preferences that would make the idea invalid. Final functions assembled into single system and errors checked Hand sketch storyboards based on personas Hand sketch storyboards based on personas Visualization A Wizard of Oz prototype (users can experience look and feel of service without all operational details functioning). List of requirements for each abstraction plus simple input / process / output diagrams Code developed or adapted for You can consider this “co-design”, it is aand specific functions Flow diagrams for each time to show early ideas and riff on them possible abstraction features. Components can be with experts and users. tested separately Working algorithm that can be tested in a controlled environment Design sketches and list of requirements Set of off the shelf and existing components stitched together to demonstrate functionality (representing how the robot will work) Code and mechanics integrated (works- like prototype) Code and mechanics sufficiently developed (separately) to allow for testing. 6 Conduct stakeholder feedback reviews to get feedback and improve our concepts: Show your concept boards to stakeholders: Ask questions and collect feedback: This is an opportunity to share your concept with others to get another perspective and understand if you are on the right track with your idea. For each concept, ask the following questions: Getting feedback is one of the best ways to improve, edit or shift your concept during the design process. Getting feedback from others helps reduce the inherent bias you may bring to an idea. The sooner you can get feedback the less time you will spend on ideas that may not be solving the problem you defined. Asking for feedback also allows us to communicate their ideas to non-expert audiences. Why would you use it? What is most attractive or relevant to you? How would you use this? Is there a specific time recently it would have helped? What would prevent you from using it? What potential barriers do you see? What questions do you have? What questions would need to be answered for you the feel comfortable using this concept? If talking to experts modify to ”how do you see customers using this…” Record the answers. When complete ask the stakeholder if they want to change or review any of the feedback on any of the concepts. Do as many reviews as possible. Talk to experts, the people you originally interviewed, and potential customers / users. Typical projects will include 5 – 20 review sessions. Your concept boards, storyboards, and strategies can become Boundary Objects – they create a shared understanding of what we are talking about but allow different groups to express their own perspectives. © 2024 ID Academy Process and use the feedback: First, review your concept tests: What did you hear and observe from users or experts? What did they say worked? What did not? What questions did they have? Did they have any ideas for making the concept better? Which concepts had the strongest reactions? Why? Ideas with the strongest reactions (both positive and negative) are often the most promising. Next, select the most promising idea to move forward: What feedback and/or data suggests that this is the strongest idea to move to the next release? What elements or solutions from the other ideas can be added to the selected solution to make it more robust without sacrificing internal consistency and elegance (in other words… what can you add to the solution without messing the whole thing up) Your concepts are not valuable -- the feedback you get from them is valuable. Abstract the feedback, don’t fall into repair service behavior. 7 Use the interview guide and feedback templates to determine which three ideas should move into the DVF screen: Documenting feedback template: Concept name: How would you use this? Is there a specific time recently it would have helped? Analysis of feedback: Reviewer: What would prevent you from using it? What potential barriers do you see? What did we hear? What did we learn? Text goes here Text goes here What does it mean for the next version of our concept? Text goes here Always be sure to make an abstraction of the critique: is their critique evidence of a larger opportunity, and what general direction does it suggest? Of course, always consider, will this help the project? Never overreact or under-react. You may use the apply, deny, or modify framework for all feedback: Why would you use it? What is most attractive or relevant to you? What questions do you have? What questions would need to be answered for you the feel comfortable using this concept? Apply: make the exact changes they recommend Deny: do not use the critique, but create a rational for why you are not using it Modify: use part of the critique, again citing a rational © 2024 ID Academy 8 Student example: stakeholder feedback © 2024 ID Academy 9 In this class we will focus on v1 (and the process of engaging with users). Our previous exercise covered the sketch phase (you already were prototyping and did not know it!) V1 (rough draft) Assessment of critical functions Design sketches and list of requirements Cheap materials / origami Components built separately Looks like / works like protype of materials for we will focus with more (nearly) Inadequate this section onaccurate how to testing / ortest components materials. V1 prototypes with users thanDigital we will3D onmodel developed building and tested complete. anddigitally assembling the prototypes. App on the phone (a digital service) Hand sketch storyboards based on personas Layout in Android studio, Adobe XD, or similar New coffee shop (physical service design) Hand sketch storyboards based on personas One or more skits that allow users to experience the service and benefits Sketch phase Jewelry box (simple product) An algorithm A robot © 2024 ID Academy List of requirements for each abstraction plus simple input / process / output diagrams Flow diagrams for each possible abstraction Design sketches and list of requirements Set of off the shelf and existing components stitched together to demonstrate functionality (representing how the robot will work) Visualization Code developed or adapted for Final functions assembled into Since fidelity is often low in the V1 stage just specific functions and single system and errors about anyone will be able to build and create features. checked prototypes that can be effectively tested. This is especially fun and A Wizard rewarding of Oz forprototype people that are not often (users asked can to experience be creative look and A/B testing and operations or make things. However easy it is to start feel of service without all research prototyping, it can take operational an entire career details to achieve mastery. functioning). Code developed or adapted for In this class you will learn a process for that can be Working algorithm specific functions and deploying the prototypes to gain tested in acritical controlled features. Components can be feedback, enable co-design, and help you environment tested separately create more robust iterations of your concepts. Code and mechanics Code and mechanics sufficiently developed integrated (works- like (separately) to allow for prototype) testing. 10 A four step process for prototyping M.I.A. Activity Example: Behaviors Determine the Most Important assumption for your concept Identify the user behaviors closely associated with your MIA, around which you will build your prototype engagement Asymmetric doors on a CTA train car will reduce crowding near the doors and make it easier to enter and exit the train. Simulate the behavior of boarding, finding a seat, riding, and then exiting a train in this new (asymmetric) door configuration. Asymmetric design: doors are staggered to reduce crowding. © 2024 ID Academy 4. 3. 2. 1. Simulated experience Create a real enough, but not too real way for users to engage in these behaviors in order to test your MIA Go / Kill / Iterate Analyze the results of your test and determine next steps: go, kill, or iterate The new design caused some confusion – difficult for people to know where the closest door is. People still preferred to crowd around the doors, problem actually worsened. Idea was killed. Full size mock-up using tape and stools, simulated ride from downtown to 35th street on the CTA Red Line 11 Steps 1 and 2: Determine your M.I.A and associated new behaviors M.I.A.: Most Important Assumption New Behaviors: the actions and activities you would observe Assumption: An element of your idea that must work or be valid in order for the idea to be implemented. Imagine your concept or solution has been implemented. What new behaviors will be observed? How will people act and think differently once your concept has been made real? What will they be doing, feeling, seeing, hearing, (and even sometimes) smelling? What will they be interacting with? What will they be asked to do, think about, and decide on? Most Important: the element that is hardest to get right, is the keystone (holds everything together), or is least likely to be true. We identify and test our M.I.A. first. This ensures we don’t waste time and money testing things that are easy and obvious, we want to test the hardest part first in order to give us the confidence required to continue. To identify your assumptions ask: what would have to be true in order for us to commit to this choice? For example, a group of biomedical engineers were designing an aftermarket enhancement for microwaves that would allow people with limited vision to quickly and accurately cook their food. The MIA of the concept was that adding a few raised buttons would be enough to enable accurate and fast time selection. There were other assumptions, including that the product would be easy to install / retrofit, but the team determined the function and utility of the buttons to be the most important element to test. © 2024 ID Academy We need to define these new behaviors and then design prototypes that enable people to try out and experience them. Creating one of these experiences can be complex, we only want to focus on simulating one to three new behaviors, and the behaviors should be directly related to the MIA. In our example the MIA was “adding a few raised buttons would be enough to enable accurate and fast time selection”. The behavior associated with this assumption was “using the buttons to select the cooking time”. The prototype was built to simulate the experience of using raised buttons (with limited vision provided by a bandana) to select a cooking time. We would not make prototypes related to selecting food, putting food in the microwave, or removing food from the microwave. You can see that the prototype was only the buttons (in several configurations). Here is the simple prototype that was used to test the behavior: using a limited set of raised buttons to select a cooking time. As you see the team created a few configurations to test different options. 12 MIA / new behaviors worksheet M.I.A.: Most Important Assumption New Behaviors: what actions and activities you would see List all of the possible assumptions. Circle your MIA: List all of the new behaviors you might see if your solution is implemented. Circle the one to three behaviors most directly related to your MIA: © 2024 ID Academy 13 Creating simulated experiences: Your prototype and simulated experience should include the following elements: When creating your sets / prompts and the script be intentional about what is critical and what can be ignored: Participants: the people experiencing and using the prototype. Often your target customers, sometimes can be experts, occasionally friends / family / coworkers. Facilitators: people that guide and manage the participant’s engagement with the prototypes. Sometimes they play the role of the system (if you are simulating a digital / AI / automated experience). Great prototypes do not require extensive facilitation. Researchers: people that are observing the engagement, running co-design activities, and asking questions. Ideally your researchers are not also your facilitators (sometimes this can’t be avoided). Sets and prompts: the materials, spaces, and communications that make up the physical aspects of the prototype. Script: the process, key steps, decision logic (if participants do this, this should happen), and story (including any required background and exposition) that orchestrates the engagement. Critical elements that need to feel real: the few, highly important aspects of the prototype. Should be directly related to your MIA and the related new behaviors. These are the elements that need to be high-fidelity and well thought out. What can be faked: parts of the prototype that help provide context, scale, and add support the critical elements. These can be low resolution, low cost, and do not need to seem real. What can be ignored: elements that do not need to be part of the prototype. They are not essential for testing the MIA and including them may be distracting or an inefficient use of resources. © 2024 ID Academy Microwave example Participants: five people from the Chicago Lighthouse Facilitators: one designer that welcomes the participant, explains the project, and asks them to “select a cook time of five minutes and 30 seconds” on each of the three options. Played the role of the microwave speaker (when a button was depressed they would say the number, and what was adjacent to the button). Researchers: observe the participants using the prototypes. Ask questions about what worked and what did not. Seek new ideas and improvements. Sets and prompts: Critical: the buttons. What can be faked: the key pad. What can be ignored: the microwave itself, the food. Script: based on common cooking needs at the Lighthouse, prompts to select different cooking times. 14 Simulated experience template: Concept we are prototyping: Participants: MIA: New behaviors we are simulating: Script: Sets and prompts: Critical elements that need to feel real: Facilitators: Researchers: © 2024 ID Academy What can be faked: What can be ignored: 15 Simulated experience template example: Concept we are prototyping: Asymmetric door design for CTA trains MIA: Asymmetric doors on a CTA train car will reduce crowding near the doors and make it easier to enter and exit the train. Participants: 25 students. All take the El and are familiar with public transportation. New behaviors we are simulating: Simulate the behavior of boarding, finding a seat, riding, and then exiting a train in this new (asymmetric) door configuration. Script: Note: the script was much more detailed than shown here: it required a second page. The team took the real El from downtown Chicago to 35th street and documented the stops, timing, and announcements. These prompts were replicated and used for the prototype. Since there was no need for the participants to wait… the team “fast forwarded” between stops in order to focus attention and time on entry / exit / finding a seat. Sets and prompts: Critical elements that need to feel real: Size and spacing of train car. Car size and spacing is full size: tape is used to demarcate the actual size of the car and doors. Facilitators: What can be faked: Explains the project to the participants. Acts as the voice of the train, e.g. “doors are closing, Cermak is the next stop”. Stops participants from violating laws of physics “you can’t exit that way, it is the wall of the train”. Moves group from prototype to prototype. Seats and doors. We will use stools from the shop as the seats. We will have foam core boards (slid open and closed by team members) to replicate the doors opening and closing. Researchers: What can be ignored: Walls and windows. Train does not need to move. Observe all activities and flows. Use stopwatch to time entry / exit. Conduct counts of where people move and congregate. Conduct postexperiment reviews with the participants. Take photos of each key moment. © 2024 ID Academy 16 Step 4: Go, Kill, Iterate. Making a final decision: And leverages keys to success, avoiding failure modes: Quantitative Data Expert Intuition Qualitative Data Risk A good decision includes four types of input: DEFENSIBLE DECISION Making the decision deploy, develop, or delete a concept should be based on analysis of the data collected via evaluation methods. It is important not to stall a decision just because you don't have perfect data. Waiting to decide results in inaction which destroys value and brings innovation to a halt. Begin analysis with all available data about a prototype. First, pull out the relevant pieces of information. Look for patterns, themes and trends. Then, cluster similar data. Look for connections between qualitative and quantitative data. Ultimately, the decision to deploy, develop, or delete should be based on how far the prototype gets you towards your objective. 0% 100% 50% Failure modes: Keys to success: Rely only on instinct and intuition Drive towards richness and complexity Analysis is cumbersome Best choice is obvious No support from leaders Final recommendation feels anemic Leadership is committed to action There is buy in from the team When you “kill” and idea, go back to your solution architecture or portfolio and select a new idea to move into prototyping. As ideas are eliminated, new ideas should enter into the prototyping stage. © 2024 ID Academy 17 Go / kill / iterate worksheet: DECISION Prototype name Decision circle one GO take steps to implement the prototype KILL stop working on the prototype ITERATE revise and retest the prototype Quantitative evidence briefly summarize the data that supports your decision Qualitative data briefly summarize the data that supports your decision RISKS Potential risks what risks come with the decision? Severity and probability how serious are the risks? how likely are they to happen? Mitigation and management what could we do to mitigate the risks? how well could we deal with them if they arise? Expert intuition how does your experience inform your decision? © 2024 ID Academy 18 ”A wonderful example of how if the universe is one entity, and it tries ideas out- and goes “this is a cool idea, but this person isn’t the right person to be doing it because they have other things going on”” - Kevin Shields, My Bloody Valentine