Software Development Evaluation

Evaluation

• Evaluates the usability and functionality of a system
• Occurs in laboratory, field, and/or in collaboration with users
• Evaluates both design and implementation

Goals of Evaluation

• Evaluates level of system functionality
• Evaluates effect of interface on user
• Identifies specific problems

Evaluating Designs

• Cognitive Walkthrough: evaluates design on how well it supports user in learning tasks
• Heuristic Evaluation: evaluates design based on usability criteria (heuristics)
• Review-based evaluation: uses written reviews to support or refute parts of design

Cognitive Walkthrough

• Evaluates design on how well it supports user in learning tasks
• Performed by expert in cognitive psychology
• Expert 'walks through' design to identify possible problems using psychological principles
• Forms used to guide analysis
• For each task, walkthrough considers:
  • What impact will interaction have on user?
  • What cognitive processes are required?
  • What learning problems may occur?

Heuristic Evaluation

• Proposed by Nielsen and Molich
• Evaluates design based on usability criteria (heuristics)
• Examples of heuristics:
  • System behavior is predictable
  • System behavior is consistent
  • Feedback is provided
• Heuristic evaluation 'debugs' design

Review-based Evaluation

• Results from written review used to support or refute parts of design
• Care needed to ensure results are transferable to new design

Evaluating through User Participation

• Laboratory studies:
  • Advantages: specialist equipment available, uninterrupted environment
  • Disadvantages: lack of context, difficult to observe several users cooperating
• Field Studies:
  • Advantages: natural environment, context retained
  • Disadvantages: distractions, noise

Evaluating Implementations

• Requires an artefact: simulation, prototype, or full implementation
• Experimental evaluation:
  • Controlled evaluation of interactive behavior
  • Evaluator chooses hypothesis to be tested
  • A number of experimental conditions are considered, which differ only in the value of some controlled variable
  • Changes in behavioral measure are attributed to different conditions

Experimental Factors

• Subjects:
  • Who: representative, sufficient sample
• Variables:
  • What to modify and measure
• Hypothesis:
  • What you'd like to show
• Experimental design:
  • How you are going to do it

Variables

• Independent variable (IV):
  • Characteristic changed to produce different conditions
  • E.g. interface style, number of menu items
• Dependent variable (DV):
  • Characteristics measured in the experiment
  • E.g. time taken, number of errors

Hypothesis (Thesis or Theory)

• Prediction of outcome
• Framed in terms of IV and DV
• E.g. "error rate will increase as font size decreases"
• Null hypothesis:
  • States no difference between conditions
  • E.g. "no change with font size"