Chapter 2: Interface Evaluation
Outline – 2 methods
Usability testing: evaluating user interface through User Participation
Heuristic evaluation 1. What is usability testing?
To prove whether a user interface achieves
Usability goals
User experience goals: satisfying and enjoyable
Find flaws and give designers evaluative feedback that guides their work
It becomes an established and accepted part of the design process
Stage 1: Preparation: Make sure test ready to go before user arrives
Stage 2: Introduction
Say purpose is to test software, not the user!
Consent form explains procedures and deals with ethical issues
Give instructions
Pre-test questionnaire (personal info, background, etc.)
Stage 3: Running the test
Stage 4: Debriefing after the test:
Post-test questionnaire, explain purpose, thanks 2. Who runs the experiment?
Trained usability engineers know how to run a valid study
Instructor: instruct the user (explain the test scripts), encourage users to speak aloud their thoughts, and help users whenever it is necessary (note: but do not interfere and do not help early)
Observer (note-taker): note down problems that the user encounters, record the time and paths that users have taken
Useful for developers & designers to watch 3. Where to do the usability testing?
Controlled usability testing:
Environmental settings
Involves recording performance of typical users doing typical tasks
Users are observed and timed
Data is recorded on video & key presses are logged
Remote usability testing: web-based questionnaire
4. Who will join the test?
Participants should be chosen to represent the intended user communities, with attention to: Age, gender, background in computing, experience with the task, motivation, education, and ability with the language used in the interface
Number of users required
Participation should always be voluntary, and informed consent should be obtained
5. What will users do? - Task scenarios
Task design is a difficult part of usability testing
Representative of “real” tasks
Different systems deserve different tasks
Some systems support several typical tasks. Identify these tasks before asking users to do!
Realistic and compelling so users are motivated to finish
Can let users create their own tasks if relevant
Short enough to be finished
6. What are to be measured, and how?
Time to learn: How long does it take for typical members of the community to learn how to perform relevant tasks with the interface?
->Number of times users ask for help or manuals
Speed of performance: How long does it take to perform relevant tasks?
->Time to complete a task
->Percent of users who complete task successfully
Rate of errors by users: How many and what kinds of errors are made by users during performing the tasks?
->Number and type of errors per task
->Number of errors per unit of time
->Percent of users making a particular error
Retention over time: How well do users maintain their knowledge after an hour, a day, or a week?
Subjective satisfaction: How much did users like using various aspects of the interface?
->Questionnaire design Questionnaire Design - Two parts
1. Pre-test questionnaire to collect general demographic information of users that may be relevant: Age, gender, education, job, computer experience, personality, familiarity with the interface, etc.
2. Post-task questionnaire to ask users’ subjective impressions/feelings about specific aspects of the interface
7. What to report? - Usability testing report
1. Executive summary
2. Methodology
3. Introductory questions and tasks
4. Findings and recommendations (e.g., per scenario)
5. Exit questions and user impressions
6. Appendices Executive summary
Organize problems by scope and severity
Scope: How widespread is the problem?
How many users have encountered similar problem?
How critical is the problem?
Did the problem impede users from completing the
task?
A “top ten” proble***ist
Levels of severity help rank the problems
Heuristic Evaluation
Nielsen’s 10 heuristics
1. Visibility of system status
2. Match between system and real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, recover from errors
10. Help and documentation
Outline – 2 methods
Usability testing: evaluating user interface through User Participation
Heuristic evaluation 1. What is usability testing?
To prove whether a user interface achieves
Usability goals
User experience goals: satisfying and enjoyable
Find flaws and give designers evaluative feedback that guides their work
It becomes an established and accepted part of the design process
Stage 1: Preparation: Make sure test ready to go before user arrives
Stage 2: Introduction
Say purpose is to test software, not the user!
Consent form explains procedures and deals with ethical issues
Give instructions
Pre-test questionnaire (personal info, background, etc.)
Stage 3: Running the test
Stage 4: Debriefing after the test:
Post-test questionnaire, explain purpose, thanks 2. Who runs the experiment?
Trained usability engineers know how to run a valid study
Instructor: instruct the user (explain the test scripts), encourage users to speak aloud their thoughts, and help users whenever it is necessary (note: but do not interfere and do not help early)
Observer (note-taker): note down problems that the user encounters, record the time and paths that users have taken
Useful for developers & designers to watch 3. Where to do the usability testing?
Controlled usability testing:
Environmental settings
Involves recording performance of typical users doing typical tasks
Users are observed and timed
Data is recorded on video & key presses are logged
Remote usability testing: web-based questionnaire
4. Who will join the test?
Participants should be chosen to represent the intended user communities, with attention to: Age, gender, background in computing, experience with the task, motivation, education, and ability with the language used in the interface
Number of users required
Participation should always be voluntary, and informed consent should be obtained
5. What will users do? - Task scenarios
Task design is a difficult part of usability testing
Representative of “real” tasks
Different systems deserve different tasks
Some systems support several typical tasks. Identify these tasks before asking users to do!
Realistic and compelling so users are motivated to finish
Can let users create their own tasks if relevant
Short enough to be finished
6. What are to be measured, and how?
Time to learn: How long does it take for typical members of the community to learn how to perform relevant tasks with the interface?
->Number of times users ask for help or manuals
Speed of performance: How long does it take to perform relevant tasks?
->Time to complete a task
->Percent of users who complete task successfully
Rate of errors by users: How many and what kinds of errors are made by users during performing the tasks?
->Number and type of errors per task
->Number of errors per unit of time
->Percent of users making a particular error
Retention over time: How well do users maintain their knowledge after an hour, a day, or a week?
Subjective satisfaction: How much did users like using various aspects of the interface?
->Questionnaire design Questionnaire Design - Two parts
1. Pre-test questionnaire to collect general demographic information of users that may be relevant: Age, gender, education, job, computer experience, personality, familiarity with the interface, etc.
2. Post-task questionnaire to ask users’ subjective impressions/feelings about specific aspects of the interface
7. What to report? - Usability testing report
1. Executive summary
2. Methodology
3. Introductory questions and tasks
4. Findings and recommendations (e.g., per scenario)
5. Exit questions and user impressions
6. Appendices Executive summary
Organize problems by scope and severity
Scope: How widespread is the problem?
How many users have encountered similar problem?
How critical is the problem?
Did the problem impede users from completing the
task?
A “top ten” proble***ist
Levels of severity help rank the problems
Heuristic Evaluation
Nielsen’s 10 heuristics
1. Visibility of system status
2. Match between system and real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, recover from errors
10. Help and documentation
