The requirement is originally defined as:
"Blankchester is a cathedral city in the centre of England. It is an historical city with tourists visiting but also has modern industry. As part of the "Blankchester - a New Millennium" project, Blankchester City Council have commissioned a web site to raise the profile of Blankchester. The project will also contribute to the "Blankchester - access to all" equal opportunities project. It is anticipated that it will be used by tourists and by residents for information about Blankchester, and by local tourist information officers for finding information when answering telephone enquiries. The web site may be accessed via Internet browsers, or for local authority employees via the proposed Intranet. In the Intranet the local authority employees will have access to additional information and pages. The site may be used simply to browse information and additionally to make bookings for tourist accommodation and events. The website shall be designed with reference to the requirements of the Disability Discrimination Act, and shall therefore comply to WAI and RNIB guidelines."
A particular difficulty with setting requirements for a web site is correctly identifying the tasks. The only safe way is to find out what users actually want to do with the site (perhaps by investigating how people use other similar sites, or by user-based evaluation of very early paper prototypes). The IT department visits two other Tourist Offices, at Dashmouth-on-Sea and Cipher Hills, to see how the tourist officers used the site and to look at their site feedback. This information is fed into the validation and review process. The validation and review of these requirements gives rise to a context checklist, part of which is shown below:
Figure 1 context checklist:
Blankchester City web site - Context Checklist |
|||||
Goals for this system: raise the profile of Blankchester, provide access to information ("access to all" campaign), provide booking system for accommodation and events in the City. |
|||||
Users |
|
|
|
||
Type |
Tourist |
Resident |
Tourist Information officer |
||
Skills and knowledge No / low / good Web/IT knowledge No / low / good Local knowledge No / low/high previous experience of Hotel booking |
ü ü ü |
ü ü ü |
ü ü ü |
||
Attributes (people) with sight / hearing / language / mobility difficulties / with other special needs / without special needs |
ü
|
ü
|
ü
|
||
Smoking / non smoking With children / without children With pets / without pets |
ü ü ü |
ü ü ü |
ü ü ü |
||
List of tasks Browsing general information / tourist attractions / events Hotel/accommodation/events searches and bookings Browse additional intranet pages (for space in the example this is not expanded) |
ü ü N/A |
ü ü N/A |
ü ü ü |
||
Task checklist |
Browsing |
Searches |
Bookings |
||
Goal User Blankchester City |
Finds information Profile raised |
Finds information More rooms let |
Room / event ticket More rooms let Larger attendance at events |
||
Output |
List of general information meeting personal criteria |
List of hotels meeting personal criteria |
Booked and confirmed accommodation |
||
Frequency |
5000 hits per day |
5000 hits per day |
500 per month |
||
Duration |
4 clicks max traverse. |
Hotel search 1 click from home page |
book hotel less than 1 minute from selection |
||
Environment in which the software/system is to be used |
Tourist office - Tourist Information Officer's PC - IE5 Tourist office / library public PC IE5 Other user - could be a range of browsers - test for IE5 + IE4 + Netscape4 Design note: ensure text only version of the site available. (Note: this is a newly identified requirement and must be fed back to the designers) |
||||
Note: A context checklist may also consider side effects, flexibility, physical/mental demands, dependencies, safety, and criticality.
The requirements, context checklist and paper layout of the screens is inspected, using heuristic evaluation checklists. The context checklist is then used to draw up tests using equivalence partitioning and boundary value analysis for users against environment.
Figure 2 Partitions
The user partitions are:
1. Tourist
2. Resident
3. Tourist officer
4. No / low Web/IT knowledge
5. No / low Local knowledge
6. No / low previous experience of Hotel booking
7. good Web/IT knowledge
8. good Local knowledge
9. previous experience of Hotel booking
10. with sight difficulties
11. without sight difficulties
12. with hearing difficulties
13. without hearing difficulties
14. with language difficulties
15. without language difficulties
16. with mobility difficulties
17. without mobility difficulties
18. with other special needs
19. without other special needs
20. smoker
21. non smoker
22. with children
23. without children
24. with pet
25. without pet
The environment partitions are:
26. Tourist office - TI's PC - IE5
27. Library public PC IE5
28. User's own equipment - IE 5
29. User's own equipment - IE 4
30. User's own equipment - Netscape 4
31. Tourist officer uninterrupted
32. Tourist officer also answering phone enquiries
The tasks will also be partitioned, from the context checklist, but for ease of explanation we will consider only one task: hotel booking. BS 7925-2 says that more than one partition can be checked in a single test case, so we combine the partitions to make a smaller number of tests. The table below shows 7 tests:
Figure 3 Action/task list – test cases
ACTION/TASK: Book a hotel room |
||||
Test case |
New Partitions tested |
user type / skill / Attribute (people) / environment |
Actions |
Expected outcome |
1 |
1, 4, 5, 14, 16, 20, 23, 24, 27 |
A tourist with sight difficulties and a guide dog, little web experience, and no local knowledge, no children, uses the library public PC. She wants a smoking room. English is not her first language. |
Browses for a 4 star hotel and books it |
Able to book a room (smokers) at the 4 star Jupiter Hotel, which welcomes pets and guide dogs. Site / PC has tools to offer information in audio and in languages other than English |
2 |
2, 6, 7, 8, 10, 11, 13, 15,17,19, 30 |
A local person with good local knowledge and good IT /web skills but no experience of hotel booking uses their own PC (Netscape 4) to book hotel rooms for a visiting party of friends. No Special needs |
Browses for hotel in the £80-100 per night range and books hotel |
Able to find and book Mars hotel, able to find and print map to Mars hotel |
3 |
3, 9, 26, 32 |
A tourist officer with little web experience, good local knowledge, previous experience of hotel booking, no special needs, at own PC, takes enquiry and booking from tourist in office, while being interrupted by phone calls |
Browses for hotel in the £40-60 per night range and books hotel |
Able to find and book Saturn hotel, able to answer additional questions, provide map to Saturn hotel. No timeouts. |
4 |
12, 28 |
A local person with good local knowledge and good IT /web skills but no experience of hotel booking uses their own PC to book hotel rooms for a visiting party of friends. Hearing difficulties, own PC (IE5) |
Browses for hotel in the £80-100 per night range and books hotel |
Able to find and book Mars hotel, able to find and print map to Mars hotel. Major information given visually. |
5 |
16 |
A local person with good local knowledge and good IT /web skills but no experience of hotel booking. Person with mobility difficulties (wheelchair and arthritis in hands and wrists) Uses library PC to book a hotel room for a friend |
Browses for hotel in the £80-100 per night range and books hotel |
Able to find and book Gaia hotel, able to find and print map to Gaia hotel. Height and angle of PC suits needs. Keyboard and pointer device easy to handle and control without pain. |
6 |
31 |
A tourist officer with little web experience, good local knowledge, previous experience of hotel booking, no special needs, at own PC, takes enquiry and booking from tourist in office, without being interrupted by phone calls |
Browses for hotel in the £40-60 per night range and books hotel |
Able to find and book Saturn hotel, able to answer additional questions, provide map to Saturn hotel. |
7 |
18, 21, 22, 25, 29 |
A tourist whose child has an allergy to animals, no pets, little web experience, and no local knowledge, her own PC (IE4) to book a no-smoking room |
Browses for a 4 star hotel and books it |
Able to book a no-smokers room for self and child at the 4 star Cathedral Hotel which has a no pets policy and check for allergy aware hotels |
Note 1 - table could alternatively be drawn up showing all the partitions tested by each case thus allowing the tests to be used in any order
Note 2: a similar table of tests would be drawn up for the other tasks
Tests are also drawn up using use cases. An example use case is shown in Figure 4 on page 4.
Figure 4 Test Scenarios
Scenario name: |
Enquiry for hotel |
||
System: |
Blankchester web site |
V0.9 |
Date: 12-1-2000 |
Goal of test |
The goal of this use case is to test the basic enquiry job, where a (potential) tourist is searching for accommodation |
||
|
Description |
||
Context |
This is core functionality, so this is an important test of a frequent usage function |
||
Actors |
Any user may use this function : Tourist - low web user, Tourist - high web user, BCC Tourist officer (trained), Local person - low web user, Local person - high web user |
||
|
|
Checked |
Comments |
Preconditions |
The PC is on and functional The user is connected via their normal ISP The user is online and looking at the web site home page and has chosen to use the drop down selections |
|
|
Description of main flow |
2 star in walking distance of cathedral 1. the user clicks on Hotels 2. the user clicks on 2 star 3. the user clicks on location - cathedral 4. the user clicks on Find |
|
|
Expected results |
the system displays 2 hotels details - "Wardens House" and "Deans House" and the user can print or save the information |
|
|
Alternative flows |
The user wants to see 3 star hotels - all in Blankchester 1. the user clicks on Hotels 2. the user clicks on 3 star 3. the user clicks on find |
|
|
Expected results |
the system displays a drop down list with all 3 star hotels (15 on test database) |
|
|
Failure conditions |
No booking conformation number displayed Wrong rating hotels displayed |
|
|
Additional attention points |
Note response times |
|
|
With the constraints on local authority budgets, it is decided not to use a full usability lab test, but instead to use the "Thinking Aloud" protocol with an observer logging what the user does, using the use cases and selecting the user group from the equivalence partitioning tests already defined. After the tests are run, the users will be asked to complete a SUMI questionnaire. Representative users will be asked to complete typical tasks, and measures taken of effectiveness, efficiency and satisfaction. It was expected that all Tourist Office users could successfully complete a hotel booking in an average time of less than 5 minutes. All SUMI scores should be above the industry average of 50.
Context used for the Tourist Office test: For this part of the test, eight typical users from the Tourist Office were selected who had the key characteristics and capabilities, but no previous experience of the web site. The other characteristics of the participants that might influence usability were recorded, together with the age group and gender (see Figure 5 on page 4):
Figure 5 User group characteristics
|
Time in job (years) |
Windows experience (years) |
Internet / web site experience (years) |
Attitude to computers/ web |
Gender |
Age group |
1 |
5.5 |
3.5 |
0 |
6 |
F |
20-35 |
2 |
0.8 |
2.1 |
0.8 |
1 |
F |
20-35 |
3 |
2.1 |
2.5 |
2.1 |
3 |
M |
20-35 |
4 |
4.9 |
3.5 |
1.5 |
2 |
F |
36-50 |
5 |
0.7 |
0.7 |
0.7 |
2 |
M |
20-35 |
6 |
1.6 |
2.1 |
0 |
3 |
F |
36-50 |
7 |
4.3 |
1.4 |
0 |
4 |
M |
36-50 |
8 |
2.7 |
4.6 |
2.7 |
4 |
M |
20-35 |
*1=prefer to use a computer as much as possible, 7=prefer to use a computer as little as possible
The intended context of use for this test was the Tourist Office, and so the tests will be carried out there, but it is arranged so that participants could work alone without any interruptions. They are to be observed by an observer/tester. Rules for carrying out the tests are agreed: participants would be shown the evaluation suite, informed that their interaction would be recorded, they would be asked to sign a release form, asked to confirm the information they had provided about, and to score their attitude towards use of computers. Tasks were to be timed using a stopwatch. Sessions would not be videotaped but the user asked to use "Thinking aloud" so that the observer/tester could take notes. At the end of the sessions, participants would complete the SUMI satisfaction questionnaire.
Instructions for user testers would be written, for example:
Figure 6 User instructions
Thank you for helping us in this evaluation. The purpose of this exercise is to find out how easily people like you can use the city web site. To achieve this, we will ask you to perform some tasks, and your performance will be observed. Then, to help us understand the results, we will ask you to complete a standard questionnaire, and to answer a few questions about yourself and your usual workplace. The aim of this evaluation is to help assess the product, and the results may be used to help in the design of new versions. Please remember that we are testing the software, not you.
When you have finished each task, or got as far as you can, please phone us by dialling 1234. I am afraid that we cannot give you any assistance with the tasks.
You have just been shown the Blankchester web site. You now need to find out whether it could meet your current needs. You will perform the following tasks: 1. take some time to familiarise yourself with it and specifically the hotel "find and book" functions, 2. book a hotel room meeting your requirements We are interested to know how you go about these tasks and whether you find the software helpful or not. LET US KNOW WHEN YOU ARE READY TO BEGIN Task 1– Familiarisation period - Spend as long as you need to familiarise yourself with the web site. (YOU HAVE UP TO 20 MINUTES) LET US KNOW WHEN YOU ARE READY TO MOVE ON Task 2 – Book hotel room (YOU HAVE ABOUT 15 MINUTES FOR THIS TASK) Use the software to book a hotel room: DATE: 23 November 2001, PLACE: The Blue Flag Inn, Blankchester. LET US KNOW WHEN YOU HAVE FINISHED |
After the tests, the mean completion rate, mean goal achievement, mean task time, mean completion rate efficiency and mean goal achievement efficiency would be calculated for the task "booking a hotel room". After the last task, participants would be asked to complete a subjective ratings scale and the SUMI questionnaire. The evaluator would then asked them about any difficulties they had encountered. Metrics to be gathered were decided as:
§ Effectiveness: Completion Rate Percentage of participants who completed each task correctly. Mean goal achievement Mean extent to which each task was completely and correctly achieved, scored as a percentage.
§ Efficiency: Task time: mean time taken to complete each task (for correctly completed tasks). Completion rate efficiency: mean completion rate/mean task time. Goal achievement efficiency: mean goal achievement/mean task time.
§ Satisfaction: measured using the SUMI questionnaire, at the end of the session, giving scores for each participant’s perception of: overall satisfaction, efficiency, affect, controllability and learnability.
Note: SUMI provides a set of statistical tables and graphs – these are not shown here but can be viewed in the web-sites in the references.
In addition to these user tests, the following accessibility checks are carried out:
§ The web site design is checked against the WAI guidelines (inspections against checklist)
§ The web site is checked against the RNIB guidelines (inspections against checklist)
§ The web site is checked against the Bobby tool (www.cast.org) for suitability for use with a text browser
The usability tests are to be documented using the Common Industry Format described in ISO9126.