Index Up Site Map Latest News Working Practices Discussion & Review Glossary Module Testing Non-Func. Testing Links Test Papers Feedback Administration

Usability - e-Business

Example - The Blankchester City Web Site

The requirement is originally defined as:

"Blankchester is a cathedral city in the centre of England.  It is an historical city with tourists visiting but also has modern industry. As part of the "Blankchester - a New Millennium" project, Blankchester City Council have commissioned a web site to raise the profile of Blankchester. The project will also contribute to the "Blankchester - access to all" equal opportunities project. It is anticipated that it will be used by tourists and by residents for information about Blankchester, and by local tourist information officers for finding information when answering telephone enquiries. The web site may be accessed via Internet browsers, or for local authority employees via the proposed Intranet. In the Intranet the local authority employees will have access to additional information and pages. The site may be used simply to browse information and additionally to make bookings for tourist accommodation and events. The website shall be designed with reference to the requirements of the Disability Discrimination Act, and shall therefore comply to WAI and RNIB guidelines."

A particular difficulty with setting requirements for a web site is correctly identifying the tasks. The only safe way is to find out what users actually want to do with the site (perhaps by investigating how people use other similar sites, or by user-based evaluation of very early paper prototypes).   The IT department visits two other Tourist Offices, at Dashmouth-on-Sea and Cipher Hills, to see how the tourist officers used the site and to look at their site feedback.  This information is fed into the validation and review process. The validation and review of these requirements gives rise to a context checklist, part of which is shown below:


Figure 1 context checklist:

Blankchester City web site - Context Checklist

Goals for this system: raise the profile of Blankchester, provide access to information ("access to all" campaign), provide booking system for accommodation and events in the City.








Tourist Information officer

Skills and knowledge

No / low / good  Web/IT knowledge

No / low / good Local knowledge

No / low/high  previous experience of Hotel booking




Attributes (people)

with sight / hearing / language /  mobility difficulties /

with  other special needs / without special needs







Smoking / non smoking

With children / without children

With pets / without pets

List of tasks

Browsing general information / tourist attractions / events

Hotel/accommodation/events searches and bookings

Browse additional intranet pages

(for space in the example this is not expanded)






                Task checklist




Goal           User

Blankchester City

Finds information

Profile raised

Finds information

More rooms let

Room / event ticket

More rooms let

Larger attendance at events


List of general information meeting personal criteria

List of hotels meeting personal criteria

Booked and confirmed accommodation


5000 hits per day

5000 hits per day

500 per month


4 clicks max traverse. 

Hotel search  1 click from home page

book hotel less than 1 minute from selection

Environment in which the software/system is to be used

Tourist office - Tourist Information Officer's PC - IE5

Tourist office / library public PC IE5

Other user - could be a range of browsers - test for IE5 + IE4 + Netscape4

Design note: ensure text only version of the site available.  (Note: this is a newly identified requirement and must be fed back to the designers)


Note: A context checklist may also consider side effects, flexibility, physical/mental demands, dependencies, safety, and criticality.

 The requirements, context checklist and paper layout of the screens is inspected, using heuristic evaluation checklists. The context checklist is then used to draw up tests using equivalence partitioning and boundary value analysis for users against environment.

Figure 2 Partitions


The user partitions are:

1.               Tourist

2.               Resident

3.               Tourist officer

4.               No / low  Web/IT knowledge

5.               No / low Local knowledge

6.               No / low previous experience of Hotel booking

7.               good Web/IT knowledge

8.               good Local knowledge

9.               previous experience of Hotel booking

10.            with sight difficulties

11.            without sight difficulties

12.            with hearing difficulties

13.            without hearing difficulties

14.            with language difficulties

15.            without language difficulties

16.            with mobility difficulties

17.            without mobility difficulties

18.            with other special needs

19.            without other special needs

20.            smoker

21.            non smoker

22.            with children

23.            without children

24.            with pet

25.            without pet


The environment partitions are:

26.            Tourist office - TI's PC - IE5

27.            Library public PC IE5

28.            User's own equipment - IE 5

29.            User's own equipment - IE 4

30.            User's own equipment - Netscape 4

31.            Tourist officer uninterrupted

32.            Tourist officer also answering phone enquiries


The tasks will also be partitioned, from the context checklist, but for ease of explanation we will consider only one task: hotel booking.  BS 7925-2 says that more than one partition can be checked in a single test case, so we combine the partitions to make a smaller number of tests.  The table below shows 7 tests:

Figure 3 Action/task list test cases

ACTION/TASK: Book a hotel room

Test case

New Partitions


user type / skill / Attribute (people) / environment


Expected outcome


1, 4, 5, 14, 16, 20, 23, 24, 27

A tourist with sight difficulties and a guide dog, little web experience, and no local knowledge, no children, uses the library public PC. She wants a smoking room. English is not her first language.

Browses for a 4 star hotel and books it

Able to book a room (smokers) at the 4 star Jupiter Hotel, which welcomes pets and guide dogs. Site / PC has tools to offer information in audio and in languages other than English


2, 6, 7, 8,  10, 11, 13, 15,17,19,  30

A local person with good local knowledge and good IT /web skills but no experience of hotel booking uses their own PC (Netscape 4) to book hotel rooms for a visiting party of friends. No Special needs

Browses for hotel in the 80-100 per night range and books hotel

Able to find and book Mars hotel, able to find and print map to Mars hotel


3, 9, 26, 32

A tourist officer with little web experience, good local knowledge, previous experience of hotel booking, no special needs, at own PC, takes enquiry and booking from tourist in office, while being interrupted by phone calls

Browses for hotel in the 40-60 per night range and books hotel

Able to find and book Saturn hotel, able to answer additional questions, provide map to Saturn hotel. No timeouts.


12, 28

A local person with good local knowledge and good IT /web skills but no experience of hotel booking uses their own PC to book hotel rooms for a visiting party of friends. Hearing difficulties, own PC (IE5)

Browses for hotel in the 80-100 per night range and books hotel

Able to find and book Mars hotel, able to find and print map to Mars hotel.  Major information given visually.



A local person with good local knowledge and good IT /web skills but no experience of hotel booking. Person with mobility difficulties (wheelchair and arthritis in hands and wrists) Uses library PC to book a hotel room for a friend

Browses for hotel in the 80-100 per night range and books hotel

Able to find and book Gaia hotel, able to find and print map to Gaia hotel. Height and angle of PC suits needs. Keyboard and pointer device easy to handle and control without pain.



A tourist officer with little web experience, good local knowledge, previous experience of hotel booking, no special needs, at own PC, takes enquiry and booking from tourist in office, without being interrupted by phone calls

Browses for hotel in the 40-60 per night range and books hotel

Able to find and book Saturn hotel, able to answer additional questions, provide map to Saturn hotel.


18, 21, 22, 25, 29

A tourist whose child has an allergy to animals, no pets, little web experience, and no local knowledge, her own PC (IE4) to book a no-smoking room

Browses for a 4 star hotel and books it

Able to book a no-smokers room for self and child at the 4 star Cathedral Hotel which has a no pets policy and check for allergy aware hotels

Note 1 - table could alternatively be drawn up showing all the partitions tested by each case thus allowing the tests to be used in any order

Note 2: a similar table of tests would be drawn up for the other tasks


Tests are also drawn up using use cases. An example use case is shown in Figure 4 on page 4.

Figure 4 Test Scenarios

Scenario name:

Enquiry for hotel


Blankchester web site


Date: 12-1-2000

Goal of test

The goal of this use case is to test the basic enquiry job, where a (potential) tourist is searching for accommodation




This is core functionality, so this is an important test of  a frequent usage function


Any user may use this function :

Tourist - low web user, Tourist - high web user, BCC Tourist officer (trained), Local person - low web user, Local person - high web user






The PC is on and functional

The user is connected via their normal ISP

The user is online and looking at the web site home page and has chosen to use the drop down selections



Description of main flow

2 star in walking distance of cathedral

1.        the user clicks on Hotels

2.        the user clicks on 2 star

3.        the user clicks on location - cathedral

4.        the user clicks on Find



Expected results

the system displays 2 hotels details - "Wardens House" and "Deans House" and the user can print or save the information



Alternative flows

The user wants to see 3 star hotels - all in Blankchester

1.        the user clicks on Hotels

2.        the user clicks on 3 star

3.        the user clicks on find



Expected results

the system displays a drop down list with all 3 star hotels (15 on test database)



Failure conditions

No booking conformation number displayed

Wrong rating hotels displayed



Additional attention points

Note response times




With the constraints on local authority budgets, it is decided not to use a full usability lab test, but instead to use the "Thinking Aloud" protocol with an observer logging what the user does, using the use cases and selecting the user group from the equivalence partitioning tests already defined.  After the tests are run, the users will be asked to complete a SUMI questionnaire.  Representative users will be asked to complete typical tasks, and measures taken of effectiveness, efficiency and satisfaction.  It was expected that all Tourist Office users could successfully complete a hotel booking in an average time of less than 5 minutes.  All SUMI scores should be above the industry average of 50.  


Context used for the Tourist Office test: For this part of the test, eight typical users from the Tourist Office were selected who had the key characteristics and capabilities, but no previous experience of the web site.  The other characteristics of the participants that might influence usability were recorded, together with the age group and gender (see Figure 5 on page 4):

Figure 5 User Group Characteristics



Time in job (years)

Windows experience (years)

Internet / web site experience (years)

Attitude to computers/ web


Age group

























































*1=prefer to use a computer as much as possible, 7=prefer to use a computer as little as possible

The intended context of use for this test was the Tourist Office, and so the tests will be carried out there, but it is arranged so that participants could work alone without any interruptions. They are to be observed by an observer/tester. Rules for carrying out the tests are agreed: participants would be shown the evaluation suite, informed that their interaction would be recorded, they would be asked to sign a release form, asked to confirm the information they had provided about, and to score their attitude towards use of computers.  Tasks were to be timed using a stopwatch.  Sessions would not be videotaped but the user asked to use "Thinking aloud" so that the observer/tester could take notes.  At the end of the sessions, participants would complete the SUMI satisfaction questionnaire.

Instructions for user testers would be written, for example:

Figure 6 User instructions

Thank you for helping us in this evaluation. The purpose of this exercise is to find out how easily people like you can use the city web site. To achieve this, we will ask you to perform some tasks, and your performance will be observed.  Then, to help us understand the results, we will ask you to complete a standard questionnaire, and to answer a few questions about yourself and your usual workplace. The aim of this evaluation is to help assess the product, and the results may be used to help in the design of new versions. Please remember that we are testing the software, not you.


When you have finished each task, or got as far as you can, please phone us by dialling 1234.  I am afraid that we cannot give you any assistance with the tasks.


You have just been shown the Blankchester web site.   You now need to find out whether it could meet your current needs.  You will perform the following tasks:

1.       take some time to familiarise yourself with it and specifically the hotel "find and book" functions,

2.       book a hotel room meeting your requirements

 We are interested to know how you go about these tasks and whether you find the software helpful or not.   LET US KNOW WHEN YOU ARE READY TO BEGIN

Task 1 Familiarisation period - Spend as long as you need to familiarise yourself with the web site. (YOU HAVE UP TO 20 MINUTES)  LET US KNOW WHEN YOU ARE READY TO MOVE ON

Task 2 Book hotel room (YOU HAVE ABOUT 15 MINUTES FOR THIS TASK) Use the software to book a hotel room: DATE: 23 November 2001, PLACE: The Blue Flag Inn, Blankchester. LET US KNOW WHEN YOU HAVE FINISHED


After the tests, the mean completion rate, mean goal achievement, mean task time, mean completion rate efficiency and mean goal achievement efficiency would be calculated for the task "booking a hotel room".  After the last task, participants would be asked to complete a subjective ratings scale and the SUMI questionnaire.  The evaluator would then asked them about any difficulties they had encountered.  Metrics to be gathered were decided as:

         Effectiveness: Completion Rate Percentage of participants who completed each task correctly. Mean goal achievement Mean extent to which each task was completely and correctly achieved, scored as a percentage.

         Efficiency: Task time: mean time taken to complete each task (for correctly completed tasks). Completion rate efficiency: mean completion rate/mean task time. Goal achievement efficiency: mean goal achievement/mean task time.

         Satisfaction: measured using the SUMI questionnaire, at the end of the session, giving scores for each participants perception of: overall satisfaction, efficiency, affect, controllability and learnability. 


Note: SUMI provides a set of statistical tables and graphs these are not shown here but can be viewed in the web-sites in the references.


In addition to these user tests, the following accessibility checks are carried out:

          The web site design is checked against the WAI guidelines (inspections against checklist)

          The web site is checked against the RNIB guidelines (inspections against checklist)

          The web site is checked against the Bobby tool ( for suitability for use with a text browser


The usability tests are to be documented using the Common Industry Format described in ISO9126.

A.      Other techniques and sources of information

A1.               Techniques listed in the literature and on the web

We do not have space here to cover more of the techniques available.  Other techniques are:

Type of technique




Contextual Inquiry

Ethnographic Study / Field Observation

Interviews and Focus Groups



Journaled Sessions

Self-reporting Logs

Screen Snapshots

These are a selection of methods that gather information as a system is in use, either by observation of the user, or by asking the user to comment.


Heuristic evaluation

Cognitive Walkthroughs

Formal Usability Inspections

Pluralistic Walkthroughs

Feature Inspection

Consistency Inspection

Standards Inspection

Guideline checklists

These are all variations on Review, walkthrough and inspection techniques, with specialised checklists or specialist reviewers.


Thinking Aloud protocol

Co-discovery method

Question asking protocol

Performance measurement


These are techniques which help the user and the usability tester/analyst to discuss/discover how the user is using and thinking about the system

Related Techniques

Prototyping :Low-fidelity / High-fidelity / Horizontal  / Vertical

Affinity Diagrams

Archetype Research

Blind voting


Education Evaluation

Although not primarily usability techniques, these are techniques that some writers on usability have recommended. They are all ways to elicit reactions from users.


A2.               Sources of information

Usability research continues and information in this draft is based on drafts of standards ISO 9126 and ISO 14598, a workshop run by Improve QS bv , material from the following web sites and documents:


ISO TR 16982 is somewhat academic, but should be referenced for more information on usability methods.

ISO TR 16982

To relate usability and usability testing to the life cycle see for example see the ISO 13407 on treatment of evaluation.

ISO 13407

For the distinction between evaluation of paper prototypes, machine prototypes and usability testing see the SERCO website.


James Hom's web site with a list of usability design and test methods, very extensive book list and list of other web sites

Human Factors Research Group - SUMI, WAMMI and other information

Golden section explained (aesthetics)

Human Computer Interaction Resource Network

Human Factors Research Group UCC


SUMI background reading

Publications list

The interface hall of shame


SERCO usability web pages

IN USE handbook

SERCO Usability requirements web page

A3.               Accessibility

We have not in this example covered the design methods and the test methods for website accessibility.  Readers are strongly recommended to use the following resources for information the reasons for designing with accessibility in mind, or tools, checklists and hints for designing and testing for website accessibility:

For Usability and accessibility standards and guidelines go to the Testing Links page.