Prepared for EuroStar (2003), Amsterdam, Netherlands

By Deborah Abratt, Wayne Mallinson and Antonet Bekker

Test and Data Services (Pty) Ltd

Physical Address:   285 Cradock Ave, Lyttelton, Pretoria, 0157, South Africa.

Postal Address:   P.O.Box 14138, Lyttelton, Pretoria, 0140, South Africa.

Telephone:  +27 12 664 6475

Fax:   +27 12 664 6495

E-mail:   info@testdata.co.za

Website:  www.testdata.co.za

1. Non-Functional Testing

When striving for software quality, it is important to take a holistic approach. Although non-functional system testing is as important as functional system testing, it is often overlooked and not well specified. The BS 7925-1 (1998) standard defines non-functional system testing as “testing of those requirements that do not relate to functionality.” In other words, non-functional system testing analyses those aspects of a system that are measured on a scale and cannot be verified as valid or invalid. These “subjective aspects” of software quality (Roth, 1999) are more difficult, although not impossible, to define and measure and require specialist skills to test.

 The following are examples of non-functional testing types (BS 7925-1, 1998):

·         Performance Testing – testing conducted to evaluate the compliance of a system or component with specified performance requirements

·         Stress Testing: – testing conducted to evaluate a system or component at or beyond the limits of its specified requirements

·         Security Testing – testing whether the system meets its specified security objectives

·         Storage Testing – testing whether the system meets its specified storage objectives

·         Usability Testing – testing the ease with which users can learn and use a product

Testers should be aware of non-functional system attributes so that they can provide worthwhile feedback on these attributes to the rest of the development team.

2. What is Software Usability?

According to ISO 9241-11 (1998), usability is the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” Important aspects of this definition include:

·         “specified users”: It is important to note that when systems are being designed for usability, the first step should be to identify target user population. Usability is not an absolute term but, rather, a relative one. A system can only be usable relative to the user population it serves.

·         “specified goals”: The functionality designed into a usable system will be relevant to its users. It is often the case that unnecessary or inappropriate functionality is incorporated into a system. This functionality can “clutter” the interface and make relevant functionality more difficult to access. On the other hand, a usable system presents its users with routes to achieve their goals in a clear fashion.

·         “effectiveness”: Effectiveness is the accuracy and completeness with which users achieve specified goals. For example, a user is effective if they can complete tasks making a minimal amount of errors.

·         “efficiency”: Efficiency is the resources expended in relation to the accuracy and completeness with which users achieve specified goals. For example, a user is efficient if they are able to achieve their goals quickly (saving time) or cheaply (saving money). Examples of ways that usability allows users to be efficient are minimizing the number of user keystrokes or giving users clear paths to achieving their goals.

·         “satisfaction”: Satisfaction is the freedom from discomfort and positive attitudes towards the use of the product. For example, a user is satisfied if they can achieve their goals with a positive frame of mind. The user may also prefer one particular system to other systems.

·         “context of use”: The context of use constitutes the broader framework in which a product is operated. It concerns the system’s particular users, their tasks and the system’s broader environment of use. An example of a context of use could, therefore, be “a novice user buying a book on the Web in his home” or “an expert air traffic controller directing planes from a darkened control room.” It is very important to consider context of use when designing for usability because a usable system in one context of use may be completely unusable in another.

The discipline most concerned with the implementation of computer system usability is known as Human-Computer Interaction (HCI). As defined by ACM SIGCHI (Association for Computing Machinery Special Interest Group in Computer-Human Interaction) (1992), HCI is “the discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and the study of major phenomena surrounding them.” HCI is a diverse multidisciplinary field, containing theoretical principles based in disciplines such as Ergonomics, Psychology and Computer Science. HCI’s main focus area is the system-user interface – those aspects of the system with which the user comes into contact. The system-user interface is the one of the most critical areas of a system under development. As Jef Raskin, the creator of Macintosh comments, “As far as the customer is concerned, the interface is the product” (Raskin, 2000).

 3. What is Usability Testing?

As mentioned above, usability testing refers to evaluating the ease with which users can learn and use a product. The scope of this phrase, however, can mean different things to different people. Some people refer to usability testing as only the process that employs participants who are representative of the target population to evaluate the degree to which a product meets specific usability criteria (Barnum, 2002). Other practitioners use the term more globally to refer to any technique used to evaluate how easy a product or system is to use. The authors tend to concur with the latter definition because our research has shown that participant involvement is a desirable, but not essential, part of evaluating all aspects of system usability.

There are two main categories of usability testing: usability testing with users and usability testing without users.

3.1 Usability Testing with Users

This category of usability testing involves both usability testers and representative users. Users have specific characteristics that differ from person to person. Examples of these are their task and product skills, knowledge and experience, training level, input abilities as well as various physical and psychological characteristics. To take account of these specific user characteristics, one has to test usability with the direct involvement of the users themselves. Examples of usability testing techniques involving users are:

·         Performance Evaluation: Performance Evaluation involves measuring users’ performance while they perform tasks on a system or prototype. This technique often occurs in a formalized usability laboratory. Performance measurements (e.g. task time and error rate) serve as quantitative criteria with which to evaluate the system.

·         Card-Sorting: Card-Sorting is a technique for exploring how users categorize and find informaton. It involves sessions where users sort cards containing names of topics of relevance to the system under development. This information helps develop systems with a maximized probability that users will understand its structure and find what they are looking for.

·         Field Studies: Field Studies involve observing users using a product in its actual environment of use. Depending on the product, this could be in a user’s home, office, factory or even store. Field Studies are most useful early on in the design process.

3.2 Usability Testing without Users

This category of usability testing involves usability testers alone, without the involvement of users. This is form of testing is important because, apart from users having specific characteristics that differ from person to person, they also have general characteristics that are common to most people. These are human performance factors that are independent of a user’s age, gender, cultural background or level of expertise. Examples are the way we see the world around us, how much we can remember at once and certain norms of society.

According to Nielsen and Mack (1994), usability testing without users is widely used because:

·         It is inexpensive

·         It does not usually require special equipment

·         It can be integrated easily into the product development lifecycle

·         Problems and recommendations are generated without delay

Examples of usability testing techniques without users are:

·         Expert Review: The Expert Review is a comprehensive usability guidelines audit. Important screens are sampled and inspected according to compliance with a list of thorough usability guidelines. This gives insights into how to enhance user productivity and satisfaction.

·         Competitive Analysis: Competitive Analysis is an invaluable testing technique for products that operate in a highly competitive industry. It evaluates the usability, not only of a company’s product, but also of equivalent products of key competitors.

·         Keystroke Level Model Analysis: Keystroke Level Model Analysis is a methodology for evaluating users’ data entry efficiency. This not only helps the development team identify specific aspects of a task that are slowing users down, it also can be used to compare different designs to ascertain which would allow the user to work more efficiently.

4. Cost-Benefit Value of Usability Testing

The basic assumption of a cost-benefit analysis of usability strategies is that a usable system will result in tangible, measurable benefits. There are many different areas where usability strategies can have cost-benefit value. Four of these are discussed in the forthcoming sections:

·         Development – how usability results in optimization of the development process

·         Technical Support – how usability results in technical support savings

·         Sales – how usability increases product sales

·         Use – how usability positively impacts user performance and satisfaction

4.1 Development Benefits

Usability techniques can impact positively on a company’s development process in a number of ways:

·         Reduced development time: A user-centred design approach ensures that the needs of users are built into the product development cycle. This, in turn, ensures that only relevant functionality is designed into a product – no more, no less. The user-centred design approach combats the common industry misconception that “the more features a product has, the better”. An explosion of features in a system can be highly detrimental because extra features can clutter an interface and make it more difficult to find useful features. Features are also expensive to develop and, if they are not used, developing them becomes futile.

·         Reduced rework: If usability testing is done early in the design life cycle, changes may be made to eliminate problems and achieve benefits much more cheaply because code has not been written. Thus, changing and fixing usability problems early in the development life cycle can substantially reduce rework costs later on.

Development Case Study – Israel Aircraft Industries (Bevan, 2001).

Company: Israel Aircraft Industries (IAI), a company that designs and builds aircraft and avionics equipment

Project Leader: Serco Usability Services led the project, as part of the Trial Usability Maturity Process (TRUMP) project (part-funded by the European Commission). Serco Usability Services is a London-based user-centred design consultancy.

Project: The project involved the development of a new Mission Planning Centre. The Mission Planning Centre enables a pilot to plan an airborne mission that is then loaded onto a cartridge and taken by the pilot to the aircraft. In the aircraft, the pilot loads the data into the aircraft’s main mission computer.

Project Objectives: The following were the company’s main project objectives:

·         Improve the operational requirements definition and evaluation process

·         Increase usability of IAI products

·         Increase customer satisfaction from IAI products

Techniques Used: The following are some of the key usability techniques that were used:

·         Usability maturity assessment: The usability maturity of the organization was assessed using the Usability Maturity Scale (INUSE, 1998) to identify which user-centred methods were needed.

·         Context of use analysis: A long checklist covering many aspects of the user’s skills, tasks and the product’s working environment was used.

·         Evaluation of the usability of existing system: Four users evaluated the existing system. Each user was given a mission to prepare and commented as he went along. Comments were captured by the facilitators generating a detailed list of about fifty problems. The users filled out satisfaction questionnaires after the evaluation.

·         Setting usability requirements: Goals for task time were agreed on and a list of potential user errors was compiled.

·         Testing usability against requirements: The system was tested against timing requirements defined for two typical tasks. Eight pilots participated in the test and filled in the satisfaction questionnaires.

Outcome: IAI estimated that the methods used resulted in savings in development costs of between $5,000 and $70,000 for each method, with a total saving of $330,000. The cost of using the methods was only $22,000, giving a cost-benefit ratio of 1:15.

 4.2 Technical Support Benefits

Usable systems decrease technical support costs in a number of ways:

·         Decreased user training: Many organizations spend large amounts of money training users on new software systems. When usable systems are designed, they become more intuitive and users require less training on the system before can they use it.

·         Decreased help-desk calls and on-site technical support: Usability decreases the burden on a company’s help-desk and on-site technical support staff because users do not require assistance addressing usability problems.

·         Decreased unofficial technical support: Users often choose to disturb non-technical colleagues to help them solve usability problems. When a system is more usable, these disturbances are less likely to occur.

Technical Support Case Study: Ford Motor Company (Bias and Mayhew, 1994)

Company: The Ford Motor Company, a leading consumer company for automotive products and services

Project Leader: The Ford Motor Company

Project:  The Ford Motor Company’s accounting system for their small car dealerships.

Project objectives:  The main project objective was to improve the effectiveness, efficiency and satisfaction of users of Ford’s accounting system.

Techniques Used: User testing was conducted in a usability lab.  A usability lab usually consists of a main room where the user performs a list of tasks, an adjacent room where the testers are located, and a one-way mirror joining the two rooms.  Testers can observe the user performing tasks through the mirror.  There are also cameras recording the user’s performance.  The testers can observe what the cameras record on monitors in the testers’ room. The user’s level of performance is measured during the test, for example the number of errors made, the number of tasks completed and how long the user takes to perform tasks. The users are also often asked for their opinions regarding the usability of the interface.

Outcome:  Through the usability study, the following problems were discovered:

·         Car dealers needed to make an average of three calls to the help-line to be able to start using the system.

·         The commands used to enter credits and debits were designed by the engineers without first consulting users to learn the commonly used abbreviations.

Ford changed the abbreviations as well as 90% of their accounting system. The new system was so easy to use that the calls to the help-line dropped to zero. It was estimated that this new version saved the company $100,000. Since the cost of the usability testing was $70,000, the company gained a cost-benefit ratio of 1:1.4.

4.3 Sales Benefits

Usability strategies have a positive effect on company sales revenues. This is due to a number of reasons:

·         Competitive edge of marketing usable products: Usable software can accrue increased revenues from external customers due to increased marketability. An area where this is particularly critical is that of e-commerce. On the Web, instead of buying software first and then experiencing its level of usability later, customers encounter usability first and then decide whether to pay or not. In addition, in e-commerce, the company’s website becomes the crucial primary interface to the customer.

·         More satisfied customers: When users suffer frustration caused by unusable software, the company’s credibility is tarnished. In contrast, satisfied customers have brand loyalty and are much more likely to buy the same brand in the future with less research into similar products.

·         Higher ratings for usability in the press: Usability is one of the main criteria used to critically rate software packages in commercial publications. A wide variety of trade magazines evaluate new software releases that are sent to market. According to Nielsen (1993), approximately 15% of reviews by trade journals are devoted to analyzing the “user-friendliness” of new software products.

Sales Case Study: Website Redesign (Nielsen J, 2003).

Companies: 42 anonymous companies

Project Leader: The Nielsen-Norman Group, a leading US-based usability consultancy

Project: The Nielsen-Norman Group analyzed data from 42 cases where usability metrics were available for website designs.

Project Objective: The main project objective was to quantify the cost-benefit value of usability redesign on websites.

Techniques Used: For this analysis, the Nielsen-Norman Group used projects that collected the same metric both before and after the redesign. This allowed them to accurately compare them and estimate the percentage improvement in usability. Each of the websites was redesigned according to well-known usability design guidelines.

Outcome: The improvement in usability metrics differed depending on the metric, as the following table (Table I) shows

Table I. Results of website redesign study by the Nielsen-Norman Group

The results indicate a remarkable increase in number of sales or conversion rate (a site's ability to "convert" visitors to buyers) of 100%.

4.4 Use Benefits

The last area where usability strategies can have a major financial impact is in terms of user performance and satisfaction. Examples of these are:

·         Reduced user task time: Optimizing user efficiency relates to cutting down unnecessary work-hours resulting in potentially large savings to companies. The benefits due to improved user efficiency are particularly evident for high-volume, repetitive tasks, such as data capturing.

·         Reduced user errors: Users will not make as many errors on a usable system because they have a better understanding of the system.  Typical error-prone situations can be “designed-out”. This, of course, relates closely to the previous benefit of improved user task time.

·         Improved user satisfaction: This relates to higher levels of staff morale, which have been shown to reduce staff turnover and increase job motivation and company loyalty.

Use Case Study: The Pick ‘n Pay Group

Company: The Pick 'n Pay Group, one of Africa's largest and most consistently successful retailers of food, clothing and general merchandise for the past three decades

Project Leader: Test and Data Services, a leading South African software testing company

Project: The system tested was Pick ‘n Pay’s core Enterprise Resource Planning application.

Project Objectives: The main project objectives were:

·         to estimate current user data entry efficiency

·         to make design recommendations to increase user data entry efficiency

 Techniques Used: Tasks were reviewed in terms of data entry efficiency using the Keystroke Level Model (Card et al., 1980). This method is used to predict the time an expert user would take to complete a task on a user interface. Time estimates are attained by breaking down tasks into sub-units called operators. There are seven main types of operators possible when a user performs any task (see Table II). These predictions are based on a study involving extensive tests with users.

Table II. The operators and estimated times used in the Keystroke Level Model (Card et al., 1980)

In this project, two frequently performed tasks were analyzed. The results from the analysis of one of these tasks (“Capturing an Order”) are detailed below. This task is performed in a manual receiving environment after goods have been received at the delivery area and a hard copy of the order with its invoice has been manually written. The capturing clerk then captures the order onto the system and specifies, among other details, what goods were received, in what quantity they were received and when they were received.

Outcomes

Data Entry Method 1: Current design – using mainly the mouse

This analysis estimates how long it would take an expert user to capture an order while using the current system design which mainly uses the mouse for data entry. Table III shows the summarized calculation. The total estimated task execution time is 71.81 seconds.

Table III. The operators and estimated time for capturing an order using the current interface design and mainly the mouse for data entry

Data Entry Method 2: Improved design – using only the keyboard

The second analysis estimates how long it would take an expert user to capture an order while using an improved system design that uses solely the keyboard for data entry. From the summarized results in Table IV, it can be seen that the total task time is 33.15 seconds. This is less than half of the time it takes to perform the identical task using mainly the mouse (71.81 seconds).

Table IV. The operators and estimated time for capturing an order using an improved interface design requiring use of only the keyboard

 To illustrate the organizational impact of improving task efficiency, one can estimate the amount of working time taken when performing different forms of data entry. According to in-house company statistics, an average store captures about 450 orders per store per month. Figure 2 shows the estimates for the amount of hours per month the different types of data entry would take. From these results, one can see that by performing a few simple usability design improvements to the interface stores have the potential to improve order capturing efficiency by 56%.

Figure 2. Estimated hours per month per store spent capturing orders using different data entry methods

5. Usability in Developing Countries

In a recent article in Interactions, Dray et al. (2003) comment, “Technology can knit the world together or it can create a schism”. This is a very pertinent statement, particularly for developing countries such as South Africa (where the authors are based). In terms of a developing country, information technology can have a positive or negative contribution to economic development. On the negative side, the proliferation of technology in developing countries has been blamed for widening the gap between the rich and the poor. This takes the form of a “digital divide” between those that do and do not have access to technology.

On the positive side, developing countries can take advantage of information technology to achieve economic gains. Technology can assist in supplying basic education, improved health services and communication networks. It can also allow local businesses to market their products online to a global market. This has occurred in countries such as South Korea, Taiwan and Singapore where investments in technology have allowed these countries to establish themselves as technologically self-sufficient. These countries now export home-grown technology to other nations. HCI is one of the most important disciplines in making sure that information technology is used in a positive manner within developing countries. By designing systems to suit individuals’ unique economic, political, social and cultural identity, technology can be a powerful catalyst towards national economic empowerment.

6. Conclusion

Investment into software usability testing can offer significant returns. Examples of these are in the areas of development costs, technical support costs, sales and user productivity. These benefits can only be realized through rigorous usability testing of software throughout the development process, whether it is by testers alone or with user involvement. It is inevitable that as the software testing industry matures, non-functional testing types will be given higher priority. This holistic approach is essential in delivering products that are not only functionally correct, but that also support their users.

7. References

ACM SIGCHI (1992). Curricula for Human-Computer Interaction.

Barnum C (2002). Usability Testing and Research. New York: Pearson Education.

Bevan N (2001).  Introducing User Centered Design methods at IAI from www.usability.serco.com.

Bias R and Mayhew D (1994) (eds). Cost-Justifying Usability. San Diego: Academic Press

BS 7925-1 (1998). Software Testing. Vocabulary.

Card SK, Moran TP and Newell A (1980). The Keystroke-Level Model for User Performance Time with Interactive Systems. Communications of the ACM, 23(7):396-410.

Dray S, Siegel A and Kotzé P (2003). Indra's Net: HCI in the Developing World. Interactions, 10(2):28-37

INUSE (1998). Usability Maturity Model: Human-Centeredness Scale, Lloyd’s Register project IE2016 INUSE Deliverable D5.1.4s.

ISO 9241-11 (1998). Ergonomics requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability.

Nielsen J (2003). Return on Investment for Usability

Nielsen J (1993). Usability Engineering. Boston: Academic Press.

Nielsen J and Mack R (1994). Usability Inspection Methods. New York: John Wiley & Sons.

Raskin J (2000). The Humane Interface. Boston: Addison-Wesley

Roth A (1999).  Using the Iced T Model to Test Subjective Software Qualities. Proceedings of the 1999 International Conference on Software Testing, Analysis and Review held in San José, California.