Index Up Site Map Latest News Working Practices Discussion & Review Glossary Module Testing Non-Func. Testing Domains-Techniques Links Test Papers Feedback Administration

Installability Guidelines

Related sections

When using this section of the website, refer to the procedure testing and usability testing sections for techniques which may used for testing the installation procedures and the usability of the installation process:

          The procedure testing section discusses audit and reconciliation techniques which may be adapted for use in Installability testing, for example to see if a new version of a library has been overwritten by the installation with an old version with no warning messages displayed to advise the user that this would happen.

          The usability testing section discusses techniques which can be employed for measuring the usability of the installation process and for identifying the contexts of use, including the profile of the target installation team.

It may be necessary to test that applications that hitherto worked successfully still function correctly following the installation of other applications. Regression testing of this form is not covered by this section. The compatibility testing section discusses techniques for measuring the affect on existing systems and software of an installation, for example use of "before and after" software audits.

Modelling the Installability requirements

Installability testing uses a model of the Installability requirements, specified in terms of the Installation process (as described in the installation manual or guidelines), the people who will carry out the installation, the target platform(s) and the software/system to be installed. 

Ideally, the installability requirement will be defined as part of the overall requirements. If the defined installability requirements are incomplete or missing, the installability tester should consider and document possible attributes of the installation which are more or less desirable. From these attributes it is then possible to derive and validate the installation requirements, in order that tests may be built.  Success/acceptance criteria may also include a requirement to complete a subset of the functional tests to demonstrate, or measure, that the system has installed.

The installability requirements are specified in terms of acceptance or success criteria based on consideration of:

         the installation process for example for a particular installation process one or more of the following may be specified:

       the contents of the installation manual, for example that it should state any particular requirements such as minimum memory or processor speed. 

       the maximum number of installation steps

       time that the installation process is expected to take (including maximum acceptable / performance test)

       resources the installation process is expected to take (including minimum, maximum and optimum resources)

       any variations in the entry criteria, process and exit criteria for the installation process (for example for platform and regional variations)

       ability to back out of the install, including definition of the system configuration on completion of a back out, steps to back out

       ability to uninstall, including definition of the system configuration on completion, steps to uninstall

       installation media and whether remote or local installation

       tools and techniques required by the installer

       standards that apply to the installation

       dependencies in the process

         the people who will carry out the installation for example:

       their expectations

       the skills required by the installer(s), (may require some usability testing) e.g. experienced technician, technician - not trained in technology, end user, customer

       installation staff numbers

       installation guide requirements for example, suitability for audience, help for troubleshooting, media, understanding and meeting the installation team's expectations

         the configuration of the target platform(s), for example:

       existing software and hardware with version and functionality defined

       the state of the existing software and hardware (any applicable settings)

       the data, files, records and other stored information needed

       any security issues that apply to cover data integrity and system integrity such as virus checking, firewalls, safety, confidentiality, irreversible actions

       any potential conflicts (hardware and software)

       levels of system access e.g. root, administrator, restricted use

       system configurations, for example single workstation, network installation, internal, business to business, external, client, server, mainframe, single or multiple platforms

       maximum and minimum configuration of the target system (for example memory)

       configuration of target platform pre installation: including system configuration files, for a new installation on clean machine, new installation on machine containing other applications, reinstall or upgrade, de-install

       configuration of target platform post installation, including system configuration files, a new installation on clean machine, new installation on machine containing other applications, reinstall or upgrade, de-install, back-out

       dependencies in the configuration

       permitted upgrade paths, including new installation, upgrade from last version, upgrade from older versions, downgrade

         the type of software/system to be installed, for example:

       bespoke - internally developed application

       bespoke - third party developed application

       product - standardised installation

       product - customised installation

Test coverage measurement

Test coverage is measured as the percentage of Installability requirements exercised during the testing.  As a subsidiary measure, if formal test case design techniques are used, test coverage can be measured using the measurement techniques defined in BS7925-2.

Completion criteria and measurement of Installability

A decision will have been made when writing the test strategy as to what determines the acceptance criteria for the installation process. This enables the  tester to evaluate the results of testing to see whether the software / system has passed the installation test or not, and whether the testing is completed. The completion criteria of the installation testing could be measured in terms of the following:

          Were the acceptance criteria defined earlier measured satisfactorily?

          Were the results of the post-installation audit checked satisfactorily?

          After installation, is the installed software / system resident and run-able on all targeted hardware platforms?

          Were all the tests defined in test design executed and passed?

 Test case outline

The standard requires that for each test case, the following shall be specified:

          The start/entry state

          Which Installability requirement is being tested

          The actions and inputs

          The expected outcome

The installation tester may wish to document additional material in the test cases:

Test case parts:

May include:

The start/entry state

Starting condition

Entry criteria

Pre-requisites

Which Installability requirement is being tested

The test object

The test objective(s)

The scope of the test (what is to be included and what excluded in the testing work)

The actions and inputs

The steps to be used described in the description, text or form to be used by the installer

The expected outcome

Closing condition

Exit criteria

Acceptance criteria

Completion measurement

The method that is to be used to check that the exit criteria can be judged to have been met and the installation met installation requirements

The expected status of the installed system covering both hardware and software

Test case design

The standard requires that test cases are designed to exercise the Installability of the system under specified conditions. 

Techniques used to design the Installability tests include the techniques described in the procedure testing section and usability sections of this standard, together with formal test design techniques described in BS7925-2.

 Use of boundary value analysis

Boundary value analysis as described in BS7925-2 may be used to test the Installability.  For example, if one of the installation requirements is that the installation must complete within a certain time, the tester can push this time to its boundary by setting the test environment conditions:

Requirement for installation time: to be complete in less than 60 minutes

To design the test, the tester looks at the memory and processor requirements for the system, and selects a test platform with the minimum memory and processor requirements.  The tester also considers the traffic on the system during the installation and arranges for the test installation to take place with maximum traffic.  If the test installation completes within the specified 60 minutes in these conditions it has met the installation time requirement. 

Test case:

The start/entry state

Minimum memory (x Mb)

Minimum processor speed (xyz processor)

Maximum traffic (z transactions per minute)

Which installability requirement is being tested

Requirement for installation time: to be complete in less than 60 minutes

The actions and inputs

Install as described in the installation guide steps 1 to 16

The expected outcome

Install completes within 60 minutes

Use of branch testing

Branch testing as described in BS7925-2 may be used to test the Installability process.  The example in BS7925-2 uses code, but the method may used to analyse a procedural or process flow.  This technique may be used for test coverage and test case design where different options are available to the installer during the installation. For example, one of the installation requirements may be to allow the installer to select a country/location which will affect the remainder of the installation.

Requirement for installation location: to be able to select a country and currency. Countries are USA, UK,  Japan.  Currencies are US Dollar (ISO code USD), Euro (ISO code EUR), UK Sterling (ISO code GBP), Japanese Yen (ISO code JPY).  The countries and codes are related as follows

Country

Valid Currency settings

USA

USD

UK

GBP or EUR

Japan

JPY

The tester draws up a control flow diagram from the installation guide, to describe the choices within the installation. In the figure below, the steps are numbered 1 to 7:

1.       Initial installation processes

2.       Select USA

3.       Select Japan

4.       Select UK

5.       Select EUR

6.       Select GBP

7.       Complete installation

 

Figure: Installation choices:

 

 

 

 

 

 

 

 

 

 

 

Tests will be drawn up for routes through the installation.  To give a minimum coverage of branches according to BS7925-2, 4 tests would be required. Note that while this does cover the branches, it does not cover all the tests which the tester may need, to address risk or uncover possible faults.  A set of test cases to give minimum coverage are shown below:

Test case 1:

The start/entry state

System is ready for install

Which Installability requirement is being tested

USA with USD

Branch

1, 2, 7

The actions and inputs

Select USA as the country

The expected outcome

USD is auto-selected as the currency and language is auto-selected as US English. Location map on start up shows USA.

Test case 2:

The start/entry state

System is ready for install

Which Installability requirement is being tested

Japan with JPY

Branch

1, 3, 7

The actions and inputs

Select Japan as the country

The expected outcome

JPY is auto-selected as the currency and language is auto-selected as Japanese. The limited character set is used. Location map on start up shows Japan.

Test case 3:

The start/entry state

System is ready for install

Which Installability requirement is being tested

UK with Euro

Branch

1, 4, 5, 7

The actions and inputs 1

Select UK as country

The expected outcome 1

Offered chance to select currency - EUR or GBP 

The actions and inputs 2

Select EUR 

The expected outcome 2

EUR is set as the currency, language is set as UK English. Location map on start up shows Europe including UK 

Test case 4:

The start/entry state

System is ready for install

Which Installability requirement is being tested

UK with GBP

Branch

1, 4, 6, 7

The actions and inputs 1

Select UK as country

The expected outcome 1

Offered chance to select currency - EUR or GBP 

The actions and inputs 2

Select GBP

The expected outcome 2

GBP is set as the currency, language is set as UK English. Location map on start up shows UK

Use of scenarios and use cases

Tests may also be derived using scenarios.  In this case the tests are designed not by using one of the formal techniques in BS7925-2 but by considering how the installation process might be used by different installers or on different platforms.