This example is for review for review at the meeting on 8th April, with comments required by 3rd April. All comments should be submitted using the feedback form.
Procedure Testing
Analysis
Procedure testing shall model the procedural requirements of the software system as a complete and delivered unit. Procedure Requirements shall define what is expected of any procedural documentation and shall be written in the form of Procedural instructions. These procedural instructions will normally come in the form of one of the following documents:
· A user Guide
· An Instruction Manual
· A User Reference Manual
This information will normally define how the user is meant to:
· Set up the system for normal usage
· Operate the system in normal conditions
· Become a competent user of the system (tutorial files)
· Trouble-shoot the system when faults arise
· Re-configure the system
Design
There shall be 2 types of testing carried-out when performing procedural testing: Static and Dynamic
Static testing of the procedural instructions themselves should firstly be carried out. This would include an assessment of the system, examining things such as, set-up, main areas of operation, complex areas of operation, tutorial file examples, trouble-shooting, etc. The result of the assessment would group together a series of procedural instructions thought to be a requirement of the end-user in order to use the system effectively. These would then go through a series of reviews that would include the end-user, or a representative of the end-user. This purpose of these reviews would be to: assess the importance of the procedural instruction to become part of the manual, it's usefulness to the end-user and the degree of its ability to be understood by the end-user.
Dynamic testing of the system shall be conducted using Test Cases. Primarily Test Cases shall be guided by the procedural instructions with the aim of ascertaining whether the procedural requirements have been met. Test Cases shall be designed to exercise the procedural instructions of the system under specified conditions.
For each test case the following shall be specified:
1. Set-up of the system
2. The specific procedural sequence to be exercised
3. The expected outcome of the system
4. The expected outcome of the User, i.e. what the user has achieved/understood.
Measurement
Procedural testing shall be measured in the following ways:
Static testing – shall be measured as the percentage of the total specified procedural requirements, which have been covered by procedural instructions, reviewed.
Dynamic testing – as a percentage of the specified procedural requirements which have been executed.
 |
Figure 1 - Procedure testing Flow diagram
| Comment from Margaret Edney | My
understanding of procedure testing is that it verifies the interactions
between the manual and automated processes within a system. This
definition looks at the system very much from a user's point of view. The
emphasis seems to be on reviewing the use of the system, rather than
verifying the interfaces between the user and system. Perhaps the
definition needs to cover both views.
I agree with Erik's suggestion on including Process cycle testing, certainly within the individual domain examples. |
| Comment from Mark Robison | As we're
aware, there are a number of different definitions
of a procedure. One of the simplest seems to be that
of the isogroup.
"A description of tasks that realise a process." Procedures in turn have a series of streams often known as workflows." Procedure testing is the discipline of targeting these various workflows in order to demonstrate they meet the defined requirements.It is suggested that this testing has two key elements, the functional aspect and the non-functional. The procedures functional aspects focus on the basic integrity of the procedure and confirm it meets the defined objective.However, the non-functional aspects are linked to the numerous other characteristics covered in other non-functional techniques.Most would agree, the most challenging aspects of procedure testing are the rather subjective characteristics such as useability. Often a procedure is given a clean bill of health only to prove utterly worthless when released to customers in the field.This rather laboured background is given to make a point about the technique under review. It is suggested that it is incredibly important to mention the links to the other non-functional techniques in this description.Analysis ---------- This section does not cover the analysis of the procedures in question. It gives a good overview of types of document centric procedures but does not say how you would analyse such procedures for the static and dynamic test design.How does an organisation decide what its testing mission is when tackling procedures ?
Design -------- How are processes selected for static or dynamic testing ? OR is the implication that both are required ? Some think the static test reviews with the end users are too late and that these sorts of reviews are better employed when defining/modelling requirements. To review procedures for larger systems in this way would also be costly and time consuming.Measurement --------------- I wonder how we would use requirement coverage for many of the non-functional procedure type requirements ? My experience with the definition of NF requirements is that they often lose their 'testability' characteristics and we are left trying to design tests but are actually establishing a more detailed requirement layer.Again I think we need the link to other techniques such as Usability and Maintainability.We could possibly look beyond basic coverage metrics, possibly to more refined statistical techniques.Some possible non-functional relations (I haven't had time to read all techniques, so I apologise for the losedefinitions :) Usability - Are the procedures understandable, learnable, convenient ....... Portability/Configuration- - will the procedure operate in a number of different environments or geographical locations.Maintainability - -Can the procedure be refined or amended without wholesale changes to workflowsCompatibility - Do procedures reflect product compatibility requirementsEvolvability- - has the procedure been prepared to facilitate future development or possible adaptations.Efficiency - -Do the procedures reflect a systematic approach to the tasks .... |
| Comment from Isabel Evans | Procedure -
definition - agree with Erik's comment. Important also to add testing of
the manual business procedures around the software. This could be for
example:
for a conversion/migration project - dress rehearsal of the manual check procedures and sign off around the s/w processes for a business processes - dress rehearsal of the manual processes around the software system These business / manual processes can be tested by e.g. using control flow diagrams, and by the test methods Erik mentions |
| Comment from Marco Giromini | “Analysis”
1st line: Procedure testing shall “verify” the procedural requirements
…
“Analysis – 2nd list” I suggest to add the following items: - recovery and restart the system (when failures arise) - control systematic failures to achieve necessary risk reduction, e.g. Failure detection by on-line monitoring, Input acknowledgement (these techniques and measures are based on procedure instructions [IEC 61508]). “Dynamic testing”: Shall be measured “as a percentage of the specified procedural requirements which have been executed” by any test case. I suggest to add: Procedure testing shall be measured as the percentage exercised by static or dynamic testing, of the total specified procedural requirements. I think that specific testing techniques, such as Use Case testing, should be referred in the Procedure Testing Definition; while general testing techniques, such as Conformance testing, Boundary testing, Random testing, should be referred in the Procedure Testing Guideline. |
| Comment from Erik van Veenendaal | For the procedure testing standard at least a reference should be made to the Procescyclustest (as described in TMap) and to use case testing (as described in STQE). Both techniques are the ones that I teach and use very effectively within projects |