PA 5.2 SG 2 – Testing is performed using statistical methods

Tests are designed and executed using statistical methods (e.g. sampling, fault seeding) based operational or usage profiles.

Specific Practices

SP 2.1 Develop operational profiles

There are many types of operational profiles. Not all of them may be needed for any particular project:

  • User profile: the set of user groups and their associated frequency distribution.
  • System mode profile: a set of functions or operations grouped in order to analyse execution behaviour.
  • Functional profile: a quantitative view of the relative use of each of the different system functions (per system mode).
  • Operational profile: A series of steps needs to be performed for developing the final operational profile using information from the profiles already developed, for example:
    • Dividing the execution into runs
    • Identifying the input space (a comprehensive list of input variables)
    • Partitioning the input space into operations
    • Determining the occurrence probability for operations

SP 2.2 Generate and execute statistically selected test cases

Test cases will reflect probabilities in the usage model or operational profile and represent a sample of the input space according to the usage patterns

SP 2.3 Apply statistical test data to make stop-test decisions

Measurements will be made to determine if testing is representative of actual use. Only when testing is satisfied that the tests are sufficient to simulate expected operation in the field, can they use the test results along with other data to help make decisions about stopping testing.

Deployment notes

  • Are the tests based on operational or usage profiles?
  • Are requirements examined to understand the user profile of the application being tested?
  • Are measurements made to determine if testing is representative of actual use?
  • Are the statistical samples used to develop conclusions about the entire population of customers and uses, for example, using reliability models?