CMMI and testing

CMMI and testing

CMMI claims their models provide guidance for developing or improving processes that meet the business goals of an organisation. CMM and CMMI are often stated as the industry standard for software process improvement (and CMM did provide process improvement projects with process and structure).

However, only a limited portion of the CMMI model is devoted to testing. Given that testing can take up anything from 30 – 80% of total project costs (depending on the type of application and development method) you might expect the model to pay more attention to it.

At Level 3 of CMMI there is verification and validation for the testing process but these areas are covered at such a high level as to not be of much practical use. Verification (has the system been built correctly) and validation (has the right system been built) provide some more testing focus, but there is no step-to-step guidance to aid in improving the testing process.

The focus of CMMI is not so much on a mature testing process, but more on organisational and systems engineering processes – in other words, the bigger picture. The CMMI model validates software engineering processes to ensure that procedures are in place. It does not, however, ensure that the test process is effective, efficient and actually used by the organisation. So…

TMMi supports CMMI

TMMi complements CMMI and is a detailed model for test process improvement.

Comparison of TMMi and CMMI Levels 2 and 3 show:

  • TMMi covers test-related quality in much greater detail than CMMI
  • Nearly half of TMMi is not covered by a CMMI equivalent
  • Non-test Process Areas and practices that are elaborated in CMMI are mostly not repeated in TMMi, only referenced (see below for details).
  • To gain the most benefit, the higher levels of TMMi need strong CMMI-like processes across development
  • The value added by TMMi includes:
    • TMMi assesses testing more deeply than CMMI
    • Almost 50% of TMMi covers test elements not in CMMI

Some more detailed facts and figures

  • 33% of the Practices in CMMI levels 2 and 3 do not involve test at all
  • 66% of CMMI Levels 2 and 3 can be mapped to levels 2 and 3 of TMMi
  • 1% of CMMI is covered in TMMi Level 4

TMMi does not validate software engineering processes specifically, but it does ensure that:

  • the processes in place are used effectively across the organisation and
  • that test and quality processes are integrated with them where appropriate. For example, requirements definition, requirements management and configuration management.

TMMi is a reference model which examines the specific processes at every level above Level 1. A large part of CMMI level 2 looks only generically at a high level.

Looking at TMMi Levels 2 and 3 coverage:

  • 58% of TMMi Levels 2 and 3 concern test-specific practices in contrast to the generic overview in the CMMI Software Engineering context (for example, testing rather than project planning)
  • For each CMMI Specific Practice that includes test elements, there is an average of 2.6 TMMi Specific Practices
  • 42% of TMMi Levels 2 and 3 is looking at Specific Practices NOT covered by CMMI levels 2 and 3

TMMi Levels need specific support from Process Areas at its corresponding CMMI Level or from lower CMMI Levels. In exceptional cases there is even a relationship to higher CMMI Levels.

Non-test Process Areas and practices elaborated in CMMI are mostly not repeated in TMMi, only referenced. For example, the Process Area ‘Configuration Management’, which is applicable to test (work) products or test ware, is not elaborated upon in TMMi. Instead the practices from CMMI are referenced and re-used implicitly.

Important CMMI Process Areas

At CMMI Maturity Level 2 – Managed there are the following Process Areas

  • CM – Configuration Management
  • MA – Measurement and Analysis
  • PMC – Project Monitoring and Control
  • PP – Project Planning
  • PPQA – Process and Product Quality Assurance
  • REQM – Requirements Management
  • SAM – Supplier Agreement Management

The CMMI Process Areas in italics above are those representing processes and activities which are not covered by TMMi. However, they are highly important and necessary to achieve TMMi, as they support TMMi Level 2 activities (in italics below) which cannot run in their own right. Without the CMMI Level 2 Process Areas being achieved the TMMi Level 2 would also fail:

  • TMMi Level 2 – Managed
    • Test Policy and Strategy
    • Test Planning
    • Test Monitoring and Control
    • Test Design and Execution
    • Test Environment

Below are just some examples of CMMI Process Areas which are not specifically covered by TMMi:

  • CMM Level 2
    • PMC – Project Monitoring and Control
    • PP – Project Planning
  • CMMI Level 3
    • DAR – Decision Analysis and Resolution