Simon Frankish, Author at Experimentus

7 July 2022 by Simon Frankish Leave a Comment

How can you become a TMMi assessor?

In this series of posts I’ve spoken about what a TMMi assessment is and how they are carried out. In this, final, blog in the series I’ll look at how you can become a TMMi assessor.

It is important to note that the TMMi framework is a resource that is freely available to anybody. However, as with any process improvement framework, best results will be attained if there is a full understanding of the framework, its intent and nuances, and of testing itself.

To this end, the TMMi Foundation have developed a career path (see the graphic above) that includes two training courses, one examination and lists the other qualifications that are mandatory to become an accredited TMMi assessor or lead assessor.

To become a TMMi assessor you need to have;

An ISTQB Foundation Level certificate, or equivalent
TMMi Professional certification (i.e. have passed the TMMi Professional examination)
Taken a recognised TMMi Assessor training course
Applied to the TMMi Foundation for accredited TMMi assessor status and been accepted

In order to maintain your TMMi assessor status (which is valid for three years from date of accreditation) you need to demonstrate at least 50 hours of participation in assessments in the previous 3 years.

TMMi lead assessors must also have;

The ISTQB Advanced Test Manager certificate, or equivalent
Taken a recognised lead TMMi Lead Assessor course
Participated in at least 75 hours of TMMi assessments as an assessor in the previous year
Applied to the TMMi Foundation for accredited TMMi lead assessor status and been accepted

To maintain lead assessor status (which is also valid for three years) you would need to demonstrate at least 225 hours of participation in assessments in the previous 3 years – which must include all aspects of the assessment process.

Becoming a TMMi assessor puts you in a great position to help your organisation improve their test processes in a consistent and proven way.

Experimentus are the leading provider of TMMi training. You can see details of our TMMi Professional training here. Our assessor training (which includes lead assessor training) is detailed here. We can also provide mentoring from our vastly experienced lead assessors which will help you get up to speed quickly and make the most of your TMMi journey.

I hope that this blog series was enjoyable and we look forward to helping you with your test process improvement and TMMi needs.

30 June 2022 by Simon Frankish Leave a Comment

How are TMMi assessments done?

In this second blog on TMMi assessments I will outline the steps carried out to actually perform the assessment. A TMMi assessment is a relatively straightforward process, but it must pass through certain phases, as laid out by the TMMi Foundation’s test assessment method application requirements (TAMAR). These phases are;

Planning
- Agree the scope of the assessment (project, programme, organisation)
- Identify the maturity level being sought
- Plan the activities and deliverables
- Arrange interview times
- Prepare the assessment team and assessment tool
Preparation
- Provide list of artefacts to be reviewed
- Arrange any necessary access (to sites, people and tools/repositories)
- Ensure back up plans are in place
Interviews
- Conduct planned interviews (being realistic about how many can happen in a day)
- Ensure that interviewees are put at ease – this is not a test of their knowledge!
- Keep information from interviews confidential
- Request any artefacts mentioned during interviews for review if not previously supplied
- View applications / tools if required
Reporting
- Provide initial feedback to validate findings and obtain agreement
- Perform the final analysis and determines scores and the TMMi level indicated
- Formulate recommendations for follow up actions
- Create the report and/or process improvement plan
- Formally present the report and/or process improvement plan. The report will include;
  - A list of the inputs to the assessment (artefacts and people interviewed)
  - The findings from the documentation review and interviews
  - An overview of the assessment method used
  - An indication of the TMMi maturity level achieved
  - Optionally, a detailed process improvement plan

TMMi assessments are best carried out by an accredited TMMi assessor or lead assessor following an accredited assessment method as this will ensure consistency of approach and recommendations. If you are seeking certification to a TMMi maturity level the this is mandatory. Good assessors will be able to assess your processes not only against the TMMi model, but also taking into account the context of your organisation and development methodologies in use.

I hope that this overview of the process of carrying out a TMMi assessment has been interesting and useful. If you would like to know more, please take a look at our TMMi page.

In my next blog, I’ll talk about how you can become a TMMi assessor and use those skills to help your organisation improve its testing practice.

23 June 2022 by Simon Frankish 1 Comment

What is a TMMi assessment

The TMMi Maturity Levels and Process Areas

In my last series of posts I spoke about why test process improvement is important and how it helps organisations. In this series, I’d like to talk about the test process improvement framework, Test Maturity Model integration (TMMi) and how we use that to effect test process improvement.

Any test process improvement framework must provide a consistency of approach to assessment for all activities related to testing;

Be a standard reference model applicable to all industries and development methodologies
Identify testing strengths and best practices
Improve testing effectiveness and efficiency
Identify issues and risks
Identify a programme of test process improvements
Provide a detailed framework for test process improvement which is complementary to CMMi

And, in TMMi’s case;

Provide public governance, measurement and accreditation
Use maturity levels for process evaluation and improvement

TMMi uses the reference model to assess existing processes (we’ll see exactly how that is done in subsequent blogs). There are two types of TMMi assessment;

Informal – typically used to identify improvements or to set a baseline, can be carried out by anybody with an understanding of TMMi but cannot lead to a formal maturity rating.
Formal – typically used to provide a formal maturity rating or certification, must be carried out by two accredited TMMi assessors (one of whom must be an accredited lead assessor). More rigour in the corroboration of verbal evidence is expected.

I’ll discuss how these assessments are carried out in my next blog, and the final blog in this series will cover how you can become a TMMi assessor.

Meanwhile, if you are interested in learning more now, take a look at our TMMi page.

19 May 2022 by Simon Frankish Leave a Comment

Calculating ROI for test process improvement

In this third blog, we look in more detail about how return on investment (ROI) for a test process improvement initiative can be calculated.

There are numerous ways of working out the potential, and actual, financial benefits of implementing test process improvements. Here we show one way, based on the method Experimentus uses and for a TMMi driven initiative.

Below is an example of a spreadsheet used to help calculate and document the costs and benefits for a project improvement plan (PIP) for a real client:

Calculating ROI for test process improvement

For each area of improvement, we estimate how many person-days we believe it will take to make the improvement.
We then agree with the client what the estimated average cost of a person day should be
- This will include both the client’s resource costs and Experimentus’ resource costs – dependent on our level of involvement.
These are then used to calculate the estimated cost of implementation. Here we have estimated £1,000 per day.
We then calculate the anticipated benefit, and, by subtracting the cost, we can estimate the net potential benefit for each of the activities.

In the table above, for PIP ID 5, you can see that we have an improvement with a potential benefit of zero against it. Often we find that there are some improvements for which there are no financial benefit. The benefit they do provide is that they are foundation stones for the other changes. If these are not implemented, the benefits of the other changes would not be delivered.

These items still need to be in the costings, and as long as the overall calculation still demonstrates a sufficient benefit, there is not an issue. The advantage is that by including these zero-benefit items with the positive-amount benefits, the overall cost of the implementation is fully understood.

The above is an example of a potential benefits summary where there is positive value-add to the business.

We have included a confidence factor (in other words, how confident we are that the benefits will be realised) which can be adjusted to each situation. The implementation factor (how much could implementation costs vary from the amount originally estimated) can be positive or negative.

For this example, from experience with this particular client, we have estimated an extra 50% could be needed. The final net potential benefit is calculated by subtracting the adjusted cost from the adjusted benefit.

We hope you enjoyed this series of blogs on the need for and benefits of test process improvement. It can be difficult to justify a test process improvement initiative, but it will provide a lot of benefit, both direct and indirect. For more information on how we can help you, please take a look at our test process improvement page and feel free to contact us.

5 May 2022 by Simon Frankish Leave a Comment

Benefits of test process improvement

Typical benefits of process improvement

In this second blog on test process improvement, we look at the benefits that you can expect to realise as you implement new and improved ways of working.

Test process improvement benefits can be broken down into direct and indirect types. Sometimes the indirect benefits are larger, or more important, than the direct ones, so they should not be ignored. It is well worth taking into account the indirect benefits when the overall value of an improvement program is being calculated.

Because long-term investments can be allocated to improvement programs, it is also critical that the return on investment (ROI) of an improvement program is calculated accurately and consistently.

Direct benefits of process improvement

Direct benefits include:

A rise in productivity
Early defect detection resulting in a reduced need for live fixing

Indirect benefits of process improvement

Indirect benefits include:

Improved staff experiences leading to increased morale and motivation and as a result, better quality work
Increased customer loyalty
Allows greater movement of employees within projects
Improved working environments
Recognition of testing as a profession
Integration of testing into the development process, irrespective of the methodology being followed (for example following a traditional model like v- model or where agile is used).
TMMi will help guide the introduction and use of a structured and controlled set of test processes.

Return on investment

Return on investment (ROI) is calculated by direct costs and benefits, since it is easier to quantify direct costs than indirect ones.

Our next blog will give more detail on how ROI can be calculated for a test process improvement initiative.

General benefits

While there are specific benefits, there are also general ones to consider that can be just as important and valuable, so they should not be ignored. General benefits include:

Organisations are enabled to be more effective and efficient
Product quality can be improved
Increased testing productivity – potentially leading to shorter delivery time frames
TMMi suits any development life cycle model
There is a shift-left paradigm, moving from defect detection to defect prevention

This is not a full list, but just to set a scene for the kinds of improvements that may be expected.

As yet, there is little published information about costs and benefits from assessments. In fact, most organisations will want to keep that information confidential, particularly those that are not performing well.

Take a look at our test process improvement page for more detail on how we can help you realise these benefits.