All for a better wine

The food and drink industry has numerous examples where testing laboratories have produced varying results on the same sample. This can be caused by different testing methods, the quality of the equipment, the process or the working practices of an individual. This occurs even between different laboratories that belong to the same parent organisation.

In one instance a sample of wine was being transported and was tested by both the exporting laboratory and import laboratory – both of which produced different results. In disputes of data like this, whose results do you trust to provide valid data? This example also reinforced the need for an internationally recognised standard for tests, which is where proficiency testing can provide a solution, helping to generate confidence in the quality of testing procedures within laboratories. 

The benefits of proficiency testing

Proficiency testing provides laboratories with objective evidence of their ability to produce accurate and precise data by independently testing laboratory procedures. Participating in relevant testing schemes allows a laboratory to demonstrate its competence to its managers and clients, and to gain and maintain ISO 17025 accreditation. Proficiency testing can also mitigate risks for laboratories over issues such as product failure, non-compliance, brand reputation, and legal requirements. The ISO 17025 standard covers testing and calibration performed using national standard, non-standard, and laboratory-developed methods, and it applies to all organisations that perform tests and/or calibrations. The standard also applies to all laboratories, regardless of their size or the scope of their testing or calibration activities.

How the process works

A proficiency test works because many laboratories take part in it. Once laboratories have signed up to participate, Fapas sends each lab an aliquot of the same test sample along with instructions describing what the particular proficiency test is looking for and how to submit results. Generally, participants are given a period of four weeks to carry out their analysis, which could be anything from alcohol determination or heavy metals analysis to nutritional tests.

Once the participating laboratories submit their result, Fapas provides them with assessments based on how close their results were to the assigned value. Each laboratory receives a z-score, which is a common metric that indicates how close or far their result was from the correct one.

The further the result deviates from zero, the worse the analysis was; a z-score of zero means the result was absolutely correct, a positive z-score means the result was an overestimate, and a negative z-score means the result was an underestimate.

Generally, if the result is within plus or minus two z-scores, it is deemed satisfactory. Anything outside of plus or minus three z-scores is unsatisfactory. Between two and three z-scores is a grey area because the score may indicate there was a problem with the analysis, or it could simply be a statistical anomaly. The correct interpretation is something for the laboratory to rationalise.

It is important to point out that one test done at a particular moment in time doesn’t necessarily reflect the ongoing performance of a laboratory. This is why it is recommended laboratories take part in multiple proficiency tests throughout the year so they obtain a long-term assessment of their performance.

Proficiency testing for wine

With the research organisation, Technavio, predicting in its Global Wine Market 2016-2020 report that the global wine market will grow steadily and reach a global consumption of more than 30 billion litres by 2020. it’s important to recognise the part proficiency testing in the wine industry will play in ensuring a consistent end product.

Proficiency testing helps to ensure safety and quality in the food and beverage industry. The high value of wine motivates sellers to undertake testing to help ensure the quality of their product. Although wine tasting is a prevalent means of assessing wine quality for the consumer, it is a subjective, opinion-based, and potentially biased analysis method. Proficiency testing offers a standardised and internationally recognised alternative mode of analysis that is reproducible and capable of uncovering a range of potential issues that wine tasting cannot detect.

A wide range of proficiency tests to support testing laboratories is available

With the wine market scheduled for growth, Fapas has worked in partnership with the Italian Wine Federation and Star Ecotronics to develop a myriad of new wine proficiency tests. A total of 41 laboratories took part in the programme in March 2018, with the next round (which is open to laboratories across the world) scheduled for October 2018. Further rounds are planned for 2019. The programme is relevant to everyone in the wine industry, from independent wineries to large, multi-site laboratories. The tests span a range of wines, both red and white, and covering more than 60 parameters – with the aim of making the Fapas scheme the most comprehensive wine proficiency testing scheme world-wide.

To the existing basic wine proficiency tests for alcohol content, sulphur dioxide and acidity, Fapas has added tests for alcohol-related congeners, inorganic ions, organic acids, quality indexes including colour, sugars, heavy metals, nutritional elements, and oxygen-18 isotope ratio. The new proficiency tests are much wider in scope than anything we’ve done before and aim to provide confidence in the products being produced. The new proficiency tests therefore cover the entire spectrum of analyses for wine quality, authenticity and safety.

Applying the results to make changes to testing practices

The outcome of the first round of PT showed that key quality parameters were well reported and results demonstrating a high rate of proficiency, with laboratories across the board performing well. This is extremely promising for the critical parameters that are most frequently analysed by laboratories in this sector.
Looking at some of the individual tests, however, indicates where the proficiency testing process has identified where laboratories need to make improvements.

What the results show when looking at volumic mass at the defined temperature of 20°C, for example, is that most laboratories were successful in achieving a satisfactory z-score across the three wine samples. However, there was a small population of laboratories that consistently over-estimated the result. This is evident in the z-score histograms
(Figure 1 below: histograms reveal a very tight distribution around the central zero z-score, but a clear grouping to the right of each chart, where there is a very tight distribution around the central zero z-score but a clear grouping to the right of each chart.)

Figure 1: Histograms reveal a very tight distribution around the central zero z‐score, but a clear grouping to the right of each chart

This outcome is consistent to the same group of laboratories over the three different wine samples, therefore the problem is not related to issues with the test samples. Instead, there is a strong indication that the methods in practice at these laboratories are causing a systematic bias. There is insufficient method information in the PT to troubleshoot the root cause of this observation but this is something for the wine industry to be aware of and advise on best practice.

Method dependency is a key parameter that can be investigated and observed in proficiency test data. To do this effectively, large volumes of data are required to ensure a robust analysis, as small amounts of data are limited in their statistical power. As highlighted previously, repeat amounts of data are more beneficial as they allow trend analysis to be performed, rather than a one-off sample taken at a particular time, which could be subject to other outside factors. Also of importance is the method parameter information. Although method submission is not a mandatory part of the PT results submission, it is an extremely useful tool for investigating unusual data distributions. Participants in proficiency tests are encouraged to submit method information for this purpose.

Knowing the methods used in the tests will enable Fapas to make recommendations on improved testing practices. For example, the measurement of alcohol volume can produce significantly different results, depending on whether this is carried out by direct or indirect methods. Knowledge of exactly which method was applied by each laboratory enables more reliable assessments of performance and demonstrate the effect of method dependency.

There is an opportunity here for all stakeholders to learn from the first round of proficiency testing. Following feedback from experts in the industry and from laboratories preparing for the second round in October, Fapas has reviewed the scope of the tests. The proficiency test itself is evolving and improving to enable laboratories to meet the changing and complex demands of their customers – who are looking for a consistent product that meets international standards.

Fapas is inviting laboratories – from those in small, independent wineries to large multi-site operations – to take part in its wine proficiency tests to help create an international standard for wine testing and ensure that high quality, consistent wine products reach consumers.

See https://fapas.com/wine for details