The ‘Sense-Award’ scoring system: Objective adjudication for a multi category food awards competition

Increasingly, food preference has become based on the mantra ‘we taste therefore we eat’, with consumers consistently seeking quality information across the product spectrum1. Thus, the use of ranking indications from food awards has become important to a product’s impact in the marketplace, particularly artisanal foods2. However, do these awards in sensory terms designate objective assessment? The Blas na hÉireann Irish National Food Awards have been held for the last three years on the first weekend of October in Dingle County Kerry as an integral part of the annual Dingle Food and Drink Festival. Upwards of eight hundred products have been submitted to the awards competition each year and scored from approximately 200 companies across over 27 categories.

All products entered from across all food categories required comprehensive and independent assessment and consequently, a robust scientific scoring system was required. A non-complex categorised scoring scheme which we call the ‘Sense-Award’ scoring system was applied to this large multi-category generic food awards with the aim of determining the validity of judge scoring in terms of description, discrimination and ranking of product quality. This protocol was designed by sensory experts from University College Cork (UCC) and the University of Copenhagen (UCPH)3-5. In this system, judges were ‘untrained’4 but provided with basic instruction and consisted of individuals from different parts of the food sector, including commercial buyers, chefs, retailers and food writers but foremost consumers of the respective products tested. Scoring consisted of objective category scales for Appearance, Aroma-Flavour, Texture plus the subjective terms of Overall Quality and Mass Appeal for products presented blind to the judges to ensure impartiality. This ‘Sense-Award’ method was proven to be a statistically valid system for judge scoring in terms of description, discrimination and ranking of product quality and was easily applied and effective in allowing judges to objectively assess all products using only one scale and without producing numerous tied results. This article describes in detail the ‘Sense-Award’ scoring system which is objective, consumer-relevant, user-friendly, fair, impartial and transparent.

Traditional foods and the artisanal producer

Many food awards acknowledge regional traditional, culturally unique, food production for reasons of quality, nationality and perhaps even nostalgia and the Irish National Food Awards (Blas na hÉireann) is no exception. Additionally, traditional foods are an important element of the European cultural heritage6 and in an age where consumers want to know more about the origins of the food that they consume, traceability is far easier to define for traditional foods than more generic varieties.

In recognising the importance of this area of food science, UCC is a member of the FOODUNIQUE network coordinated by sensory scientists working on ‘food perception and identity’ at the University of Copenhagen (www.foodunique.eu). This is a network with the mandate of promoting, improving and preserving unique regional foods in Europe. FOODUNIQUE is a European Network of key world class interdisciplinary research groups, encompassing Genetics, Location, Identity, Perception, Marketing, Traditional, Technology, Processing and Health researchers, branch organisations, tourist boards and governmental bodies aimed at ‘researching, understanding, defining and promoting the benefits of food uniqueness in the European Union’.

The Irish National Food Awards (Blas na hÉireann) supports the FOODUNIQUE philosophy and has enabled the development of the ‘Sense-Award’ system which has added tremendous value to the quality and standard of these awards and has marked them out as being truly unique in a global context.

Food awards background

There are a plethora of different food awards and competitions that take place every year throughout, the world, and Ireland is no exception. All are relevant in that they promote the food and beverage products of regions and often reward artisanal or organic producers for making traditional, quality products. They are a means of promoting national or regional foods to the consumer as well as rewarding the producer by a commendation or award device which can be used to further sales and ensure the market penetration for products of note. Consequently, the consumer has increased access to these products and the production system is maintained throughout the economy. However, are medallion d’or competitions reflective of objective human sensory perception? In this respect, we have briefly reviewed a selection of current Food/Beverage Award competitions which vary in the scoring or judging methods employed.

In the United Kingdom, The Great Taste Awards, an annual event, is organised by the Guild of Fine Food. In 2010, 6,000 products were judged over a period of two months by over 350 experts, including; key buyers, retailers, chefs and food writers who blind tasted entries to ensure they are fairly and independently assessed. A minimum of eight experts, and more often 16 individuals judge the various products (www.finefoodworld.co.uk). Another UK based awards includes the World Food Awards, which are again an annual, national awards ceremony which recognises operators in the world food sector from the Europe and the UK (http://www.worldfoodawards.com). Judges include food writers and chefs.

At a European level, The Superior Taste Awards have been held annually (since 2005) by the International Taste and Quality Institute (iTQi), based in Brussels. Food and beverage products from producers of all sizes, which are available in stores, are eligible to participate in the competition, with entries from more than 60 countries (www.itqi.com). Products are tested by selected European Chefs and Sommeliers, the iTQi works with the Association de la Sommellerie Internationale (ASI). Additionally, The SIAL D’or Awards are judged by a panel of 30 judges which consist of food writers from 30 international magazines in the food and beverage industry, representing 30 different countries. Nine categories of product are judged including; Savoury grocery, Sweet grocery, Fresh (non-dairy) products, Fresh dairy products, Frozen savoury foods, Frozen sweet foods, Nonalcoholic beverages, Alcoholic beverages (excluding wine), and Wine. The SIAL (Salon International de l’Alimentation) is held every two years and is one of the world’s largest food tradeshows (www.sial.fr).

In the United States, The Good Food Awards, an annual Awards Ceremony held in San Francisco, acknowledges food producers and the ingredient supplying farmers with the presentation of the Good Food Awards seal. Twelve food experts representative of the entire US judge each category. Prizes are presented to winners in seven categories: beer, charcuterie, cheese, chocolate, coffee, pickles and preserves (www.goodfoodawards.org).

From an Irish perspective, The Good Food Ireland Awards highlights places committed to local Irish food to inform consumers where to go to eat out or stay. Members are independently assessed by an expert judging panel, from all sectors of the hospitality and food industry and the public nominate their favourite Good Food Ireland Members throughout the course of the year (www.goodfoodireland.ie). Moreover, The Irish Food Writers Guild Awards recognises Irish artisan farm-based food producers. This competition is not open to any producer who wishes to enter, instead members of the Guild are solely invited to nominate products they believe are worthy of an award. However, ‘worthiness’ can be considered a difficult quality to define. Worthy entrants must be produced in Ireland along with the products main ingredient and trading for at least three years. Finally, every product that is selected is purchased and at a formal tasting meeting members vote for the winners using proportional representation (www.irishfoodwritersguild.ie).

Sense-Award methodology

The ‘Sense-Award’ system was developed in the summer of 2008 from collaboration between UCC and UCPH for The Blas na hÉireann Irish Food Awards held in October of that year and has been subsequently used for the competition over the last three years. It was designed to allow the assessment by judges of a very diverse selection of food and beverage products. Gold, Silver and Bronze awards were presented to winners in each category. All products had to be on sale in Ireland for a period of no shorter than six months and be available to purchase in a minimum of three retail locations. Judges were ‘untrained’ but provided with basic instruction and undertaken in two stages. In the first stage, which took place at UCC, eight to ten judges, consisting of consumers of that product category, judged all products and the top five products in each category went forward to the second stage. The second stage which took place in Dingle during the award competition (Blas na hÉireann) with judges from different parts of the food sector including: commercial buyers, chefs, retailers, food writers and again a consumer of the respective products category was included.

Judges were asked to rate the attributes of each coded, blind, sample product on a devised non-complex category scale by placing a circle in the box for each attribute scale (Figure 1). The scales were designed for ease of use by judges without any cognitive ambiguity, but were complex enough to ensure product discrimination. Scoring consisted of objective category scales for Appearance, Aroma-Flavour, Texture plus the hedonic terms Overall Quality and Mass Appeal (Figure 1). The hedonic attribute, ‘Consumer Relevance’, was included in order to maintain the focus of the awards on consumer products and was effective in the evaluation of the mass consumer appeal of products.

Figure 1 Category scale used for objective and subjective sensory attributes. Appearance, Aroma/Flavour, Texture and the hedonic attributes; Overall Acceptability and Consumer Relevance

The ‘Sense-Award’ approach has proven to be highly discriminatory and effective in terms of categorising and ranking samples, the judges found the scales easy to use and secondary adjudication was limited over the three years of application. Figure 2, as an example, presents an ANOVA-Partial Least Squares Regression7 (APLSR) plot of the products and judging scores for the entire Bread Category in the 2008 competition (Blas na hÉireann). Essentially this is a plot of the statistical correlations of products to scores. Where products are positioned close to a score, they are highly correlated. Samples were systematically grouped over the increasing ranking order of the labelled magnitude scales. The 52 breads, subdivided into six groups, tested by judges showed significant (P<0.05 per cent) description and clear quality discrimination. The location of the ‘Gold Standards’ were significantly (P<0.001) related to the highest scoring. The Silver and Bronze designations were correspondingly significant (P<0.01) and related to the descriptors at the ‘Excellent’ rank. Nonsignificant scoring variation was apparent within ‘Fair’ and ‘Gold Standard’ groups respectively, reflecting the unanimous agreement of judges. Whereas, significantly (P<0.05) greater variation was observed for intermediate ranked products. The method was easily applied and effective in allowing judges to objectively assess all products anonymously, using a single scale type.

Figure 2 APLSR of 52 bread samples as categorised into six sub classes (1. White, 2. Rye, 3. Flavoured, 4. Multigrain, 5. Spelt, 6. Brown) versus ranked scoring of breads Appearance (-A) Flavour/Aroma (-F) Texture (-Tx) Quality (-Q) and Mass Appeal (-MA) from Fair to Gold Standard

Conclusion

The ‘Sense-Award’ non-complex categorised scoring system was proven to be a statistically valid method of judge scoring in terms of description, discrimination and ranking of product quality, as well as being userfriendly, fair, impartial, consumer relevant and transparent. The method was easily applied and effective in allowing judges to objectively assess all products using a single scale type. The application of the ‘Sense-Award’ method at the Irish National Food Awards (Blas na hÉireann) has demonstrated that the appropri – ate application of sensory analysis can be highly successful in the objective discrimination of cross-category food product quality.

Acknowledgements

We would like to thank the committee for the National Irish Food Awards, Blas na hÉireann, Dingle, Co. Kerry, Ireland for their collaboration and assistance. In addition, we would like to thank the European Network www.foodunique.eu and its members for their continued support.

References

1. Lyman, B. (1989). A psychology of food: more than a matter of taste, Van Norstrand Reinhold, New York

2. Shifferstein, H. (2001). Effects of products beliefs on product perception and liking, in: L.J. Frewer, E. Risvik, H. Schifferstein (Eds), Food, People and Society – A European Perspective of Consumers’ Food Choice, Springer, Berlin, pp. 73-96

3. O’Sullivan, M.G., Martens, H., Boberg, S., Kristensen, L. & Martens, M. (2002). The reliability of naïve assessors in sensory evaluation visualised by pragmatical multivariate analysis. Journal of Food Quality, 25, 395-408

4. O’Sullivan, M.G., Byrne D. V. and Martens, M. (2003). Evaluation of pork colour: sensory colour assessment using trained and untrained sensory panellists. Meat Science, 63, 119–129

5. Dijksterhuis, G and Byrne, D.V. (2005). Does the mind reflect the mouth? Sensory profiling and the future. Critical Reviews in Food Science and Nutrition ;45 (7-8) :527-34

6. Guerrero, L., Claret, A., Verbeke, W., Enderli, G., Zakowska- Biemans, S., Vanhonacker, F., Issanchou, S., Sajdakowska, M., Granli, B.S., Scalvedi, L., Contel, M. And Hersleth, M. (2010). Perception of traditional food products in six European regions using free word association. Food Quality and Preference (21) 225–233

7. Martens, H. and Martens, M. (2001). Interpretation of Many types of data X – Y: Exploring relationships in interdisciplinary data sets. In: Multivariate analysis of quality: An introduction, Ch.8, page 139-156, John Wiley & Sons, Ltd. Chichester, England

About the Authors

Maurice G. O’Sullivan has a PhD in Sensory Science (2002) from Faculty of Life Science (LIFE), University of Copenhagen as well as MSc (1995) and BSc (1993) degrees in Food Science, Technology from University College Cork (UCC), Ireland. Subsequent to his PhD Maurice worked as a flavour chemist with Diageo Baileys Global Supply and is currently a Research Fellow at The School of Food and Nutritional Sciences, University College Cork, Ireland. Sensory and consumer science are his principal areas of research.

Derek V. Byrne has a PhD in Sensory Science (2000) from Faculty of Life Science (LIFE), University of Copenhagen as well as MSc (1995) and BSc (1992) degrees in Food Science, Technology and Nutrition from University College Cork (UCC), Ireland. Subsequent to his PhD, Derek became an assistant professor (2001) and associate professor (2003) at the sensory group at LIFE. More recently in 2007, he became deputy head of the Sensory Science Department at LIFE. In terms of research focus, he leads a research group on sensory food perception and identity. He is coordinator of the Danish led Pan-European FOOUNIQUE Network. www.foodunique.eu.

Mary P. O’Sullivan is a social scientist and graduated with a degree in social studies (2006) from University College Cork (UCC). Mary has worked in the area of social science in Ireland and undertakes part time research and contributed background research to the presented article.

Joseph P. Kerry is a college lecturer and Head of the Food Packaging Group in the School of Food and Nutritional Sciences at University College Cork (UCC). He received his Doctorate in Microbiology at University College Galway. Dr Kerry is also a qualified member of the Institute of Packaging. Primary research interests address various aspects of food packaging, shelf-life stability, food composition and numerous aspects of food quality, especially in relation to product sensory attributes. He also has very strong links with industry and his research team assists companies in relation to many aspects of new food product development.