It depends. It depends whether the aim is to determine the best option in the current set without reference to any external considerations (‘ideal’  Ratings are needed), to determine the best option in the current set with reference to some external considerations (‘simple’ ratings are appropriate), or to distribute something across the current set of options taking only internal considerations into account (‘distributed’ ratings are needed.) These are very familiar to most of us in educational settings where different amounts of marks are allocated to different parts of an examination or to different qualities displayed in an assignment. For example 60% of marks are awarded for paper A and 40% to paper B; or 20% for essay structuring, 50% for content and 30% for style. These percentages are the equivalent of attribute weights and they are not affected by the performance of the candidates on the different papers (e.g. if candidates do much worse on paper B than paper A). In cases where the aim is to have final scores that are comparable across space (candidate groups in different areas of the country) and through time (candidates in previous years) the final evaluation is appropriately based on ‘simple’ ratings. In many situations, including medicine, sport and other forms of competition, we are concerned only to identify the best among the current set of options – the ‘winner’ - with no ‘external’ considerations entering. For example is the best course of action for this cancer patient surgery, chemotherapy or expectant management? For example who is the best flower arranger in this village fete competition? In this case the final evaluation should be based on ‘ideal’ ratings in order that the relevant attributes are truly given the weight specified. In a set of ideal ratings the best option on each attribute is rated 1, whatever its ‘simple’ (or absolute) rating, and the others options are rated proportionately below it In other situations, most obviously when making resource allocation decisions, we are interested in spreading something across the current set of options, not just determining the best option. For example, deciding how much of the available resource pool should we given to different programmes preventing different sources of mortality and morbidity. In this case the final evaluation should be based on ‘distributed’ ratings, in which the ratings for all options on each attribute have been ‘normalised’ to add to 1 In fact the ‘simple’ and ‘ideal’ modes will very, very often produce the same ‘best’. But this doesn’t mean that there may not be occasions when it matters a lot (e.g. the gold medal winner in a combined event such as the decathlon or triathlon might be different if the ideal rather than simple rating system was used.) The distributive mode will also very often produce the same winner as the other two, but as we have seen it has a different purpose. Take the first attribute. 1 Envision the endpoints of the scale for this attribute as running from the best possible (which will be given the value 1) to worst possible (which will be given the value 0); if you find it helpful assign these endpoints verbal descriptions (e.g. full health-dead) or map them on to a set of data (e.g. €1,000- €10,000. 2 Identify the option among your set that rates best on this attribute and use the sliding cursor to leave the bar length for this option at the appropriate point between these endpoints. N.B. If your preference is ‘nonlinear’ as in the Goldilocks situation where ‘just right’ is between too big and too small, the best possible option may not be the largest or smallest number. 3 Do the same for the remaining options and continue to revise the length of the bars for all options until you are satisfied they are placed correctly in relation to the end points and to each other for this attribute 4 Simple: Do nothing more – you have just entered your simple ratings 4 Ideal: Take the highest rating on this attribute and convert it to 1 by multiplying it by its reciprocal. Then multiply all the other ratings for this attribute by the same number. In other words, if the highest simple rating was h then multiply all ratings by 1/h (e.g. if the highest simple rating was .8 multiply all the ratings by 1.25) 4 Distributive: Sum the ratings for this attribute for all options and then express each as a percentage of the total. (In other words ‘normalise’ them to add to 1) When you carried out the fourth step procedure for all attributes, check that in the idealised mode each attribute has at least one option with the value 1. And check that in the distributive mode all the ratings are lower than they were in their simple form. The gallery below contains the simplest possible illustration of the 3 modes using only 2 options and 2 attributes – and simple ratings selected to produce a different ideal result. (which, as we said above, is relatively rare in practice). In the first we see Bob emerges best when the simple ratings are multiplied by the weights and summed. But this means ‘reliability’ has not actually received as much weight as ‘high quality work’. Jack was three times more reliable than Bob, whereas Bob was only 50% better on high quality work. So genuinely weighting the two attributes equally Jack must win – and he does when the ‘ideal’ mode is used (or the distributive one). This situation typically happens when most options perform absolutely poorly on an attribute, as here A pdf of this tutorial is to be found here
|
THE BASICS 
