An effort to clarify the role of deliberative evaluation in the planning and policy-making process.
Thorbjoern Mann
EVALUATION: TYPES OF JUDGMENTS
Different kinds of judgments can be distinguished according to the purpose for which they are made and communicated: Some judgments are aimed at supporting or indicating decisions, a c t i o n s: to accept or reject a plan proposal, to initiate processes. Others are are just intending to provide d e s c r i p t i o n s, about what things, environments, buildings, plans are like. Such descriptions can be merely informative, but the label ‘judgment’ is often held to mainly apply to ‘evaluation’ expressions of goodness, value, appropriateness, to the things we evaluate, ‘a p p r e c i a t i v e judgments and that these judgments are ‘subjective’. Is it important to recognize that expressions of description of those objects which we expect to be ‘true’, ‘objective’ matters of fact are often also only e s t i m a t e s, that is, judgments of degrees of certainty that things are really so? (and that these judgments are personal and seem more subjective than objective?)
Classes of judgments usually distinguished in evaluation discussions are
- ‘Descriptive‘ and ‘Evaluative’ judgments;
- ‘Offhand’ (‘spontaneous’) and ‘Deliberated judgments;
- ‘Overall’ (about the ‘whole’) and Partial judgments (about parts, sub-aspects etc.)
This leads to the following sets of judgments:
- ‘Action’ or decision-oriented judgments and arguably evaluative:
- overall offhand and
- overall deliberated
- Descriptive:
- overall offhand,
- overall deliberated
- partial offhand
- partial deliberated
- Evaluative
- overall offhand,
- overall deliberated
- partial offhand
- partial deliberated
If we admit m e a s u r e m e n t s as a kind of description judgment, ‘e s t i m a t e s‘ of results of measurements (that have not yet been done, or predicted for the future) will also have to be added; and though the aim may be to have both measurements and estimates be as close to actual ‘objective’ properties as possible, are these estimates subjective or objective? There are scientific protocols (precise description of where, when, by what means and tools measurements have been taken, so that they can be repeated by anyone, anytime, anywhere and yielding the same results, etc.) that must be met to claim validity of objectivity claims: Do we need to be more exigent about similar protocols for predictions, forecasts?
Additionally, the distinctions between judgments seen as ‘objectively’ true or ‘factual’ , as opposed to ’subjective’ opinions, are the subject of persistent controversies: are evaluative judgments objective or subjective? Has the legitimate and ‘responsible’ human quest to base decisions on facts and truth, that is, objective premises that confer certainty and authority, led to tendencies to not take subjective judgments as seriously as objective judgments or measurements?
The distinctions are not clear-cut. It is tempting, for the sake of simplicity and clarity, to declare ‘objective’ descriptions as related to features of the object evaluated, (which then everybody should accept as such, whether we like it or not, provided those protocols have been met) and ‘subjective’ evaluations as pertaining to the effects generated in the person (subject) doing the evaluating, and as such changeable and personal taste, ‘about which we cannot argue’. But consider a judgment such as ‘affordable’ — eminently personal and thus subjective: I cannot afford the multi million-dollar yacht or mansion a billionaire has no trouble buying. Isn’t that judgment based on the objective fact of the cost of the yacht exceeding the amount of money in my bank account? Or we observe — objectively? — that some people like a certain building and call it ‘beautiful’, while others do not: even if we declare the judgments of the latter group ‘wrong‘ or misguided, (on what ‘objective’ grounds?) — are they not still judgment facts that will influence the decisions made? Some, like C. Alexander, are arguing that aspects like beauty and value are “matters of objective fact”. Is the implication that differing judgments — opinions — should be dismissed? Can these differences be resolved? How should they be handled in public deliberative evaluation efforts?
Which brings us to the distinction between ‘i n d i v i d u a l’ judgments by single participants in the process, from ‘g r o u p’ judgments or indicators based on all those individual scores. Is it appropriate to speak of a ‘group judgment’ — groups are seldom if ever one unified entity that ‘have’ a unified judgment; there usually are significant differences in the judgments of individuals and sub-groups within a collective? Sure a group can agree on a judgment. But even using the term ‘group judgment‘ and demanding its pursuit as a patriotic or solidarity, moral duty, can be seen as an effort to enforce the views of one party in the group over the objections of another, dismissing the latter’s concerns. It may be better to just consider the different kinds of statistical descriptions of the set of individual judgments: different statistical indicators that can guide the group’s decision in different ways.
How all these issues should be dealt with in the design of support systems for the planning discourse is very much a matter needing discussion, since extreme positions on some of the issues would have significant implications for the role of evaluation in the process.
–o–
Abbe Boulah's Weblog 
The Systems Changes research team has recently like Geoffrey Vickers’ approach in appreciative systems.
He describes three judgments: reality judgments, value judgments and instrumental judgments.
We’re tweaking some of the ideas so that they’re more about systems changes (breaking the frame of systems as static with infrequent shifts). I’ve presented a variety of different approaches to the team members. They like Geoffrey Vickers.
David, thanks. In my ‘argumentative planning’ perspective, these distinctions appear as the premises of pro/con arguments, and as the evaluative judgments about those premises:
Argument parts: (claims) Assessments:
“Adopt Plan A (conclusion, proposal)
because
1. PLAN A will produce effect B give conditions C (Factual-instrumental claim)
and
2. Effect (consequence) B ought to be pursued (Deontic claim)
and
3. Conditions C are (or will be ) present” (Factual claim)
I am sure premises 1 and 3 will be called ‘reality’ judgments according to the Vickers view, while premise 2 will be seen by some as a ‘value’ judgment; — but by others as a ‘reality’ claim based on findings (‘factual’ data?) that many people feel that B ought to be?
So I have made the distinction cut in a different place: all the premises are just proposed claims — forget the types — , but to determine the merit of the argument, they will all have to be ‘evaluated’ –‘appreciate judgments’. I suggest — for discussion — to use ‘plausibility’ for all of them, which includes ‘truth’, ‘probability’ or just ‘plausible, makes sense’) and in addition ‘weight of relative importance’ for premise 2, relative to all the deontic concerns in all the other pro or con arguments that will support or question the plan proposal. (the questions about how the overall deliberated judgment for the plan will be ‘aggregated’ form all the partial judgments will be discussed in a following segment.)
These different views are the reason I felt it useful to review the entire issue of evaluation — in a public discourse where the vocabulary to be used cannot be imposed by one consultant and their ‘brand’, however superior, the question arises how these differences should be dealt with. Can we reach a global uniform agreement? We seem to need at least extensive ‘translation’ or dictionaries, and procedures for resolving questions of this kind. As I said: discussion.