Experts
Reading a recent issue of Critical Review Journal (Vol. 22, #4, 2010) which is devoted to reviews of Philip Tetlock’s book Expert Political Judgment (2005), I was encouraged by what seemed to be a long overdue examination of the role of experts in society in general, and politics in particular. I had not read the book, and was looking at the articles in the journal mainly for material related to my own major interest, that of arguments we use in planning, design, political discourse, and of the evaluation of such arguments.
I was a little disappointed to see that the book appears (judging by the discussion in the six papers reviewing it) to have focused on one specific issue — that of the reliability of experts’ predictions of political and other developments in society. I am not in the least disputing the value of such an investigation, nor am I in the position of judging the validity of the methods used by Tetlock, and thus not of his findings. They seem to be quite critical of expert judgments, perhaps even too critical, in the opinion of some of the reviewers.
The issue forced me to re-examine my own views of the role of experts, specifically in planning, where I had long supported the position of complementing (at least) expert contributions to the planning discourse with participation by the people affected by a plan. This position was not so much based on empirical evidence as on theoretical and logical considerations: the information about consequences of plans — the way a plan affects people — is only partially predictable on the basis of expert (i.e. textbook and previous experience) knowledge, but distributed ‘out there’; and the judgments about the deontic premises involved — whether such consequences are desirable or undesirable, and to what degree, is plainly not a matter for the experts to decide, unless they are also among those affected — but such affectedness is mostly a grounds for rejecting judgments on the basis of not being sufficiently ‘disinterested’ and tainted by ‘conflict of interest’.
So there are good reasons to look at this issue in some more detail. The first question might be: what is the purpose of expert judgment? The answer could range from simply “to provide specialized information not available to the average person and political decision-maker” to “helping the average citizen to take positions, vote, or support political decision-makers, on important issues”: a kind of shortcut of judgment in the spirit of division of labor on which society depends so pervasively. This raises the question of what makes an expert and expert’s judgment reliable, trustworthy. The issue becomes significant not only because of the importance of issues for which we are bombarded with ‘expert’ advice in the media, but also because of the less savory spectacle of selection of experts in courts not on the basis of the trustworthiness of expertise but on their willingness to support the respective prosecution or defense positions.
The list of criteria for expert advice trustworthiness is long and varied:
– evidence of training, and possession of degrees in the subject matter;
– evidence of experience in official government, academic or private enterprise positions related to the subject matter;
– the kind and prestige of the respective institution;
– sometimes the position itself (editor, department director…);
– age;
– gender;
– political, ethnic, ideological group association;
– ‘being on TV’;
– extent of followership, election margin or polling results;
– stated position (and ‘voting record’) on issues.
The last item, ‘voting record’ finally begins to approach a really meaningful basis of judgment trustworthiness, one aspect of which was the target of Tetlock’s investigation: how reliable did an expert’s judgment turn out to be in a variety of significant situations, as supported by evidence? Unfortunately — or merely disappointing for me because his studies are an undeniably important start, — Tetlock chose to investigate this only for one type of judgment involved in policy-making and political decisions: that of predictions, in his case, of the economic development of different states as measured by GDP.
Seen from the perspective of my analysis of planning arguments, this kind of judgment is only one of the various types of judgments involved in those arguments. And it may be useful to examine these types in some detail, because it may help to clarify what one might expect form expert advice.
The ‘standard planning argument’ in its basic form has the following structure: it offers support for a proposed plan X, or for opposition to X, by means of the claims
‘Yes, X should be adopted (or not adopted)
because
(It is true that) X will result in consequence Y
and
Y ought to be pursued (is desirable)
in more formal notation:
D(X) (a ‘deontic’ i.e. ‘ought’ claim)
because
FI( X REL Y) (a factual-instrumental claim)
and
D(Y) a deontic claim.
REL can be one of many different types of relations; causation being the most common, but ‘being instrumental to’, (hence the general label ‘factual-instrumental’ that admittedly does not cover the range of such relationships) association, similarity, analogy, class-inclusion, property-assignment, logical implication and other relationships can occur.
The argument pattern can be expanded by providing information about the conditions C under which the relationship F(X REL Y) is thought to hold:
D(X)
because
FI (X REL Y | C)
and D(Y)
and
F(C);
perhaps even
D(X)
because
FI {(X REL Y) | C1}
and
D{(Y) |C2}
and
F(C1)
and
F(C2)
This structure can now serve to examine the different types of expertise that will support such arguments: an expert may be knowledgeable about ALL of the above claims, or just one, or a selective combination of claims.
For example:
An expert may just know all about the plan X and its details, workings, provisions. More likely, his expertise is expected to pertain to the REL claim between X and Y: based on knowledge about the laws (natural or human) behind the cause-effect link, for example. ‘Complete’ expertise for this claim type would have to cover all possible relationships between X and Y, but also — for the discussion as a whole — all possible relationships between X and all possible consequences Y. Even more exacting: the ‘wrong question’ objection to such an argument may point out that while Y may be a desirable objective, there are other means –other potential X-plans — that might be more effective in achieving Y. This quickly becomes a quite demanding expertise, especially if the effects are described in terms of effects on various affected parties — people, and other creatures or conditions, the set of which is in reality quite difficult to ascertain.
Or the expert may be merely knowledgeable about the conditions C, or C1 and C2, under which the relationship REL is expected to hold: the facts, data about these conditions. Such knowledge is based on empirical evidence — ‘data’ — for current conditions (actually most of those are based on past data); and predictions about such conditions that would have to hold for consequences for the plan in the future will have to rely on other REL- relationships. Interestingly, these are the kinds of judgments examined by Tetlock, and his results seem to indicate (again, according to the reviews) that expert judgments about these are no better than mere statistical extrapolations of past and current data. The role of such conditions should distinguish between C1 and C2 conditions: whether a factual condition for a natural causal relationship exists is a different issue than the question of conditions under which parties affected by a plan will consider its consequences desirable or undesirable.
The common criticism leveled against experts is of course that of making the jump from expertise in any or all of these ‘factual’ questions to the expert’s right to make claims and recommendations pertaining to deontic claims: ought-claims. This includes not only the consequence Y but also the very issue under discussion: Plan X.
Nevertheless, such expertise is claimed. What would it be based on? There are several possibilities.
There is the possibility or tendency of turning this question too into one of fact — by reference to
– pre-existing laws, constitutions; and perhaps logical implications of the claim Y from laws or constitutional principles;
– election or polling results that specifically express general (or majority) acceptance or preference for the goal Y in question; thereby also positing the general acceptance of the majority decision rule, or course;
The legitimacy of treating deontic questions as questions of fact has been the subject of much discussion in the literature. There is no question that some experts’ attempt to posit desirability of plan consequences on behalf of those affected, or ‘in the general interest’ or ‘common good’ runs into considerable difficulties of legitimacy and justification; and it should be obvious that expertise in the various factual issues listed above does not concern expertise in judging deontic claims on behalf of others. (The U.S. Declaration of Independence recognizes this implicitly in the provision that government actions derive their legitimacy (‘just powers’) from the consent of the governed.)
Whenever deontic claims have to be justified by further argument, these arguments will inevitably make reference to further deontic claims — on the assumption that those will be accepted or left unchallenged by questioners. These claims include the validity of previous decisions (precedents); general principles of oral, ethical nature (often backed by religious doctrine) for which experts may claim expertise based on study and familiarity with the respective texts. There is a common tendency to argue for a plan on the grounds that it pursues noble and valid ‘principles’ or ‘ideals’ — even when there are valid questions about whether the proposed plan will actually achieve this (or conform to the principles). Such arguments are being proposed with great conviction by ‘experts’ on the deontics in question, and often carry considerable weight in the discourse even when their factual-instrumental and factual claims stand on wobbly legs.
An interesting kind of claim is that of the ‘vision’ of the future situation (sometimes called ‘scenario’) thought to result from the plan. Appeal is to the desirability, beauty, preferability to that situation, or to the ‘image’ (self-image’) of the people inhabiting that further, and have contributed to it.
It should be obvious that few if any experts can convincingly demonstrate expertise in all these areas. They may be ‘experts’ with respect to one or a combination of question types. Thorough demonstration of expertise would arguably be preferable if based less on the indicators of expertise listed above than on actual arguments about the question at hand, each argument being adequately (that is, to the satisfaction of affected parties) supported by the evidence and further arguments for each premise. This, of course, would not serve the purpose of providing ‘shortcuts’ to judgment very well, in that it would require marshaling all the subject-area knowledge pertinent to each argument, and require the other participants to make judgments about those in turn (which they may not be willing, able / qualified to do.)
Can we draw any conclusions from these considerations? I would consider the following: It would be useful to clearly distinguish if not the experts, then the type of expert judgment they contribute. It would then be possible to devise surveys and experiments measuring the reliability or validity of experts’ judgments for each type. For example, the questions Tetlock used to test experts’ reliability in making predictions should be modified to include specification of the conditions C under which the predicted results can be expected to materialize, and some confidence judgment (probability) about whether they will occur (persist from the present, or emerge). For other types of claims, different kinds of tests would be necessary; and such tests should be developed.
It would be useful to structure public discourse in such a way that the different claims constituting arguments pro and con proposed plans are clearly visible to all, and the expert judgment pertinent to each type of claim be clearly and visibly associated with the respective claims. The discourse should provide opportunity to request further backup of expert judgment upon request. And finally, for decisions of great importance, there should be provisions for a process of evaluating the arguments (pro and con) in detail, considering each of the premise types separately, before taking a decision. (Such provisions were discussed e.g. in my book ‘The Fog Island Argument’, and the Informal Logic article “The Structure and Evaluation of Planning Arguments’).
Abbe Boulah's Weblog 