Posts Tagged 'argument evaluation'

Some consideration on the role of systems modeling in planning discourse


Suggestions made by proponents of ‘systems thinking’ or systems analysis to discussions we might call ‘planning or policy discourse’ often take the form of recommendations to construct models of the ‘whole system’ in question, and to use these to guide policy decisions.

A crude explanation of what such system models are and how they are used might be the following: The ‘model’ is represented as a network of all the parts (variables, components; e.g. ‘stocks’) in the ‘whole’ system. What counts as the whole system is the number of such parts that have some significant relationship (for example, ‘flows’) to one another — such that changes in the state or properties of some part will produce changes in other parts. Of particular interest to system model builders are the ‘loops’ of positive or negative ‘feedback’ in the system — such that changes in part A will produce changes in part B, but those changes will, after a small or large circle of further changes, come back to influence A. Over time, these changes will produce behaviors of the system that would be impossible to track with simple assumptions e.g. about causal relationships between individual pairs of variables such as A and B.

The usefulness of such system models — which simply means the degree of reliability with which simulation runs of those changes over time will produce predictions that would come true if the ‘real system’ that is represented by the model could be made to run according to the same assumptions. The confidence in the trustworthiness of model predictions thus relies on a number of assumptions (equally simplistically described):

– the number of ‘parts’ (variables, components, forces, ‘(stocks’) included;
– the nature and strength of relationships between the system variables;
– the magnitudes (values) of the initial system variables, e.g. stocks.

System models are presented as ‘decision-making tools’ that allow the examination of the effects of various possible interventions in the system (that is, introduction of changes in systems variables that can be influenced by human decision-makers) given various combinations of conditions in variables that cannot be influenced but must be predicted, as well as assumptions about the strength of interactions. All in order to achieve certain desirable states or system behaviors (the ‘goals’ or objectives measures by performance criteria of the system). System modelers usually refrain from positing goals but either assume them as ‘given’ by assumed social consensus or directives by authorities who are funding the study (a habit having come in for considerable criticism) or leaving it up to decision-maker ‘users’ of the system to define the goals, and use the simulations to experiment with different action variables until the desired results are achieved.

Demonstrations of the usefulness or reliability of a model rest on simulation runs for past system states (for which the data about context and past action conditions can be determined): the model is deemed reliable and valid if it can produce results that match observable ‘current’ conditions. If the needed data for this can be produced and the relationships can be adjusted with sufficient accuracy to actually produce matching outcomes, the degree of confidence we are invited to invest in such models can be quite high: very close to 100% (with qualifications such as ‘a few percentage point plus or minus’.

The usual planning discourse — that is, discussion about what actions to take to deal with situations or developments deemed undesirable by some (‘problems’) or desirable improvements of current conditions (‘goals’) — unfortunately uses arguments that are far from acknowledging such ‘whole system’ complexity. Especially in the context of citizen or user participation currently called for, the arguments mostly take a form that can be represented (simplified) by the following pattern, say, about a proposal X put forward for discussion and decision:

(1) “Yes, proposal X ought to be implemented,
implementing X will produce effect (consequence) Y
Y ought to be aimed for.”

(This is of course a ‘pro’ argument; a counterargument might sound like:

(2) ” No, X should NOT be implemented
Implementing X will produce effect Z
Z ought to be avoided.”

Of course, there are other forms of ‘con’ arguments possible, targeting either the claim that X will produce Y granted that Y is desirable; or the claim that Y is desirable, granting that X will indeed produce Y…)

A more ‘sophisticated’ version of this typical (‘standard’) planning argument would perhaps include consideration of some conditions under which the relationship X — Y holds:

(3) “Yes, X ought to be implemented,
Implementing X will produce Y if conditions c are present;
Y ought to be aimed for;
conditions c are present.”

While ‘conditions C’ are mostly thought of as simple, one-variable phenomena, the systems thinker will recognize that ‘conditions C’ should include all the assumptions about the state of the whole system in which action X is one variable that can indeed be manipulated by decision-makers (while many others are context conditions that cannot be influenced). So from this point of view, the argument should be modified to include the entire set of assumptions of the whole system. The question of how a meaningful discourse should be organized to take this expectation into account while still accommodating participation by citizens — non-experts — is a challenge that has yet to be recognized and taken on.

Meanwhile, however, the efforts to improve the planning discourse consisting of the simpler pro and con arguments might shed some interesting lights on the issue of the reliability of system models for predicting outcomes of proposed plans over time.

The improvements of the planning discourse in question have to do with the proposals I have made for a more systematic and transparent assessment of the planning argument — in response to the common claim of having public interest decisions made ‘on the merit of arguments’. The approach I developed implies that the plausibility of a planning argument of the types 1,2,3 above (in the mind of an individual evaluator) will be a function of the plausibility of all the premises. I am using the term ‘plausibility’ to apply both to the ‘factual’ premises claiming the relationship X –>Y and the presence of conditions C (which traditionally are represented as ‘probability’ or degree of confidence) as well as the to the deontic premise ‘Y ought to be aimed for’ that is not adequately characterized by ‘probability’ much less ‘truth’ or ‘falsity’ that is the stuff of traditional argument assessment. The scale on which such plausibility assessment is expressed must be one ranging from an agreed-upon value such as -1 (meaning ‘totally implausible) to +1 (meaning totally plausible, entirely certain) with a midpoint of zero (meaning ‘don’t know’; ‘can’t tell’ or even ‘don’t care’).

The plausibility of such an argument, I suggest, will be some function of the plausibilities assigned to each of the premises, arguably also to the implied claim that the argument pattern itself (the inference rule

FI(X –> Y) | C
F (C )”

applies meaningfully to the situation at hand. (D prefixes denote deontic claims, FI factual-instrumental claims, F factual claims)

(The weight of each argument among the many pro and con arguments is one step later: it will be a function of its plausibility and weight of relative importance of the goals, concerns, objectives referred to in the deontic premise.)

This means that the argument plausibility will decrease quite rapidly as the plausibilities for each of these premises deviate from 100% certainty. Experiments with a plausibility function that consists of the simple product of those plausibilities have shown that the resulting overall argument plausibility often shrinks to a value much closer to zero that to +1; and the overall proposal plausibility (e.g. a sum of all the weighted argument plausibilities) will also be far away from the comfortable certainty (decisively ‘pro’ or decisively ‘con’) hoped for by many decision-makers.

These points will require some further study and discussion in the proposed approach to systematic argument assessment. For the moment, the implication of this effect of argument plausibility tending towards zero on the issue of enhancing arguments with the proper recognition of ‘all’ the system condition assumptions of the ‘whole’ system deserve some comment.

For even when a model can be claimed to represent past system behavior with reasonable degree of certainty plausibility close to 1, the projection of those assumptions into the future must always be done with a prudent dose of qualification: all predictions are only more or less probable (plausible), none are 100% certain. The deontic premises as well are less than totally plausible — indeed usually express legitimate opposing claims by people affected in different ways by a proposed plan, differences we are asked to acknowledge and consider instead of insisting that ‘our’ interests are to be pursued with total certainty. We might even be quite mistaken about what we ask for… So when the argument plausibility function must include the uncertainty-laden plausibility assessments of all the assumptions about relationships and variable values over time in the future, the results (with the functions used thus far, for which there are plausible justification but which are admittedly still up for discussion) must be expected to decline towards zero even faster than for the simple arguments examined in previous studies.

So as the systems views of the problem situation becomes more detailed, holistic, and sophisticated, the degree of confidence in our plan proposals that we can derive from arguments including those whole system insights is likely getting lower, not higher. This nudge towards humility even about the degree of confidence we might derive from honest, careful and systematic argument assessment may be a disappointment to leaders whose success in leading depends to some extent on such degree of confidence. Then again, this may not be a bad thing.

A curious discussion about legalizing illicit drugs

(Overheard in the (mythical) Fog Island Tavern… My apologies for the incomplete account of the statements, which is not doing them justice; I simply try to focus on the aspects that raised the concerns.)

Hey Abbé Boulah, I’m glad to see you here — I have  a question for you.

And a good evening to you too, Bog-Hubert. What’s such an urgent question that won’t even let you order a drink to start with?

Ah, Vodçek knows what I need.  Well, I ran into that candidate for the Senate I told you about,  who is thinking about running on a platform of legalizing illicit drugs, so we had a little exchange about that over at the dock.  And then I remembered you’d talked about this group in Australia that held a big discussion on the internet about that same question —  whether the country should legalize drugs. Asked people on the internet to vote on it, based on the arguments they posted. So what’s you opinion about that issue?

Ahh — which issue — legalizing drugs  or that discussion? 

That sly question itself tells me you don’t want to come clean on the legalizing problem. Is that because you don’t agree with the other policies of this Senate candidate? Okay, So what’s your take on that Aussie discussion?

You know me too well, Bogmeister. But the drug problem is really an issue I haven’t made up my mind about. I haven’t really thought about it that much lately — and that discussion didn’t do much to sway my opinion either way.

Why is that? You seemed quite excited about the piece the other day?

Yes, but that was because of the curious way they stated their arguments, not so much about the content.  Let me explain.  This is a group that  puts up some interesting controversial issues for online discussion, presents  a ‘pro’ and a ‘con’ statement — a video with transcription — invites followers to join the discussion and to vote on it.  So far, I’ve only read the transcript; you know I get impatient with all those videos. And it gave me some interesting insight into how many people argue such cases, especially in view of our old problem of how one should go about carefully and systematically evaluating such arguments before deciding. And mapping the discussion to give people a better overview.

Cripes, now it’s getting theoretical.

Can’t be helped.  Cheers. Well, the first guy, the proponent of legalization begins  by stating that drugs should be legalized because the ‘war on drugs’ has been a failure. And he cites the opinion of community leaders who acknowledge this. 

Okay. Does he mention specifics of that failure?

Oh yes;  he lists a whole lot of them:  Aims of the ‘war on drugs’ that have not been achieved.

And does he provide any good evidence for that?

That’s the interesting part. No, he just states the claims. He even started out by saying those failures are ‘self-evident’ . Which means: no evidence needed?

But  wait:  those aren’t really arguments for legalization, but against — what — the war on drugs,  continued prohibition?

Good point:  you are making progress in this business. They are all arguments against the alternative proposal, the status quo — that hasn’t been stated.  But of course it’s there, implicitly, as the current ‘solution’  that should be replaced. And the interesting part is that  listing those failures of the status quo they are making some implicit claims.  Claims that maybe should have been made explicit. 

What claims? Other than that the war on drugs is a flop?

One implied claim is that there is a drug problem that should be tackled somehow; — that probably isn’t controversial, so it doesn’t have to be mentioned because  it’s assumed everybody agrees on it. But  he characterizes that problem differently:  he says it’s a health and social problem, not a criminal one. Which therefore calls for a different approach to deal with it. And even more interestingly:  doesn’t he make the implied claim that there are only two possible solutions to it —  prohibition and legalization? There’s no mention of any other possibility. And if so, the arguments against the war on drugs do become arguments for legalization (in a listener’s mind) — even though he doesn’t explicitly state those arguments. What he says, without saying it, is that legalization would in fact achieve all those aims of the war on drugs.

So what you are saying is that in order to make a thorough evaluation of all the pros and cons, these arguments should be made explicit, and listed in the evaluation worksheet  for that purpose — where you list all the premises and give them scores for plausibility and weight etc.

Right.  That bothers me. The arguments have been slipped into the audience’s mind. But they haven’t been stated openly.

Why does that bother you? If they aren’t stated, they can’t be evaluated.

True. But they are sticking in the mind. So they will subtly or not so subtly influence any vote, won’t they? They also can’t easily be shown on a map of the discourse, at least if we stick to the rule that the maps shows issues that have been raised. And if they aren’t stated and shown, they can’t be questioned and scrutinized, as some of them should be. 


Well, look at some of the so-called ‘aims’ of the war on drugs:  while some are easy to agree on, others are puzzling: for example, the ‘aim’ that drug quality — purity — should be reduced.  This helps to make drugs less appealing because more dangerous —  but is this also an aim of  a n y  overall effort to deal with the drug problem?  One that legalization would achieve more successfully? I don’t know, it looks like one that should at least be discussed, no?  But since it’s not stated,  it’s just part of a long list of arguments against the alternative of prohibition that now works just because of its length. Interesting, huh? 

Well, I’m sure that doesn’t make too much of a difference among all the other arguments for and against.  What were the other pro-legalization arguments?

There weren’t any in the proponent’s statement. Or only very few, really. 

You’re kidding me. Even I can come up with some good arguments about that.  So what was he talking about, then?

He was providing information about a very plausible question — one that any undecided person would be likely, even prudent to ask: the details of the legalization proposal. He talked about — as I mentioned — treating the problem as a health and social issue calling for health and social intervention measures, about shifting funding from law enforcement to such programs, about moving slowly and incrementally,  about taxing cannabis, about  licenses for production, sale, distribution of drugs, about establishing supervised needle  and injection centers as well as other treatment facilities. 

Sounds reasonable.  People should know enough about the details of such projects before voting on them.

I can’t argue with that. The problem is: how should such descriptions be dealt with in the argument evaluation?

Why would they? They are just descriptions of the proposal — he didn’t make them out to be ‘pro’ arguments, did he?

Not explicitly, no. But devoting a considerable part of the presentation on these details, aren’t they meant to be part of the ‘pro’ case?  So the description of all these niceties stick in the mind as favorable features — but since they are not stated as arguments, they also will not be examined, questioned for their validity, plausibility, importance in view of the  yes/no decision. For the usual practice of asking people to make a decision — vote yes or no — on the basis of such a rhetorical presentation, they will do the intended job, quite nicely. But for a systematic evaluation, prompting questions for supporting evidence or other supporting arguments,  this practice poses some tough questions.  Should such points be included as explicit arguments?  And whose responsibility is it to do that? 

Whose responsibility? You mean the proponent or the opponent?  Or some other person or component of the system?  By the way, talking about the opponent, wouldn’t he bring these issues up if he thought they would make a difference? What did he say?

Glad you remind me. The opponent’s statement was just as interesting in its own way.  He started out by flatly characterizing legalization as ‘not a viable solution’  that would not solve but rather exacerbate the drug problem. Again, no real argument for that claim was offered, but some ‘authority’ support referring to UN Drug Conventions recognizing that “drugs are an enormous social and health problem and that the trade adversely affects the global economy.”

Hey, wasn’t that just what the legalization proponent said?  That it’s a health and social problem, not a criminal one? And he used that as an argument for continued prohibition, that is, criminalization?

Curiously, yes.  But the case is made mainly by straight-out negation of the claims of ‘war on drugs’  failure (but he does not use the term ‘war on drugs’ ). He contradicts the proponent’s claims — the UN controls  a r e  working, the problem would have been much worse without them, and so on. Most of the claims of ‘failed aims’  are denied — but just like those, without offering additional evidence or support.  There are some statistics thrown in, but they are not compared with any corresponding data for the proposed legalization, so they just give more of an impression of evidence than constituting a real substantial data-based argument.  The remainder of the case is based on a number of claims of aims that legalization will not achieve, only some of which match the items in the proponent’s list of aims not achieved by the war on drugs. And most if not all of those statements are aimed at claims  and arguments the proponent did not state:  they are ‘presumed’ claims of the pro-legalization side.

They could have been claims that had been made in other documents and media statements:  things the opponent had heard before, that are common knowledge?

True. But some of them could also fall into the class of ‘straw man’ arguments — strange arguments attributed to the other side that are easy  to demolish, and thereby giving the impression that the entire case of the other side is as shaky.  But there wasn’t even much demolishing — with additional evidence — just denying those presumed claims.

So you are saying that those denied claims should be added to the proponent’s arguments before doing a detailed evaluation, is that it? All that would add some considerable complications to our evaluation process, wouldn’t it?

Very true. And to the mapping task. 

Why the mapping?

Because the picture of each sides’ issues and arguments now looks very different from the proponent’s and the opponent’s side.  Should the map show only the issues and arguments each side has actually stated?  Or the implied ones the other side is attacking? Maybe it will be necessary to make up different maps for each side.

Yeah, that will cause some additional quarreling:  which map is the ‘real’ one? Wasn’t the idea that such maps would show a kind of ‘objective’ picture of what’s being discussed?  And now you are suggesting that the maps too become partisan?  I say, that item did raise some troublesome issues about such discussions and how the framework should support them.

Indeed they did. More than it helped me make up my mind about the drug legalization issue.  Based on the arguments proposed in that experiment, I am not  ready to cast a meaningful vote on whether drugs should be legalized. But wouldn’t you also say that those very problems make more systematic evaluation that much more important? While that was probably not their intention, we should be grateful for such groups to raise the issue of how such decisions should be made and argued.  And the example might help us design a better argument evaluation  approach.

I can’t argue with that. Get to work. Cheers.