Thoughts on Analysis: Expression of Probability in Intelligence Work

Almost every intelligence organisation seems to employ a different method to express probability as part of their intelligence processes. Probability being the level of confidence the analyst has in relation to each piece of intelligence, resultant hypotheses or the overall assessment. Not surprisingly, any ambiguity in the expression of probability has an impact on the working efficiency of an intelligence organisation and has the potential to create a barrier between the intelligence analyst and their customers. Clearly, the ability to convey accurately to the customer the confidence level of the analyst is central to the success of intelligence work.

In a relatively straightforward situation, it is tempting, and probably appropriate, to apply a simple standard system as depicted below:

There are advantages to using this system; it is easily understood and remembered by both the analyst and the customer, and assuming a level of liaison occurs between these entities there is likely to result an acceptable understanding of assessment accuracy.

Although there is an argument that information or intelligence graded at less than ‘POSSIBLE’ should not routinely be included in an assessment, this does not hold up on all occasions as some outcomes, although deemed to be ‘UNLIKELY’, are of such significance that they need inclusion.  As a consequence, some prefer a broader version of the above, quite often expressed in terms of VERY LIKELY, LIKELY, POSSIBLE, UNLIKELY and VERY UNLIKELY.  These can be particularly useful when trying to convey to the customer the likelihood of the worst case scenario happening; for example, ‘nuclear war between Pakistan and India remains very unlikely’.

With both of these systems, the percentage levels of confidence will be much debated points and often consensus will be difficult to reach.  The upside to any such discussions though is that ‘informed debate’ is an essential part of all intelligence work and should be encouraged.

Intelligence predictions should always try to include an expression of time and scale in order that the prediction is useful to the customer.  However, this creates a problem when expressing confidence levels.  If, as an example, an assessment states that the North Korean regime would probably collapse in the next 12 months; this would imply a 75% or greater chance of that occurring.  In the event of the North Korean regime not falling within a year would lead to a questioning of the basic analysis behind the assessment or the timescale, or both.  Are we 75% certain that the regime will fall or 75% certain it will happen within a year?  Will the certainty of the regime falling increase or decrease after the expiry of the 12 month period?  If Kim Jong-un is replaced by another member of the regime’s hierarchy, does that count as a regime collapse?  These are questions that our simple assessment example is likely to generate and highlight the need for thorough analysis of the available information and a carefully crafted assessment to convey the correct meaning to the customer.  By necessity, precision concerning confidence levels may impact on the flow and conciseness of the report.

When there is an aggregation of information, the analyst must ensure that uncertainties are carried forward to the new overall assessment.  If this is not done there is a danger that assessments are formed using an assumption that earlier work had a 100% probability of being true.  A major question for the analyst, for example, could be – three ‘PROBABLE’ pieces of information equate to what grade when aggregated together?  …One for discussion over a beer, although a dilemma normally resolved by an analyst’s experience and common sense.

There is a case in some circumstances to apply a numerical scale to probability.  This I have found to be particularly useful when there are many variables, such as when estimating disincentives (i.e. threats) and incentives as part of an operating environment analysis.  An overall disincentive and incentive score can be used as the focus for an audience to draw their attention to fluctuations in the norm, or to indicate a trend over time.  This is particularly helpful when there is a danger of the detail hiding the overall picture.  For example, stability in a particular region may be steadily worsening due to minor changes to many factors, which would be difficult to emphasise individually, but collectively show a definite pattern.

As intimated earlier, much depends on the relationship between the analyst and the customer, and the need for an element of trust and appreciation for any system to work well.  Part of this relationship must include an understanding that the analyst will provide the best possible product in the timescale available from the sources available, whilst the customer must accept that these limitations may well impact on the completeness of the intelligence and the subsequent confidence level in it.  It must be stated that external pressure to change an assessment or the level of confidence should never happen; the fallout to both the customer and the analyst is likely to be equally severe.  The case for war in Iraq, on both sides of the Atlantic, is widely felt to have resulted from political pressure on the intelligence system to deliver results that met the political imperative at the time.  But, equally, similar pressures exist in the business environment.

Part of the analyst’s lot is to work with an uncertain and incomplete intelligence picture.  It is their job to produce as complete an assessment as possible using their skills and knowledge to fill gaps and uncertainties.  Inevitably, in some cases a single overriding assessment may not be possible to ascertain, and it is then that the best hypotheses are presented to the customer as potential courses of action by the adversary.  The probability of these events needs to be graded and prioritised by the analyst, whilst identifying the ‘most likely course of action’ and the ‘most dangerous course of action’.  This will allow the customer to make planning assumptions and will also permit the analyst to refine the intelligence collection plan.

Finally, assessment by committee is a particularly thorny issue for the intelligence world.  Inevitably there will be vested interests by some of the participants towards certain items or sources of intelligence, which could skew the overall assessment.  There may even be a lack of consensus over a critical judgement.  The customer will normally be aware of the procedure used to arrive at an assessment and the confidence level expressed within it, and will need to be cautious when dealing with an assessment made by committee.

Is your intelligence team working efficiently?  Could they be more productive and enhance your business model?  To find out more about what QRO Global could do to assist your business or to learn more about the types of projects that we have worked on in the past, see our Services Page or email us today – info@qro-global.co.uk