Project:Sandbox

You have many pieces of evidence and you have various hypotheses. How do you tell which hypothesis is most plausible?

Name of Lens
Analysis of Competing Hypotheses (ACH)

Used for

 * identifying a complete set of alternative explanations or hypotheses;
 * systematically evaluating data consistent (or inconsistent) with each hypothesis;
 * rejecting incorrect hypotheses (rather than trying to confirm them).

Description
Analysis of Competing Hypotheses (ACH) is a structured analytical technique developed originally in the CIA and now widely taught to intelligence analysts. It is intended to help analysts systematically compare multiple hypotheses simultaneously against many items of evidence. The most likely hypothesis is the one with the least relevant information against it.

ACH is particularly effective when there is a large amount of technical data to absorb and evaluate, especially when working with incomplete and ambiguous data.

ACH helps to identify and question fundamental assumptions. It also helps in avoiding the danger of developing a favoured hypothesis and searching for relevant information to confirm it, which can lead to inaccurate and incomplete judgments.

Example
(from the US Government's publication A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis, p. 15)

How to apply it

 * Canvass a wide range of possible hypotheses (using different perspectives);
 * Make a list of significant evidence and arguments for and against each hypothesis;
 * Prepare a matrix with hypotheses across the top and evidence down the side (see Example);
 * Working across rows, estimate the consistency of each item of evidence with each hypothesis;
 * Identify which items are most helpful in judging the relative likelihood of the hypotheses;
 * Delete items of evidence that have no diagnostic value - that is, those that do not point to one or more hypotheses over others;
 * Draw tentative conclusions about the relative likelihood of each hypothesis;
 * Proceed by trying to disprove the hypotheses rather than by trying to prove them;
 * Analyse how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation;
 * Report conclusions - discuss the relative likelihood of all hypotheses, not just the most likely one.

Links

 * You can find a ready-to-use Google Sheets template for ACH matrices here which was developed in intelligence studies at Mercyhurst College; but be sure to read the blog post first.
 * US Government publication on ACH, A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis.
 * Pherson, Randolph H., Weeding out the Competition - an overview of ACH.
 * Heuer, Richards J. and Pherson, Randolph H., Structured Analytic Techniques for Intelligence Analysis, pp. 181-192.
 * Center for the Study of Intelligence overview, The classic version: CIA, Psychology of Intelligence Analysis.
 * Some accounts of possible issues with the use of ACH.