Unlikely crimes

This being notice of a report produced by the RSS into some of the statistical issues around unusual clusters of deaths, clusters which might result in criminal charges being made against those involved in care. This report does not mention the worrying Letby case to be found towards the end of reference 1, but the issues it raises are very much the issues raised there. It is to be found at reference 2 and its recommendations are included in the snap above.

I have read the report, but only glanced at the appendices which make up around half of the sixty pages.

The prosecutor’s fallacy

‘… Thus, the p-value of 1 in 1 million might inadvertently be taken as proof that there is only 1 chance in 1 million that the medical professional is innocent of misconduct….’. [page 6]

The report argues that when faced with an unlikely cluster of deaths, to which well-meaning people have assigned very small probabilities, we are all too likely to fall into what lawyers have known for a long time as the prosecutor’s fallacy, amongst other names. In the jargon of statisticians, conflating or confusing the probability that given A we get B with that of given B we get A. 

In the present case, A is the event that the cluster of deaths was down to bad luck and B is the event that the patients were not exposed to any unusual risks. Given event B, otherwise the hypothesis, we can compute the probability of event A, which is very small. But having event A does not tell us as much as we might think about the probability of B. For that we need some more information and the famous rule called Baye’s rule, which you can read about at appendix 4 of the present report – or at reference 5 if you want a longer treatment.

P-values come back again

 P-values are themselves a matter of some controversy. A taster for which is to be found at reference 6 – which also suggests that four years ago I thought I had got a reasonable grip on p-values. But I was prompted by the bit in the third paragraph in the snap above to have another go, turning up another chunk of the vast literature on the matter – and I found that I had lost my grip.

So when I did a bit of statistics, many years ago now, one of the things that I learned early on was that if two events were independent of each other, then the probability of their both happening was the product of the probabilities of their happening individually. In fact, this was the mathematical definition of independence. It now seems that sums of this sort are often done in the context of investigations of this sort, leading to infinitesimally small probabilities of something having happened by chance, translating more or less directly into a very high probability that there had been a crime.

When we say, conversationally, as it were, that two events are not connected in any way, that they are independent, we are not usually thinking about the product of probabilities – but, nevertheless, why should I not combine these independent probabilities by multiplication. Is that not the definition of independence?

I fairly quickly discovered that, starting around a hundred years ago with references 7 and 8, that there were lots of ways of combining p-values. Simple multiplication did not seem to be the way forward, although I was not very clear why not.

With one point of difficulty for me being that a p-value could itself be considered as a random variable which, under the null hypothesis (and a few other assumptions), had the uniform distribution over the range [0, 1]. I had, indeed, lost my grip.

Some other matters arising in no particular order

Unusual nature

The report makes the point that investigations of this nature are unusual in that one does not know at the outset whether or not there has been a crime. On the other hand, one probably does know who the supposed perpetrator is. Perhaps from that point of view, any crime here is more like a financial crime – some sort of obscure but illegal financial trickery – than a burglary, where, most of the time at least, there is little doubt that there has been a crime and the problem is rather to find the perpetrator in the haystack in which he is hiding.

Costs

It might be argued that the recommendations of this report would be expensive and time consuming to implement – as well they might be. On the other hand, crimes of this nature are only going to come to light over time and perpetrators are unlikely to make a run for it. Hurry is not indicated. And suspected crimes of this nature are apt to be very prejudicial to the lives and careers of those involved. And they are apt to result in long running and hugely expensive public inquiries. So much better to take time and trouble over the criminal part of the investigation than to get it wrong.

Scapegoats

It seems to be built into our makeup as humans to want to blame somebody when something goes wrong. Whatever has happened has to be someone’s fault, a someone who very possibly did it on purpose, with malice aforethought, to use a more or less obsolete legal term. 

With the result that all too often we go for a scapegoat. In which we don’t usually go as far as the ancients and load our scapegoat up with all the sins to hand, although there is, or at least was, the widespread practise of persuading someone about to be convicted on one offence to put his hand up for all manner of unsolved crimes sitting on the books and looking bad in statistical reports. Called ‘taking into account’.

We are not very good at accepting that we may never know what happened, why something bad happened. In the olden days, people were more content to say that it was God’s will and to leave it at that. Or, perhaps, to assign guilt to some angry spirit who needed to be appeased in one way or another, possibly by a gift or a sacrifice. While now we are more arrogant and think that we can know it all.

It helps to remember that unlikely things do happen. If, for example, you do a million experiments, it is very likely that some of them are going to give very odd results.

Observer bias

Thoughtful people have known for centuries that we see what we want to see or perhaps what we expect to see. In the present connection, this is often called observer bias. This is not necessarily a reflection on the observer, it is just the way we are made.

This report offers the prescription that, as far as is possible, those conducting forensic tests, should not know the context of those tests. So, if they are testing the blood of a known and dangerous drunk for alcohol, they should not know that. They should test the blood ‘blind’, as it is called in the trade.

This may require the lead investigator to be both tactful and firm: some medical examiners are going to resent being kept in the dark in this way.

A related issue is that of ‘tunnel vision’. An investigator gets his idea about the answer to the puzzle and chases after evidence in favour of that idea to the exclusion of all other. We can get terribly attached to what we take to be our own ideas.

From where I associate to my own behaviour leading reviews of small IT projects. The idea was to provide the owners of a project a view of how it was going from the outside, from people who had no stake in the project – although even this simple idea was corrupted by it being found that it was helpful to have the same review team look at a project through its various stages. Another idea was to use traffic lights: projects were deemed to be green, orange or red, with red being bad – which one might gloss as no action needed, action this week and action this day. At that time, reviews always took three days and one was supposed to walk away from the project with a near complete report. What this seemed to mean in practice was that on the first day one had a more or less open mind. But by the end of the second day one wanted to know where one was headed and the third day was spent confirming that diagnosis. So tunnel vision, but at least it was someone else’s tunnel vision.

Against which it was also the case that a report team wanted to have something to report: a nil report was not very exciting. I associate to the rather different world of government finance, where the people you were investigating, if they were shrewd, threw you a few bones to keep you busy.

I also associate to advice given to me a very long time ago: decide on your conclusions and then write your report. Perhaps I should give my mind to the tricky interaction between the search for the truth and the construction of an attractive report – two activities which compete for the investigator’s time.

Red teams

Getting another view is indeed a good way of mitigating the effects of tunnel vision. To which end, in the Catholic Church, when thinking about making someone a saint, they at least used to appoint a devil’s advocate whose job it was to drag out all the reasons why someone should not be made a saint. And in UK law, a lot of business is conducted on adversarial lines – which can seem a bit crude and theatrical at times, but which does provide a second view.

I associate to a well-known (and well-paid) speaker on the management consultancy circuit years ago, who made much of the idea that you should never do something important without having thought long and hard about why you should not do it.

It seems that our own Ministry of Defence is well up to the mark on all this, with their guide on what they call red teaming now running to its third edition at reference 4. And I imagine that well-run war games address a similar issue by having a team of people trying to put themselves in the shoes of the opposition, trying to think as they would.

Perhaps all rather easier when one is not working under the harsh lights of a criminal investigation. Where, although I have not come across one, I dare say that the police have training manuals which cover the ground.

References

There are more than four pages of references at the end of the report, suggestive of a great deal of prior work on the issues. One William C. Thompson scored the most entries and I picked one of his for inspection. It turns out that he is both an expert in the field and one of the authors of the report. 

The entry I chose, reference 9, reports on a substantial experiment to test the way that lay people react to DNA and footprint evidence. An important concept here is the random match probability (RMP). 

‘… DNA analysts typically use population data to estimate the probability of finding a “matching” or “consistent” profile in someone sampled randomly from various reference groups (e.g., U.S. Caucasians; African Americans; U.S. Hispanics). Although the numbers are often framed as probabilities, they can also be presented as natural frequencies—for example, the analyst might say that among U.S. Caucasians only one person in 10 million would “match” or would be “included as a possible contributor” to a DNA sample…’.

Presentation in terms of RMPs is contrasted with presentation in terms of likelihood ratios (LRs) or verbal equivalents of likelihood ratios (VEs). In this context, some statisticians prefer the use of likelihood ratios to probabilities, arguing that they avoid some of the pitfalls associated with these last.

Conclusions

The recommendations of the report have been snipped out into the snap at the head of this post. The bias of the authors is evident in the prominence of those about the need to consult statisticians!

That apart, one hopes that those involved in leading investigations of this sort will take the time to read this report – or some equivalent guidance – before getting properly stuck into the investigation. 

While for myself, maybe it is time to stop trying to get a grip on the statistical technicalities – and rest content with delegation.

PS: a digression from the present post is to be found at reference 10.

References

Reference 1: https://psmv5.blogspot.com/2025/05/trolleys-853-and-854.html.

Reference 2: Healthcare serial killer or coincidence: Statistical issues in investigation of suspected medical misconduct – RSS Statistics and the Law Section – 2022. 

Reference 3: Combining independent tests of significance – Birnbaum, A – 1954. Not freely available, but widely referenced.

Reference 4: Red Teaming Handbook (third edition) – Ministry of Defence – 2021.

Reference 5: https://en.wikipedia.org/wiki/Bayes%27_theorem. 

Reference 6: https://psmv5.blogspot.com/2021/11/the-p-value-battle-continues.html. 

Reference 7: Statistical Methods for Research Workers – R A Fisher – 1934.

Reference 8: https://en.wikipedia.org/wiki/Fisher%27s_method. 

Reference 9: Lay understanding of forensic statistics: Evaluation of random match probabilities, likelihood ratios, and verbal equivalents - Thompson WC & Newman EJ – 2015.

Reference 10: https://psmv5.blogspot.com/2025/06/on-data.html.