Manual on Scientific Evidence

My email had a notice for the US National Academies Press and an interesting link in the just-published section. Their blurb:

The Reference Manual on Scientific Evidence, Third Edition, assists judges in managing cases involving complex scientific and technical evidence by describing the basic tenets of key scientific fields from which legal evidence is typically derived and by providing examples of cases in which that evidence has been used.

The chapter on How Science Works is really useful. It’s by David Goodstein, a physicist, and gives a quick tour through the development of the scientific method. It is an entertaining read, perhaps surprising for a manual. It starts with Francis Bacon who said the scientist should be an impartial observer of nature. Goodstein quickly skewers this – no serious thinker accepts that observation is done without prejudice – she must have assumptions about how the world works. Yet look around today, and you’ll find the Baconian ideal still alive and kicking (Plimer has it, for example. Ok, go ahead and laugh).

Popper is next. The idea of falsification and hypothesis testing is really useful for weaning us off Bacon. But even though science is asymmetric, making it easier to falsify than validate a theory, science does not spend all its time on falsification, otherwise new theories would never be developed. And later on, he also says that given the demarcation problem between science and pseudo-science, because anyone can make a falsifiable proposition, falsifiability in itself is an insufficient condition.

Enter Kuhn with paradigms. Science proceeds as a social process, where paradigms are developed by consensus and overturned in revolutions.  Except that they are not always replaced by new incommensurate paradigms as Kuhn thought. Sometimes the old is subsumed by the new, which explains more but does not negate the old theories entirely. The replacement of Newtonian mechanics by quantum mechanics was given as an example. Newton’s mechanics were not wrong, just part of a bigger picture. In this way, science actually is progressing in its understanding of the natural world.

Scientists are not Baconian but become so when describing their findings, being scrupulous about data. They should also however, reference underlying values and assumptions just as scrupulously. Scientists are not Popperian of their own theories, but if a scientist is describing something novel, then they’d better make sure the weak spots are well covered, because other scientists will happily do it for them. Scientific knowledge is not totally revolutionary but is progressive, punctuated by moments of significant advance where some things are being better understood over time.

Science is the scientific method. However, it’s not known exactly what that method is because it differs between disciplines and changes over time. Goodstein describes adversarial competition between scientists for ideas, and reputation, as the two most significant motivators for scientists. The fierce competition for tenure drives diversity in ideas because that’s how a scientist distinguishes themselves from their peers. He over-emphasises the adversarial nature in my view but is making it clear that monetary reward and power are not as important motivators for scientist as they may be for others.

One key motivator he does not mention is the sheer pleasure scientists get from solving puzzles. The creative tension from working on a set of ideas surrounded by uncertainty and wrapped in mystery is released, as the key to the puzzle clicks into place. This tension may build over days to years, involving many mistakes on the way. The pleasure lasts until you realise that the solution that looked soo good unlocks the door to another mystery and the whole process begins again.

This is why uncertainty is a job opportunity for scientists but something to be avoided for most everyone else (though my discussions with artists and writers suggest there’s much the same thing going on in their creative processes).  Money and tenure often become the means to solve puzzles but not an end in themselves. Solving such problems on the behalf of society is also a big motivator (no, really), because it combines personal gain and the greater good.

The chapter rounds out with a series of myths and refutations that are well worth reading, and a description of the Daubert standard where the US Federal Court sought to separate science from pseudo-science in 1993. The four criteria are:

1. The theoretical underpinnings of the methods must yield testable predictions by means of which the theory could be falsified.
2. The methods should preferably be published in a peer-reviewed journal.
3. There should be a known rate of error that can be used in evaluating the results.
4. The methods should be generally accepted within the relevant scientific community.

There is a lot of Popper in this and the criteria are still very limited when attending to really difficult issues but Goodstein concludes that the Daubert criteria have been useful in law.

I’m promoting this chapter because when trying to sort the scientific wheat from the chaff, it is a very useful resource for scientists and anyone else who is interested in what science is trying to say.

The report page is here – individual chapters and the whole report (13 MB pdf) are linked to that page.