Simplification in science

I am interested in the information that is lost as scientists proceed from experiment to publication. The real factors that slip through the cracks of expediency. What is more important in the communication of research, the method, or the factoid results that come from it? Are we too trusting in the scientific method? Has peer-review become a substitute for a wider interrogation of method?

These are just a few questions going through my mind as I read "Simplification in Scientific Work: An Example from Neuroscience Research" - a 1983 article by the late Susan Leigh Star. I was particularly struck by an early observation in the article that "published scientific conclusions tend to present results as faits acomplis, without mention of production of decision-making processes." I am not sure that this is so true today, but I am intrigued by the possibility that it is exactly that loss of information (as research is presented with a higher degree of 'granualrity') that opens a door for skepticism in the wider community. When a large body of research by multiple scientific schools tends to agree on a matter, there is sometimes an impression given that they are all doing exactly the same experiments. Whilst the broad methods are the same, of course expedient decisions are made and this causes subtle differences. These are not always thoroughly explained, even if they are justified. I can't help but think that something in this is relevant to the skeptical program in climate change. Is this what lets in the calls of "conspiracy"?

More to read, more to do. I have a few other things on my plate, but this is an intriguing line of research.

Scientist at heart

When you think of a scientist, you probably imagine a person dressed in a white lab coat, wearing thick glasses and adorned with white, straggly hair. Perhaps you think of Albert Einstein. The reality is quite different; scientists, overall, do not conform to this stereotype. They do not have a particular uniform, and they do not wear a badge. What makes someone a scientist?

The answer is in the way they think. They employ “the scientific method”. But what is the scientific method? The concept really boils down to a way of formulating ideas and testing them; a way of explaining the world through systematic observation and hypothesis testing. In short, a way of telling reality from fantasy.

You start with a question about the world around you. This may have come from your observations. You come up with a statement about this question, and this is known as a hypothesis. You predict what you would see if your hypothesis were correct. From there, you design and conduct an experiment to see if these predictions hold true. You then interpret the results to see if they support or disagree with your hypothesis. Finally, you report your experiment in full and submit it to review by other scientists.

The thing is, you can only support your hypothesis, not prove it. There may have been errors in your method. You may have got lucky even. However, because people can read your methods, they can repeat the experiment. Repeated success means the results are reliable.

Over time, more support may be collected, and your hypothesis might be accepted as a correct theory. That gravity makes objects of mass attracted to each other (like Newton’s proverbial apple falling to the earth) is one such theory, supported by mountains of scientific evidence.

Evidence is what it is about. That is at the core of the thinking. Theories need to be testable. Interpretation alone is not enough. Ideas and theories are not enough. Skeptical thinking doesn’t cover it. To be truly scientific you must imagine what a theory predicts, ask what evidence for that might look like, and then demand evidence that experiments actually do show it.

At the core of the scientific thinker is a rational process to discover the likely truth of the matter, supported by evidence. That is what makes a scientist, a scientist.

We can all do this, we just need to ask questions, and to think about how we might answer them in a reliable way. Do this, and the climate of uncertainty can be lifted on a great many issues. We can all be scientists at heart.

Data, on the other hand, IS an issue

One of the great frustrations in science is getting good data. Collecting it yourself can be a boring, longwinded and seemingly pointless exercise, especially if you are collecting data on multiple variables when you know you'll only use a few (a common thing in geology). Getting legacy data from others can be even harder. Incomplete data sets, different files, wrong formats, wrong headings etc etc... These are all issues we face. However, having complete data is golden - you don't want your work usurped by another on the basis that they had more complete data and so could see the real picture.

So it seems in the climate change debate, we now see a real problem emerge. One that actually does cause a few problems for the climatologists who have provided the evidence for "AGW". New Scientist recently published a piece correctly (in my opinion) highlighting this as a significant concern, but one with some seemingly intractable barriers to resolution. Large chunks of important data are sitting with commercial rights within the vaults of institutes around the world. Governments would pay penalties for their general release. This is not good for the science and only fuels speculations from the deniers. It is indeed a pity that the deniers can't get their hands on it because then they could do the same tests and come to conclusions that add to the debate. However, all this should not be mistaken as a conspiracy - it is normal in many scientific fields to have data sets locked up under commercial arrangements (or government legislation). Science has worked around this for years and continues to do so. Climate science itself has worked successfully under this regime too. Perhaps this is just another storm in a teacup.

We've had government bailouts for banks, perhaps its time for governments to put some money and legislation behind freeing up these data sets completely. Pay-off the commercial interests, legislate for data freedom. It would be a nice shot in the arm for a needlessly troubled science. I suspect only the deniers have anything to fear.

Scientific method and "controversy"

There seems to be a common thread amongst sceptics out there that science is done via something that looks a little like the Council of Nicaea. That is to say, that a committee of scientists decides what is "doctrine" before instructing publishers what to print. There is confusion between the ways the legal system (or political system) works and how science works.

Lets have a look at some typical tasks in a scientist’s professional life:

1. Data collection. This can be the longest and hardest (and most boring) phase. This is where hours are spent over test-tubes, or, in my case, hours in the hot sun staring closely at rocks. Whilst you may be thinking about the end-game in this phase, the task is usually so routine that bias hardly exists (if it does, it is because the method itself is biased, or you're just sloppy). Actually, there will be mistakes, but these tend to revert to the mean, so will be cancelled out in the final analysis.

2. Hypothesis generation. I put this after data collection to bait some people, but actually, it has to be said that hypotheses are generated throughout the scientific process. The important thing is that you are only testing the original hypothesis whilst conducting an experiment designed to test that hypothesis. Other ones must wait for other experiments. There is no harm in "hypothesis-driven research" - this is what science is. However, this is different to biased research driven towards a pre-determined conclusion. Note the difference - a hypothesis is actually tested, a pre-determined conclusion is circular.

3. Data analysis. Here comes the statistics. So you have the data, and you see patterns. Are they significant? This is a technical, statistical question that determines whether you can use your data (gathered in 1. above) to test the hypothesis (2. above). If there is no significant result, then there is no support for the hypothesis from your results. THIS DOES NOT MEAN IT IS DISPROVEN. It is more like an absence of evidence, which, as the saying goes, is not evidence of absence. If the results show a statistically significant result, then you can compare it with your hypothesis. Now a hypothesis can be disproved - proposing that the sky looks blue and finding it to look green would be an example. Unfortunately the opposite does not apply. If your result concurs with your hypothesis, it lends support to it, but does not prove it. It can never prove it due to a quirk of inductive logic that demonstrates that no matter how many positive examples you show in support of a proposition, since the set of possible examples is infinite, you cannot rule out a counter-example emerging next. Which is quite different from the deductive logic of mathematics, where 2 + 2 = 4 as a result of the system itself.

To make a long story short, the last juicy step is publication.

Now you run into trouble. You've done your experiment, and supported your brilliant earth-shattering hypothesis. Will anyone believe you?

To find out, you detail your method (and the back story - why you felt it worthy of research) and your results and a bit of discussion on what it all means. Then you send it for peer review. This is a blind (well semi-blind - sometimes people work out who the reviewers are) process where your reviewer doesn't know who wrote the paper and is asked to appraise the science, comment, and put their opinion on whether it is fit to publish. Most papers fail this test on the first pass, and the majority never make it to publication. What tends to define success is that the paper details a properly conducted line of research taking into account previous work in a similar field. Failure in peer-review doesn’t mean that there is a conspiracy against you – it usually means your paper is either not relevant to the journal in question, or that you need to write up your science better. Without peer review, this statement cannot be made with any certainty about a paper.

Also, consider that how the media treats science is not the same as the science itself. Science is only balanced in its reporting in so far as it “objectively” reports the outcomes of research and the opinions of researchers. So 90% of scientists might agree with a broad-based position, but it only takes one from the 10% to balance a journalists report – giving a 50/50 impression. Note also the diversity of opinion that will lie within the 90% who agree – these people do not speak to a common mantra, they merely assent to certain generalisations.

So next time you see controversy about methods and "conspiracies" to promote one "side" of an argument over another, consider the above and consider that most scientists are too busy with the steps involved to also hold some sort of cabinet meeting on how to bend the entire scientific community. After all, that would be like herding cats.