☰
Navigation :

Software Engineering in Science

Why is it needed?

The expectation is that your scientific code must be error free, within reason. However, there are many places where errors can creep in, and in general, scientists do not perform basic software engineering techniques which are designed to minimise “stupid” mistakes. Sources of errors can either be conceptual - a mistake in understanding of the theory - or in the implementation of the software used to solve problems. Here, we’re going to focus on the mistakes that come in from implementation issues, which really, are the easier ones to deal with. Let’s try and think about the issues that might happen when when things go wrong with software, both scientific and general.

There are a number of high profile examples which can illustrate for us the issues that might occur from coding errors.

Geoffrey Chang

Geoffrey Chang

Geoffrey Chang and coauthors published a series of papers in high profile journals which attempted to determine the structure of molecules through analysing X-Ray crystallography data. The research groups papers were very highly cited and influential. However, in 2006 Kasper Locher performed a similar technique to Chang to determine the structure for what should have been a similar crystal to one studied by Chang’s group. Instead, he found a 180° flip in the orientation of one helix, which suggested that the results were incorrect. Chang went back and looked at his software, and realised that two columns of data had been accidentally transposed, which resulted in the signs of charges in the molecular structure being flipped. In total, this led to retraction of 6 papers, and around five years of work.

You can read more about the impact of this here.

Reinhart and Rogoff

Two economists, Reinhart and Rogoff published an influential paper \“Growth in a time of Debt\” on the topic of government debt to gross domestic product ratios. The results from their study showed that growth declined once the ratio reached around 60%, the growth in countries historically had begun to decline. The paper was notable for it’s use in political circles; it was cited by government figues in the UK, EU and US as the basis for reducing government spending.

A PhD student in the US, Herndon, struggled to reproduce the results of the study, and eventually contacted the original authors of the paper requesting the non-public data that the study was based on. The authors passed this on, but Herndon and found a number of spreadsheet errors which meant that certain countries were not included in the historical data. On inclusion of these countries, the results from the paper still showed a decline in growth, but one which was substantially lower than as originally presented.

Reinhart and Rogoff BBC Article

Ariane 5 Rocket Launch 1996

The launch of the first Ariane 5 rocket was a 10 year development and a $7 Billion cost for the European Space Agency. On the first test launch in June 1996 the trajectory of the rocket went of course 37 seconds in and the self destruct mechanism kicked in.

Investigations found that the inertial reference system had software bug, which was caused by the conversion of a 64-bit float containing the velocity into a 16-bit integer, and this caused an integer overflow. It turned out that this calculation was not even necessary on the Ariane 5, and had been inherited from software on the Ariane 4, but because this rocket reached much higher speeds, the rocket attempted to make correction for altitude deviation which hadn’t happened.

This was a particularly expensive mistake!

Supercooled Water

Supercooled Water War Very recent case - but went on for years. Researchers had conflicting results in simulations of supercooled water. Eventually, one group published all of their research software, which put pressure on the other group to do so. When they eventually did, it was found that they were using a very unconventional method of initialising their molecular dynamics simulations, which led to the spurious results.

NHS Connecting for Health

NHS Connecting for Health logo

Stepping away from scientific software bugs, let’s look at how poor software design and oversight may hamper a different type of large software project, even when professional software developers are the ones writing it. It may surprise you to find out that in the UK, there is no national database of patient records. If you go to A&E on a Saturday night unconscious, in general, doctors will not be able to see your medical records until the Monday morning when your GP opens again.

In 2002, Tony Blair’s Labour government decided to rectify this, and in 2003 and 2004 awarded contracts to Accenture, Fujitsu, BT Health and Computer Sciences Corporation between 2002 and 2004, which the intention of building a complex set of regional systems which could integrate with one another. The initial projected timescale for this project was three years. The proposed system had an expected cost 2.3 Billion and would have been the biggest government IT project in the world.

The scheme was eventually cancelled in 2011 but 16 years later costs from fallout of this project are still escalating - currently at over £20 billion. Compare this, for example, to the UK science budget which in 2018 is £5 billion. A government report “Government and IT - a recipe for rip offs” in 2011 blamed lack of in-house technical knowledge and a tendency to build large complex projects that cannot adapt to changing requirements.

More examples

What are the implications of badly written software?

What should we draw from this?

Reproducibility and Repeatability

A Changing Landscape?

Think now about the requirements you’re being asked to meet for research in the UK. As EPSRC funded researchers, you are already required to: * Deposit the accepted manuscript in an institutional repository so that they are openly accessible within 3 months of acceptance. * All of your research data must now be made public and be available for up to 10 years after the last 3rd party access.

You need to care about these things if you want to stay in academia, because if you don’t meet the requirements, your publications are not countable towards REF, which directly impacts your career path. The motivations behind these policies is that as publicly funded researchers, the general public should be able to access information you produce (with some exceptions on national security or data privacy grounds).

Moving on from funder requirements, we need to look at what publishers are starting to demand. It is becoming increasingly necessary for you to release the source code that generates your publications. See, for example, Nature Publishing Group’s publication guidelines, which are moving in this direction. Where high quality journals start, others will follow! Looking at one particularly odd example, a 2004 paper was retracted from BMC Evolutionary Biology after the author refused to let scientists from countries admitting refugees use of the software, as this breached the journal’s guidelines on software availability.

The other thing to note is that other scientists - and not just funders and publishers - are becoming more demanding regarding access to software, which the recognition that complex software is often not described adequately in research papers, making the results irreproducible. Government funded organisations such as the Software Sustainability Institute are actively pushing for more research code to be made open and maintainable, and provide training. There is now a whole new career path for ‘Research Software Engineers’ - usually former academics with strong software skills, who can provide training to academics and who are included on grant applications to work on making the research software used maintainable.

What will this course teach you?

Everything in the following courses is designed to make it easier for you to writing good software. In the software industry these just some starting points from which all else follows. If you are not doing any one of these things (with the exception of containerisation), you are almost certainly not writing good software. The main point is that scientific software is not some special case - it is just as complex as software in industry, where all of these are standard practice.