In science, the raw data is sacrosanct, but a new type of analysis shows hundreds of anomalies and oddities in BOM raw data, and whole months of supposedly raw records have been cut and pasted, in one instance as far as 40 years earlier.
After all the high-minded purity of BOM claims that they provide all their raw unadjusted data to critics, an audit by Geoff Sherrington and Tom Berger has found that “almost all CDO* data sets that we have studied, about 150 stations, show some past adjustments.” In one case duplicated patterns were found separated by 82 years. ^ And then there are hundreds of repeated structured patterns called “the broken seam” where there is a pattern of repeats, with one or two different values in the middle of the sequence. In the longer stretches, two weeks are copied, then two days are different, and then another two weeks is copied.
Tom Berger is a specialist in data-based deception analysis who has assisted the FBI in the past. So, he has analyzed the BOM raw data in new very detailed ways. He found adjustments in one site were predictable in a machine-learning way from another site 800km away, suggesting that “the adjustments were following a systematic, network-wide pattern,” not just with an issue at a single station.
In one case of a duplicated month (to a tenth of a degree), Graham Lloyd at The Australian tracked down the original archived paper records and confirmed that the 1907 online data was correct but that the original values in the month of August 1867 had been replaced with a copy of the 1907 data.
Among other anomalies Berger found the maximum temperature records in Sydney show 110 years of non-random adjustments in a warmer direction. Always positive, never negative? He also found data gaps — in one case where 647 consecutive days were missing. On top of this there are 879 instances of something he calls “value traps” and over 500 “structured repeats”. One of the structural repeat patterns is known as the 5+1+5. Where there are five identical values, then one different one, and another five identical values. There are also 4+1+4 and 3+1+3 repeats as well. These were the fingerprints, he said, of a human hand on the data.
Imagine if the company accounts had duplicated receipts?
Everyone knows the BOM publishes adjusted formally “corrected” datasets, and we have debated the many versions of ACORN adjustments and homogenization many times, but if the guardians of the raw data are presenting mistaken or altered data as though it was the original recordings, that crosses a new line. Significantly, these strange duplications were not corrected in any version of the expert corrections. They were spread across the whole country, and many were even in the modern era, not just among the old, digitized records copied from handwritten notes.
Temperatures were being recorded with modern Stevenson screens as early as 1890 in Brisbane, but the BOM often ignores these long records.
As the two auditors say: “None of these findings require any knowledge of climate science – they are purely data integrity failures that would disqualify any dataset in any other field of science.”
This error prone data is being fed into climate models that farmers (used to) rely on, and town planners, health officials and insurers are trying to use. We’re betting the economy and transforming our energy grid on this data. We’ll spend a trillion dollars killing off coal but we won’t audit the original data?
The BOM Budget is 614 million dollars a year, and it has 1,803 staff, but it took an analysis by two volunteers to find a litany of irregularities are impossible by chance. The BOM just spent $96 million redesigning their website which was universally panned. But they haven’t checked their own data. It doesn’t add up.
(Don’t forgot the old BOM site still exists so bookmark https://reg.bom.gov.au/. That’s what I use!)
The BOM admits there are some (a few) issues and says it will fix them
The experts tell us life on Earth depends on understanding climate change, but they don’t seem to care what the longest and best records in the Southern Hemisphere say. Perhaps these duplications are accidental. Perhaps the Russians have hacked the database?
When asked by Andrew Wilkie MP, Stuart Minchin, CEO and director of the BOM, said contributions from citizen scientists “play an important role” and that “The BOM rejects allegations of fraudulent or deliberate manipulation of Australia’s temperature record – there is no evidence to support this assertion, and it has no basis in fact.”
The BOM acknowledges duplicated data affects fewer than 200 daily values form a dataset with more than seven million values. They claim this has no effect on the long-term temperature record.
Forensic audit exposes widespread copy-paste errors across Bureau of Meteorology raw climate data
By Graham Lloyd, The Australian
They include the original books from Sydney Observatory in 1867, a year when the city was transitioning from its penal origins into a bustling commercial hub.
The temperatures are recorded in Fahrenheit and in meticulous detail: dry bulb, wet bulb, maximum and minimum in the shade, mean, max in the sun, minimum on the grass, dew point, and on it goes.
But the minimum temperatures recorded in the Sydney Observatory books for August 1867 are not the same as Bureau of Meteorology’s official digital raw data for the same period. Far from it.
In fact, the minimum temperatures for that month in BOM’s raw digital data are exactly the same as the daily temperatures for the Sydney Observatory logbooks for 1907 – a gap of 40 years.
The raw data duplications are still visible on the BOM website. See the minimum data for 1867 at Sydney Observatory and compare it to 1907.
What are the odds of 31 days in a row of the exact same temperatures to a tenth of a degree?
How could August 1907 be an identical repeat of August 1867 to a tenth of a degree every single day?
Some readers will recognize Geoff Sherrington’s name because for years he was an integral part of the group of volunteer BOM auditors that published many examples of strange, dubious, misleading, or unbelievable things in the Australian BOM records.
I have a copy of the full report and will publish it if Sherrington and Berger give permission (possibly after editing).
The Australian has published two articles today and an editorial that calls for an independent forensic audit.:
Bureau of Meteorology admits errors after audit reveals duplicated historical temperature data
By Graham Lloyd, The Australian
Meteorological records from 1880. | Bruce Miller, CSIRO
Investigator Tom Berger, a former computer engineer and specialist in data-based deception analysis, said the bureau had “acknowledged a few trees but denied the existence of the forest”. He said the anomalies identified were so statistically unlikely they would be like winning the lottery every week for a year.
Mr. Berger argued that whatever effect the errors might have had on temperature trends, the presence of copied, duplicated and incomplete data had compromised the dataset to the point that it was no longer reliable for serious big-data analysis or machine learning.
“These are not subtle statistical anomalies; they are obvious patterns of human intervention that should have been flagged by any competent quality control system,” Mr. Berger said.
“This is not a transcription error by the same observer – it is a database‐level duplication that persisted through multiple generations of data management, across different computer systems, and through every version of the bureau’s quality control and homogenisation software.
The bottom line is that we don’t know what effect these errors have made on climate trends or seasonal analysis yet. But we do know that The BOM has not done even the basic checks on their own data to make sure it is accurate. They tell us how important they are and how accurate everything is, but clearly, they are not behaving as though life on Earth depends on getting the answer right.
Last word goes to Lance Pidgeon
Eleven years ago he trawled manually, laboriously, through BOM Maps of central Australia from the World War I era and found places with no thermometers until 1940 repeated the same temperatures day after day and year after year from 1911- 1917. He found the exact same squiggly temperature patterns “recorded” every year. So, we have known, and the BOM has known that there are issues with the data for a long time.
Long live citizen science.
***NOTE to commenters: Please refrain from casting aspersions or using the f-word (the fraud word). Right now, the BOM data needs a full independent forensic audit before conclusions can be drawn.
*CDO = Climate Data Online
^ Correction: “82 years of daily data was duplicated” should read the gap between duplicated data was 82 years.
This article originally appeared at JoNova
Go to Top
The post BOM’s scientific scandal: The raw data is not raw was first published by the Committee For A Constructive Tomorrow (CFACT), and is republished here with permission. Please support their efforts.
