#icoads — Public Fediverse posts

Open Ocean #8

The next step is to extend the time series back in time. To do this, I need a different data source to IQUAM. IQUAM looks like it has been discontinued as of May 2025 so it couldn’t be used for updates either. This is a great shame because it’s a useful dataset1 and there’s nothing else out there that’s quite like it.

The main repository of historical marine in situ data is ICOADS, the International Comprehensive Ocean Atmosphere DataSet. I used the version from NCAR. It contains marine meteorological reports from the past 300+ years. Typically, SST datasets start around 1850 because there are very few SST data before then2. There are other observation, including air temperature measurements, which were used in the GloSAT dataset to push global temperature estimates back to the 18th century (1781).

The reasons I didn’t previously use ICOADS is that it’s in an awful format: IMMA. I mean, it’s a great archival format in that it contains everything and the kitchen sink, but decoding it is a real pain. There is official Fortran code for reading the files, but I’m not working in Fortran. There are NetCDF files which contain a subset of the data, but, alas! for my needs, the wrong subset. Fortunately, Philip Brohan has written a nice python reader, pyIMMA which will decode IMMA.

Even with a decoder in hand the downside of ICOADS is that it takes forever to read the data. To mitigate this, my workflow includes a step where the data are read in and then the elements I need3 are written back out in csv files. I like csv because I can look at the data4 and because it can be read much more efficiently. It’s not necessary to do this reading-rewriting step, I guess, but makes it much easier to rerun the later stages of processing which is something I generally do a lot5. Also, ICOADS is weirdly heterogeneous and sometimes looking at the data is the best way to troubleshoot a particular problem.

Of course, the upside of ICOADS is that now we have lots of nice metadata to play with. Additional metadata can be appended to the ICOADS files via the unique ID added to each record. One major shortcoming common to both HadSSTs 3 and 4 was that the error covariances were incomplete because some of the reports in ICOADS don’t have a meaningful ship IDs, or have a generic ID like “SHIP” or “MASK6“. These reports were excluded from the error covariance calculation7, so the uncertainty would generally be an underestimate. How much of an underestimate varies through time. Since HadSST4 was developed, a method for “tracking” individual ships has been published8. It would be cool to be able to combine the two but that’s for later. Right now, I’m just trying to get ICOADS plumbed in.

One thing I did want to do during the plumbing was write in some functionality that let me add bits to the error covariance depending on arbitrary labels in the data. Previously, correlated errors were of two kinds: (1) correlated for all measurements from one ship and (2) correlated for all observations everywhere. Adding correlations based on arbitrary labels means I can estimate biases associated with measurement method, decks of data, or countries. The way to do this isn’t any different from doing it by ship ID: group the observations based on the label and then add up the components for each unique label9.

So… For June 1855, the SST anomalies gridded at 5 degrees look like this.

The 1850s were cool relative to the 1991-2020 climatology I’m using, but there are still some warm features in various places. The “deck” biases, which are the biases associated with individual “decks” of data in ICOADS look like this:

The biases are relatively modest amounting to a few tenths of a degree. The individual ship biases are larger and correspond to some of the features we saw in the gridded data.

When the data are sparse like this, it’s much easier to see how some individual ships affect the estimated field and the extent to which the interpolation deals with (or fails to deal with) examples of measurement error. The warm and cold biased ships in the South Atlantic really stand out, but there are also contrasts between ship tracks in the Pacific and Indian Oceans, though they’re more subtle. The data gridded at 1 degree resolution highlight some of these biases more clearly.

At 1×1 resolution, it’s much easier to see the correlate nature of some of these features. You can also see that even where ships do have a bias, there can still be variation in the anomaly along the ships path that is informative about changes in the local SST anomaly field.

The interpolated field looks like this:

Not only is the field smooth and less noisy, but it also balances potential errors in individual ships (and decks, though that’s much harder to see) against independent information from other ships and areas. This is most obvious where there are strongly biased ships in the basic gridded field. The interpolation seems to ignore those while picking up the background features. Of coures, it doesn’t perfectly discriminate, but the partitioning of the variance does at least take into account the possibility of measurement error.

One problem with these early SST anomaly fields is that the mean anomaly is rather negative. We are a long way from the climatology. An anomaly of zero is the prior estimate of what the SST field should be and this causes problems because the interpolated field is pulled towards the prior10 particularly if there are terms in the measurement error covariance with large spatial correlations. There are such terms (and more than one of them) so the interpolation will tend to be too warm.

In the period from 1980 on, we would have the opposite problem – the SSTs at the end of the series are much higher than the climatology – were it not for the drifting buoys. They are assumed to be unbiased, so they provide a strong observational constraint which means the posterior ends up centred on the observations with little influence from the prior. How to do this in the 1850s?

• We could bias adjust the data. The problem are the large correlated errors, so if we can correct for those by adjustment then the errors will be reduced and the error terms remaining in the covariance will be smaller and less strongly correlated.
• We know that the data are biased, so we could remove the global correlation term from the covariance matrix.
• Knowing that the data are biased, we could also modify the mean of the prior using some kind of temporally averaged global mean, or low-frequency background field. We could also increase the variance assigned to the global mean in the prior11.
• We could build in some kind of time dependence. While one month of data might not be able to pull the interpolation away from the prior, a sequence of months might. However, bias errors are also correlated in time, so we might just end up with the same problem.
• We could find a benchmark dataset. ERSST used marine air temperature. DCENT used land stations. HadSST4 used marine air temperatures and oceanographic data.
• We could use ERSST or DCENT to provide our first guess somehow. This feels a little like cheating but it could provide the scaffolding necessary to build everything else.

It really depends on what we want to achieve and in what order. My inclination is to calculate a time series from the un-interpolated data, smooth it in time and use that as a first guess. In that way, I would have a dataset that while biased at a global scale, has much reduced biases at a local scale. That might be an interesting thing to look at, but I need to think more about it.

-fin-

1. When updating a dataset, it can help to have more than one source of data in case – as inevitably happens – your major source goes down just when you need it. ↩︎
2. There are times after that too which aren’t well observed. The 1860s have very poor coverage in the Pacific and more generally (recession, civil war, recession in the US), so another logical starting date is the 1870s. ↩︎
3. Of course, I don’t always know exactly what I will need until I need it, so I might end up having to run the extraction multiple times. In the end, I don’t want to be reading masses more data than I strictly need. ↩︎
4. And even open it in Excel. I find Excel useful for some tasks as it can easily do things that are tedious to do in Python or a Jupyter notebook. ↩︎
5. I like to rerun things over and over again and then sit there looking at the output over and over again. There are around 2000 months between 1850 and the 2020s, and generally the code outputs several different diagnostics per month, so during dataset development, I would look at tens of thousands of images over and over again. It’s a fun process trying to puzzle out what’s going on in each image, why it is the way it is, if it’s the way you expected and if not, why not. After repeating this process many time, I end up with something I hate but can’t see how to improve*. ↩︎
6. Some ships hide or “mask” their callsign when reporting on the GTS – Global Telecommunication System. There are various reasons given for this including safety and commercial concerns. It’s annoying for someone working with the data, but there are workarounds. ↩︎
7. HadSST3 bodged them back in with a correction factor for certain time series, but it was impossible to implement in a general sense so for most applications, there was still a bit missing from the uncertainty. ↩︎
8. The math of the tracking is fairly simple, but making best use of the ICOADS data within the context of the maths is a big part of the work. Working with data is 90% data wrangling and 10% everything else. ↩︎
9. You can also look at the components at a per-label level: individual ships, individual decks etc. It won’t always be informative because in areas where there are lots of overlapping covariances, the variance will be partitioned proportionately between them, while the posterior covariance (the uncertainty) will remain large. That’s useful information too of course, but the quantity of information can become overwhelming. When developing HadISST2, I created covariance matrices for every single ship for every single month. Even stored efficiently, that’s a large amount of data: thousands of months, hundreds of ships per month, each of which has a covariance matrix. If I did it again, I’d try to avoid writing any of that out ↩︎
10. Or not pulled away from it, I guess. ↩︎
11. In the HadCRUT interpolation, the global mean is treated differently to what I do here, leading to a different set of interpolation equations. My hazy recollection is that the global mean term has a very elastic prior so this is less of an issue. ↩︎

* Short of going in there and fixing things by hand. I don’t like that way of doing things because it’s not exactly reproducible. Sure, you can enshrine it in code, but at the same time, it feels unsatisfactory, because the best you can say is “it looked funny”. I think there’s space for that kind of approach – we are good at spotting inarticulable oddness – but it doesn’t scale. There are billions of data points in ICOADS, so a bulk approach is the only feasible one. Consequently, most manual exclusions happen at a level where the number of entities is still manageable, i.e. whole decks of data.

#climate #climateChange #data #icoads #observations #seaSurfaceTemperature