Learning Python

After years of relying on IDL for interacting with data, I am taking the plunge and switching to Python. The tipping point was deciding that iPython notebooks are a good way of maintaining the links between results, figures and the code used to generate them. My first plot is based on SST CCI data, of course!

Geoscience Data Journal paper

An open-access journal article describing the SST CCI phase 1 datasets was published today.

It is published in Geoscience Data Journal. I think the advent of 'data journals' over the past few years is a good development. The traditional recourse of trying to shoe-horn a detailed data description into a paper with science results was not ideal, particularly for large complex datasets such as those created by reprocessing EO archives for climate.

The new paper is:

Merchant, C. J., Embury, O., Roberts-Jones, J., Fiedler, E., Bulgin, C. E., Corlett, G. K., Good, S., McLaren, A., Rayner, N., Morak-Bozzo, S. and Donlon, C. (2014), Sea surface temperature datasets for climate applications from Phase 1 of the European Space Agency Climate Change Initiative (SST CCI). Geoscience Data Journal. doi: 10.1002/gdj3.20

1. European Space Agency, ESRIN/Contract No. 4000101570/10/I-AM ‘Phase 1 of the ESA Climate Change Initiative SST_cci’

What differences to use in validation?

Validation is the comparison of (in this case) our satellite SSTs with temperature measured in situ, from buoys, ships, etc. Validation gives assurance that the satellite SSTs are, in a general sense, accurate. However, the comparison is complicated by the fact that different SSTs are genuinely different (geophysical differences), so that the difference between any two data points is a mix of error contributions and true differences. In addition, the SST CCI products include a number of SST estimates, each of which requires validation.

We therefore need to be clear about which in-situ/satellite comparisons will be made, and why. This post records the results of a review of our options, following discussions between myself and Gary Corlett (Leicester).

"Raw differences"

Here, we will compare the skin SST from the satellite to the nearest-in-time depth SST of the in situ measurement. In this case we expect certain systematic differences. (1) There is a geophysical difference based on the ocean thermal skin effect, which is typically of order -0.2 K, but also has a wind-speed dependence which should be clear in night-time differences. (2) There should be a trend in the raw difference with respect to the time separation of in situ and satellite: for example, in mid morning the ocean is typically warming, so in situ measurements after the satellite time will tend to be warmer. However, with respect to things that might affect the satellite retrieval adversely (but not directly the skin-depth SST difference) the systematic dependencies should be small; for example, a systematic effect in the raw differences with latitude should be no larger than we might be able to account for by the fact that mean wind speed (and therefore skin effect) varies with latitude.

"Skin-skin differences"

The idea here is to estimate the skin and depth effects at the time of the satellite observation and add these to the in situ observation. The in situ history is first interpolated to the satellite observation time, so giving an estimate of SST-20cm at the location and time of a satellite SST. Any 20-cm-to-subskin stratification is estimated using a model (usually small, and only ever large for day-time cases) and the skin effect is also estimated using a skin-effect model. There should ideally be no systematic effects with respect to latitude, wind, satellite-buoy-time-difference, etc, because the models are meant to account correctly (on average) for all the geophysical differences (other than those from comparing a point to a pixel, which are assumed to add zero-mean noise). This measure therefore tests the combination of "retrieval + adjustment for skin effect + adjustment in depth".

"Depth-depth differences"

In the SST products there is an adjustment provided that can be added to the fundamental satellite SST retrieval (of skin SST at the time of the satellite) to give an estimate of SST at typical drifting buoy depth (~20 cm) at standard local times of day (10.30 h or 22.30 h). To explore this, the satellite SST-20cm estimate for a standard local time will be differenced with the spatially-matched in situ SST history interpolated in time to the same local time. There should ideally be no systematic effects with respect to latitude, wind, satellite-buoy-time-difference, etc, because the adjustment is meant to account correctly (on average) for all the geophysical effects (other than those from comparing a point to a pixel, which are assumed to be zero-mean noise). This measure therefore tests the combination of "retrieval + adjustment for skin effect + adjustment in depth + adjustment in time". Compared to the skin-skin difference, this tests in addition the time-adjustment of the SST for the diurnal cycle.

"Daily mean depth-depth differences"

Although not generated in existing datasets, there has been a requirement which we are considering to estimate an adjustment to be added to the skin SST from the satellite which would give an estimate of the daily-mean SST at the location of the satellite observation. The day over which the mean is to be estimated is the UTC (i.e., GMT, not local) day which includes the time of the satellite observation. The in situ data would therefore consist of the average of the history of the in situ measurements over a 24 hour period. This comparison therefore tests "retrieval + adjustment for skin effect + adjustment in depth + adjustment to daily mean", and the spread of the results will include the uncertainty effect of estimation of the daily mean SST from a single observation. Systematic effects in the differences should be small.

"System Maturity" CORE-CLIMAX style

The SST CCI project is to create both new SST data and to prototype a system for how this SST climate data record can continue to be routinely provided in the future. The "system" is a processing chain that takes inputs (satellite radiance data, auxiliary data etc) and transforms these into SST products. It consists of something like 100000 lines of code, installed at the facility for Climate and Environmental Modelling from Space (CEMS) in Harwell.

CORE-CLIMAX [no kidding] is a European project. Within its scope is development of a means of encapsulating how "mature" systems for delivering climate data records are. It is quite instructive for a team like ours to evaluate itself against the various criteria in the "System Maturity Matrix" they propose. We just did a self evaluation, for AVHRR and analysis products, and find that in its current state, the project straddles a "research capability" and an "initial operations capability" in most areas. That seems right -- it is exactly where we would expect to be at this stage, working on science and also towards a functioning, sustainable system.

Here are our self assessment results. The green shaded boxes show the range of "maturity" of different aspects of the project within each of the metric categories (software readiness, metadata, user documentation, uncertainty characterisation, feedback/access and usage).

Heat Content vs Surface Temperature

The Guardian have (after inputs from ESA) updated the article discussed here to say that the "lousy indicator of climate change" comment applied to surface air temperature rather than sea surface temperature. Well, I suppose that is slightly better from my point of view, but I still don't agree with the point!

"Global surface temperature" time series are generally made up of surface air temperature over land and sea surface temperature over the oceans. Together, surface temperature, precipitation, wind and solar radiation are the principal elements that define our experience of weather and climate. These matter to people, and we need to present any climate changes (anthropogenic and natural) in terms of these elements. Surface temperature is not a simple indicator of the accumulated heat in the climate system, that is true. But it is an indicator of great relevance to society and the environment.

Did ESA say our data are "lousy"? [Answer: No]

Talking last Friday at the Royal Society (London) about ESA's Climate Change Initiative programme was an interesting experience. I gave a presentation on the title of "Ocean Warming". The idea of the talk was to argue that signals in data from the CCI teams are consistent with an "oceanic heat burial" hypothesis published in March this year.  This is the idea that surface temperatures during the 2000s have been fairly level (see Figure 1 below), but at the same time, the ocean overall has continued to gain heat because of greenhouse gas forcing of climate. This apparent contradiction is resolved by changes in tropical circulation that have 'hidden' heat below the ocean surface over the past decade. Patterns consistent with the associated circulation changes can be seen in sea surface temperature, ocean colour and sea level data from CCI.

Figure 1. The measurements from space (red line) are SST CCI data, which show the same year-to-year picture of sea surface temperature changes as the in situ only data from the Hadley Centre in blue. Within the next 3 years of the project, we aim to extend this time series at both ends.

There were other talks, too, including a very positive introduction by the Rt Hon David Willetts (Minister for Universities and Science), one on the cryosphere as seen in CCI data by Andrew Shepherd, and talks by representatives of ESA. 

So, it is interesting what the press picked up. The Guardian have published an article following the meeting with the headline Apparent pause in global warming blamed on 'lousy' data. Within the article it says:

Now, Stephen Briggs from the European Space Agency's Directorate of Earth Observation says that sea surface temperature data is the worst indicator of global climate that can be used, describing it as "lousy".

If you read it quickly, you might think that ESA meant our SST data are "lousy"!

In fact, the point being made was that the energy required to warm just the ocean surface and the surface air temperature is a tiny part of the total energy that the Earth is gaining because of greenhouse gas forcing of climate (see Figure 2). Turning it round the other way, this means that variations in the rate of surface temperature change do not necessarily imply that the Earth has stopped gaining heat. The heat can still be going into other, much more dominant, components of the climate. This is scientifically correct, and indeed, my talk showed a specific example of that. (There will inevitably be year-to-year, decade-to-decade variability in surface temperatures -- weather doesn't stop because of global warming.) 

So, Stephen Briggs' words did not mean he thought our SST data were terrible, despite the impression given by the headline!

In my view, there are many compelling reasons to use surface temperature as an indicator to describe global climate. People experience temperature directly (albeit, subjectively), it is relevant to human comfort and health, temperature (with wind) drives evaporation, it is relevant to agriculture (on land) and fisheries (at sea), we have instrumental records of temperature going back over one hundred and fifty years, etc.

In contrast, the total energy gain in the climate system would seem rather remote to most people, I expect -- although it would be great if everyone understood physics and climate science sufficiently well to grasp its significance.

Figure 2. Analysis of heat content of climate system, from IPCC AR5 WG1 Ch3 Box 3.1.

New version of SST CCI ATSR dataset -- v1.1

A bug-fixing upgrade to the SST CCI ATSR data is now available from the data centre. Relative to v1.0, the newly released v1.1:

• Completes the record up to the end of the AATSR mission (April 2012, cf. Dec 2010 for v1.0)
• Fixes persistent holes from missing data after midnight
• Restores missing files
• Fixes various metadata bugs

The DOI of the new version is 10.5285/79229cee-71ab-48b6-b7d6-2fceccead938.