The greater New York region, having been broken into disconnected and damaged pieces by Hurricane/Nor’Easter Sandy, is still reassembling itself. Every day sees improvements to electrical grids and mass transit and delivery of goods, though there have been many hard lessons about insufficient preparations. Here’s an impressive challenge: over a million people and thousands of businesses lack electrical power; therefore many of them are running generators, to stay warm, keep food cold, and so forth; but the generators require fuel, typically diesel or gasoline; and so there is a greater need for fuel than usual; but a significant fraction of the petrol stations can’t pump fuel for their customers… because they lack electrical power and don’t have their own generators. These and other nasty surprises of post-storm recovery should be widely noted by policy makers and the public everywhere, especially in places that, like New York when I was a child, rarely experience disasters.
Unfortunately, another storm (a simple nor’easter) is now forecast for mid-week. While much weaker than the last, it is potentially still a dangerous situation for a region whose defenses are still being repaired. As was the case with Sandy, the new storm was already signaled a week in advance by the ECMWF (European Center for Medium-range Weather Forecasting), the current European weather-forecasting computer program or “model”. Confidence in the prediction has been growing, but still, predictions so far in advance do change. Also one must keep in mind that a shift in the storm’s track of one or two hundred miles or so could very much change its impact, so the consequences of this storm, even if it occurs, are still very uncertain. But again we are reminded, as we were last week, that weather forecasting has dramatically improved compared to thirty years ago; the possibility of a significant storm can now often be noted a week in advance.
What is this European ECMWF model? what is its competitor, the US-based GFS (Global Forecast System) model? And what about the other models that also get used? All of these are computer programs for forecasting the weather; all of them use the same basic weather data as their starting point, and all have the same basic physics of weather built into their computer programs. So what makes them different, and more or less reliable than one another? I asked one of my commenters, Dan D., about this after my last post. Here’s what he said, along with my best (and hopefully accurate) attempts at translation for less experienced readers:
Both the GFS and ECMWF models are global models based on the primitive equations (i.e. the Navier-Stokes equations plus the thermodynamic energy equation and mass conservation equation), and both initialize their grids with mostly the same atmospheric data gathered from around the globe.
Editor’s translation: this means that they use the basic physics equations for the motion of fluids (such as air and water vapor), of energy, and of mass; and they both start with the same type of weather data taken from around the world.
One main difference between the two is the grid resolution. The GFS runs at a grid resolution of roughly 27 km in the horizontal, while the ECMWF runs at roughly 16 km (technically both don’t actually use grids, but rather spectral decomposition; the above are the effective grid spacings). All other things being equal, having a finer grid spacing will generally improve your forecast, since you can resolve more of the relevant weather features at smaller scales. This is assuming that your assumptions about the subgrid scale features are still valid when you change the grid resolution, which is a big assumption. There are other caveats, but I won’t go there.
Editor’s translation: the atmosphere is far too complicated to be simulated in great detail by a computer program; there just aren’t big enough computers for that. Instead, every computer program is designed to make approximations and simplifications. The approximations made by the ECMWF allow the program to keep track of more of the small details of what the atmosphere is doing, compared to the GFS; this may allow the ECMWF to be more accurate, although this depends on whether its assumptions are ok in other ways.
The other big difference between the ECMWF and the GFS, and probably the more significant one, is the method of initialization. Both use statistical variational techniques whereby weather observations are statistically optimally combined (in the least-squares sense) with a background “guess” field for the model state variables (such as temperature, moisture, wind speed and direction, etc.). The GFS takes observations centered around the given initialization time (such as 0000 UTC) and makes an analysis at a single time, using a technique known as 3DVAR. This analysis is then fed into the model grid and the forecast is launched from there. The ECMWF does a very similar thing, except that instead of utilizing an analysis at a single time, it actually uses an enhanced technique known as 4DVAR, wherein the model itself is run forward and backward in time over a certain interval (I think it’s on the order of 6 hours or so, I’d have to check), assimilating observations from within that entire window. The forecast “trajectory” is optimally corrected over several forward-backward iterations to best fit the “trajectory” of the observations within that time window. The final analysis valid at the end of that window is then used to make the subsequent forecast. Because 4DVAR makes use of a longer time window and tries to correct the model forecast trajectory, while 3DVAR merely tries to correct an initial guess valid at a single time, the former generally yields a much more accurate final analysis. The disadvantage is that 4DVAR is far more computationally expensive than 3DVAR, and one needs to create an adjoint (backward-in-time version) of the entire forecast model code, which is nothing short of a nightmare for these very complex codes.
Editor’s translation: a useful analogy here would be this: it is as though the GFS is looking at a snapshot of the atmosphere and using the information to predict the future, while the ECMWF is looking at a short video of the atmosphere and making sure that its prediction of the future is consistent with the whole video clip. This technique, while difficult to implement, assures that ECMWF is more accurate.
Both of the above contribute to generally superior forecasts by the ECMWF for most situations (in fact, I think a study was done not so long ago in which the ECMWF 4DVAR was used to initialize the GFS forecast model; the results showed a significant improvement in the GFS forecasts, nearly the same accuracy as the typical ECMWF. I’ll see if I can dig it up). The next obvious question is why the GFS doesn’t follow suit with using 4DVAR and higher resolution. The reasons are complex, but partly because (as I see it, at least) the ECMWF center only has to deal with one model, while the U.S. weather enterprise is concerned with multiple models at different scales, and thus has to spread their resources more thinly. The full GFS output is also freely available to anyone, while the ECMWF output is not. Personally, I’d like us in the U.S. to focus on a more unified weather modeling framework, while still keeping everything open access.
Editor’s note: I don’t really know exactly what this means, but it sounds as though the US system is trying to do more than the European one, while the Europeans, while being more focused, have been able to develop more advanced simulation techniques. Not sure about this.
Finally, what makes the modern versions of the GFS/ECMWF so much better than their predecessors is a combination of increasing model resolution, better numerical solution techniques, better physical parameterizations of clouds and precipitation, radiation, surface fluxes, and the like, and better initialization procedures (such as 4DVAR). The basic equations that the models are based on, however, have not changed much, only the solution details and how we handle the complex parameterizations of the other important physical phenomena not directly related to fluid flow.
Editor’s translation: Although the basic physics that goes into the newer simulation tools isn’t that different from what was in the older ones, there have been many incremental improvements, on many different fronts. In other words, it was many small steps, rather than one big one, that makes the newer models better.