There are few among us who remember a time without weather forecasts. That well-groomed, resplendent TV presenter pointing to charts and squiggly lines, complete with oddly placed coloured triangles, has become a staple of western society.

More recently, we turned to the Web for our forecasts. But where did the modern weather forecast originate? And where is it created now?

Lewis F. Richardson and his computers

In 1922, a man named Lewis Fry Richardson become my personal hero. He achieved this very distinguished status despite the fact that he failed so spectacularly at the very thing he is famous for. In 1922, a "computer" was a person who knew mathematics and had access to a pen and some paper. Richardson was in possession of both.

He was also a very talented mathematician and meteorologist, and it occurred to him that it might just be possible to use the equations of motion, for which we have Isaac Newton to thank, to predict the weather.

Think of it this way: if I have a ball and I apply a force to it, say, I throw it directly forward, will I be able to "predict" where it will go? You bet I can. (It'll fly forward, duh.) So why not do this with air? All you need do is figure out what forces are at play.

And so Richardson grabbed his pen and paper, and had a try. He figured out all the forces a parcel of air might experience, and stepped the location of the air forward in time.

As an example, he used a domain over western Europe and tried to make a weather forecast of pressure that spanned a 24 hour period. This was the first "numerical" weather forecast. The math of this one forecast took him two years to complete.

Richardson predicted a rise of 145 hPa in 6 hours, a result which is basically laughable, most likely souring his dream of having a warehouse full of computers (people with pens and paper), who would pass notes around to construct a forecast faster than the speed of the evolving weather.

The thing is, Richardson's method was correct; he was just missing a critical piece of the puzzle.

My old meteorology professor in Ireland, Peter Lynch, wrote a book about Richardson's forecast. He remarked that Richardson neglected something called "initialization". Without it, it's like trying to predict ocean tides by using waves. You need to smooth out the high-frequency stuff before you can see the true signal.

Charney, and the scariest maths you ever saw

By the 1950s, things were coming together. The initialization problem had been solved by some extremely smart people, such as Jules Charney and John von Newmann, who derived the quasi-geostrophic potential vorticity equations (which I can barely spell, much less understand).

By then, the first true supercomputer, known as ENIAC, was constructed by the Americans. The new "numerical weather forecast" was the ideal problem for giving ENIAC a test drive.

ENIAC's forecast went well, and numerical weather prediction (NWP) was born. The ingredients were all there: initialization had been solved, the equations of fluid dynamics had been discretized (split up into chunks of time, called timesteps), and the computational power had finally become available.

NWP today

Casually brushing over the following 70 years of in-depth NWP research, let's skip forward to today. What does Richardson and Charney's legacy look like now? Windy is cutting-edge in the field of NWP visualization, but try to remember that Windy is not an NWP institute.

There is maybe one NWP institute in each of the world's most developed countries – notably the US, UK, Japan, Canada, France, Germany and Norway. These are often government run, but private organisations are beginning to develop their own in-house models now as well (notably Meteoblue and IBM).

If you've used Windy for a while, you've probably heard of ECMWF. I've done courses/conferences at ECMWF, and these visits have helped to convince me that they're the best in the world at what they do.

ECMWF (it stands for European Centre for Medium-range Weather Forecasts) was incepted in 1975, as a collaborative effort throughout most of Europe (18 countries).

They're based in Reading, a suburb of London, but soon they'll move elsewhere in Europe (possibly Italy). ECMWF is a modelling centre, and its model is known as the IFS (Integrated Forecast System) — though many just know it as "the ECMWF model".

The UK Met Office are also world leaders when it comes to weather modelling – their model is known as the UM (Unified Model). The UM is second only to the IFS in its accuracy (though the accuracy rankings are always changing). The Met Office's UM partnership extends across the world, allowing UM access for many nations including Australia, New Zealand, South Korea and India. It's costly data though, and commercial UM access is prohibitively expensive.

Not be left behind, the American NOAA (National Oceanic and Atmospheric Administration) is also a world leader in NWP, with its flagship model the GFS (Global Forecast System). One important point of note about the GFS is that its output is completely free of charge. You don't even need to sign-up to grab the GRIB files from their server.

Ever wonder why there are so many weather apps? Free data for all. Not just that – you can literally sell this data. You can download NOAA's data, make it pretty, and actually sell it. All you need is enough coding knowledge to read a GRIB file and make a website. Some people do this better than others...

And lest we forget HIRLAM, AROME, ICON, CAM, WRF, NEMS, GRAF, and the whole host of other state-of-the-art NWP systems produced by organisations across the world. These days, we're spoiled for choice. But it's important not to forget the resources it takes to create a weather model, much less a competitively accurate one.

So, what's the lesson here?

If you want to get a little better at understanding a weather forecast, the first thing to do is to pay attention to where the data came from. Did it come from NOAA? ECMWF? Meteoblue? Each model will run at a different resolution, sure, but resolution isn't everything.

The skill of the scientists who coded up the model comes into play, and the approaches taken to every component of the model will matter immensely. Are you using a model in the US that was tuned to work best in the UK? Is ECMWF the best model for everyone, or does ICON actually perform better in your location? By how much do the model forecasts disagree on any given day?

On Windy, a good start is to play around with the different models. At the time of writing, the available models are the Swiss NEMS (uses machine learning AI, which is cool), France's AROME, Germany's ICON, NOAA's GFS (and NAM, a North American nest domain), and ECMWF's IFS. Some of these are "global" domains, and some are "limited area models (LAMs)", but I think I'll leave these concepts for a future article.

It's a tricky business, and it can be overwhelming for the average weather user. Many will simply look at ECMWF and that's good enough; it depends on what you're using the forecast for.

There are more advanced strategies in modelling such as ensemble forecasting (a technique that tries to get a handle on the chaotic nature of weather), but observing things like "postage stamps" and "spaghetti charts" may not be helpful to the uninitiated.

If you're interested, and bear in mind that I'm heavily biased, numerical weather prediction is a rabbit-hole that will leave you more and more fascinated the deeper you go.

After three years of doctoral study in numerical weather prediction, I recently forced myself to ask the question: what have I learned? Or more to point, what do I now know that might actually be useful to the average user of a weather forecast? Today, I'm going to discuss model resolution.

I often hear people talk about model resolution, with varying degrees of understanding about what it actually means. To put it simply, think of a giant grid – laid out evenly across the earth.

The resolution of the model is the distance between each node on this grid. One thing you might notice about doing this on a sphere: the "resolution" in the east–west direction is largest at the equator, and becomes vanishingly small near the poles.

Take the American GFS (now a.k.a the FV3) model. Often it is quoted as showing a resolution of 22km, but other times, it seems to be 27km. In actual fact, it has a resolution of 0.25°. Just google for a latitude–longitude distance calculator and you'll see that 0.25° is 27km at the equator, but its 22km over Washington DC.

Increasing resolution towards the poles is a major problem in NWP, but some really cool solutions are emerging.

Now, if you have more computing power to throw at the model, you can decrease the spacing between the grid boxes. Though this kind of computer power has traditionally been kept in-house by government Met agencies, a shift towards cloud (no pun intended) providers like Amazon's AWS is starting to take hold.

ECMWF's IFS model (the HIRES configuration that Windy uses) runs at 9km (remember that 9km is an approximation). There are some very obvious benefits to doing this. With more resolution, you can resolve more weather features, such as the downwind vortex spinning off the Big Island of Hawaiʻi in the image below – which the GFS (left) cannot see, and the IFS (right) can.

Higher resolution means that orography can be better resolved, and smaller-scale features can appear. This, in general, is a good thing.

In the interest of accuracy, I should mention that the GFS and IFS are actually 'spectral models' – they don't use a grid at all, and their "resolutions" are inferred.

Spectral models are created by summing massive amounts of trigonometric functions, and they're developed by people that are smarter than I'll ever be. Luckily, spectral models need to be converted to a grid at some point anyway – so it's pretty harmless to think of them this way.

Before I go on, I should reiterate – high resolution, where possible, is usually a good thing. We push for it. I could show you a thousand examples to support this. But before we go on a supercomputer shopping spree, I'd like to talk about a less popular topic – the problems high resolution has now opened up.

The requirement of conjecture

When talking about resolution, people frequently skip over an incredibly important topic – parametrization (of which, incidentally, nobody seems able to agree on the spelling). Without parametrization, weather forecasts would be total and complete garbage.

Within each grid box (behind the scenes, if you will), assumptions get made, and these assumptions are what makes the model work.

For instance, we don't model every photon coming from the sun. We just assume that the sun gives us X amount of Watts per square metre in location Y. We also assume that cloud droplets develop into raindrops, and melt, or condense, etc. – according to certain laws and relations that scientists have derived.

Parametrizations allow a model to work by filling in the gaps; they're also known as the "physics" of the model (while grid-scale processes are known as the "dynamics" of the model).

The grey zone

Another assumption that we make is that thunderstorms will form and dissipate where they should; passing, rain, lightning, and turbulence back to the grid, where we can see it in the output. In a convective parametrization, the updrafts within the storm cloud are not actually modelled.

The thing is, there's no reason why they can't be. Since storm clouds tend to be something like 10km deep, a model with a resolution of about half of this (known as the 'effective resolution') should, theoretically, be able to explicitly represent it. And the model will indeed try. And it will fail, spectacularly.

The 'grey zone' is a regime in NWP in which the model tries to resolve the updrafts and heat transfer in turbulent eddies and convective showers, but doesn't have the resolution to do so properly. It gets 'stuck' between the two; it can't resolve the feature, but can't parametrize it either. This leads to a grid dependence in the model, which is a very bad thing.

ECMWF will soon roll out their new 5km HIRES configuration. Hopefully, they've developed something to help out with the grey zone issue – 5km is right on the edge of starting to resolve deep convection. (4km is generally accepted as the limit of 'convection permitting' models.)

When the resolution gets too high, the assumptions that make parametrizations work start to break down, and we must either find a solution to this problem, or simply skip over the grey zone entirely and resolve each thunderstorm explicitly – something we can kind-of do in high-resolution nested models (like AROME), but certainly not with global models (yet).

Too much detail?

The next question is, how much detail do we actually want? If you had limitless computational resources, and could run your NWP model at, say, a resolution of 10 metres, would you really want to?

Let's say you need a precipitation accumulation across a twelve-hour period for your racecourse on race day. Do you need to know how many showers happened overnight? Or let's say you're a sailor, wondering whether tomorrow is a good day for a sail. Do you need to know what pattern of eddies the wind will create as it bounces off the yacht club building?

I was once a forecaster for commercial aviation. One day, we received word from HQ that a new model, known as the EURO-4, was being implemented. With the EURO-4, we had now jumped from 12km resolution to 4km. Suddenly, our once accurate display was giving us very odd wind speeds over our airfields.

Why? The new model was able to resolve convective updrafts. This was technically more representative of the real world. But we needed the average speed to make a properly representative forecast, and so had to smooth out the extra detail, just to make the forecast usable.

In summary...

As is common in science, we must drive ahead, but with a healthy dose of scepticism. Higher resolution is definitely the future. But NWP is so often misunderstood, especially now that the level of detail available is allowing us to dispense with once-essential guidance – human forecasters who spent their days drawing fronts, and using conceptual cyclone models and empirical techniques.

As with many other areas of industry, computers are rendering humans redundant. But in the rush to do so, we must not lose the knowledge that every meteorologist once possessed, knowledge that we, as direct model users, do not have – the limitations of each individual NWP model.

NWP will not be devoid of problems for many decades to come, and we, as users, must always be willing to ask the hard question: Does today's forecast represent the pinnacle of state-of-the-art NWP? Or, is it total garbage?

Yesterday I tweaked up my old Windy SkewT plugin and thought it long past time to give the Windy community a rundown of what this crazy-looking diagram actually means, and why it's the most data-rich diagram in all of meteorology – if you know how to use it.

What's a SkewT, and why do I care?

One day during my forecaster training, our instructor handed each of us an odd-looking printout. It looked like this:

"WTF is this!?" was my first thought. Truth be told, it terrified me. And I'll be honest: a SkewT is not for the faint of heart. The general public will rarely, if ever, become acquainted with it. Those who understand those straight and curvy lines will usually fall into the categories of meteorologists, pilots, balloonists, gliders, and storm chasers. However, on a different occasion during training, I asked my instructor: "If you had only one tool available on the weather-forecasting bench, what would it be?". "A SkewT", she replied.

Here's what a SkewT can tell you about, with no other information to hand: Convective cloud, upper-level wind, layer cloud, thunderstorm development potential, wind shear, upper-level instability, temperature, dewpoint, humidity, tropospheric depth, potential instability, conditional instability, fog point, adiabatic processes, airmass type... I'll be honest, the list goes on and on.

Interested yet?

Okay, let's dig in...

A weather balloon has attached to it a piece of tech called a radiosonde. A radiosonde is essentially just a sensor that reads the temperature and humidity of the air as the balloon ascends, and GPS allows it to deduce how it is moving (giving us wind data). By the time the balloon has entered the stratosphere (about 20 km up), it's expanded to about the size of a double-decker bus – then the balloon pops.

The temperature, dew point temperature, and winds get plotted on the radiosonde like so:

With a little practice, you can deduce a lot from this one diagram. For instance, the first thing I would notice is how dry the air suddenly becomes at around 825 hPa (roughly 5000 ft). This is a clear sign of high pressure, an anticyclone. I would, therefore, expect to see light winds low down, which I do. Below 5000 ft, there is more moisture, and the sounding is a little more unstable. Using just this, I could probably take a guess at the exact time of morning when fair weather cumulus will start popping up in the sky.

The air temperature at the surface in this sounding is at around 20°C. Note the 45° line extending toward the top right from there – this is an isotherm. The reason it's skewed in this way is to make the temperature curve seem more vertical and also to separate out the dewpoint curve from the temperature curve (for readability). This is from where the SkewT gets its name.

SkewT is shorthand for 'SkewT–LogP diagram'. The LogP part relates to pressure, which is plotted on a logarithmic scale. This makes sense, as pressure decreases exponentially with height in the atmosphere.

The dotted lines are isohumes – lines of equal humidity.

And then we have the adiabats. Now, our understanding of atmospheric thermodynamics is actually quite sophisticated. We know a lot about how changes to fields like temperature or pressure can affect other fields, like humidity or entropy. This is where the real magic of the SkewT happens. A parcel of air lifted to a different pressure will cool, but conserve its entropy. This allows us to do useful things like diagnosing thunderstorm potential before the cumulonimbus clouds even start to form. You can also diagnose the expected time of fog onset, predict a föhn effect, or identify an area of potential instability (potential instability can wreak havoc in rainfall totals). The cool thing is that you can do all of this from one balloon ascent, there's no computer model involved. There's a dry adiabat and a moist adiabat, and these are the remaining lines on the SkewT, but that's all I can really say for now.

Windy plugins, and how to use windy-plugin-skewt

Windy opened up their platform to outside developers a few years ago. Just click on the menu and then Install Windy Plugin to try them out. There's a nice list to choose from, and you'll find the SkewT under the SkewT diagram. You can also try Sounding, which is a related plugin.

When the plugin is loaded, simply open Windy's picker by clicking on any point on the map. Since Windy is a model visualization web app, the data plotted is model data. I also host my own complementary web app at skewt.org, which uses real-time radiosonde data from balloon launches across the world, rather than model data.

So that's it for now! I hope I've piqued your interest a little into the wonderful world of thermodynamic diagrams. As always, please send me a message on the Windy community (username johnckealy) with any questions or comments.

If I might offer an opinion on that subject...

The weather company was acquired by IBM in 2016, with some pretty cool plans to create a super-high-resolution global model. Know as the "GRAF" model, they've managed to achieve a resolution of around 3 km in many parts of the world, using GPUs and other crafty techniques. They assimilate tons of data from the Internet of Things (I think they even use smartphone data).

There's a lot of hype, and yes, if they deliver what they promise it'll be really fascinating. However, with any numerical weather prediction model, I can't understate the importance of waiting to see how it performs! Just look at the successor of the GFS, the FV3. NOAA couldn't even get it to outperform the GFS during testing (hopefully it's doing better now). And it sounds amazing to include so much observational data, but actually there's not a huge amount of research when it comes to data assimilation from the Internet of Things. Non-standard observations can create a lot of garbage, and as we say in NWP – garbage in, garbage out.

Also, bear in mind that even though GRAF is very high resolution, its lead time is really short, something like 48 hours (can't remember the exact number). So it still can't compete with ECMWF in terms of the medium range. And finally, the data is highly unlikely to be free, if available at all outside of Weather Channel applications and media.

If anyone else knows anything beyond the obvious about GRAF (or more generally, IBM's Deep Thunder project), please do share! Googling the subject leads to endless articles with nothing more than info from IBM's press releases... :)

Hey Windy community,

I've just published a new Windy plugin, named

windy-plugin-featuretracker

I've always wanted a tool that could give me timing for convective showers at my location, based on radar; which as we know, the model can't give us. I'm in a co-working space in Thailand at the moment, and I'm hoping this tool will be able to give me timings for thunderstorms before they arrive! It should hopefully work nice for midlatitude polar-maritime airmasses too.

It's in Beta now, please do try it out, the docs are on Github. Comments and suggestion are very welcome.

https://github.com/johnckealy/windy-plugin-featuretracker

Thanks to @rittels and @marekd for the helping hand.

@Gkikas-LGPZ @TomSlavkovsky @ivo @stitch @vicb, you guys be interested to take a look too.

John

Dear Windy users,

If you'd indulge me in a curiosity...

One of the most talked about and requested topics on in the windy community forum seems to be historical weather data. Windy doesn't provide this of course; we're talking about an obscenely huge amount of data. National weather services are usually the only ones equipped to maintain such a vast archive.

What I wondered was, what do you want it for? For example, I know some people are researchers; others perhaps want to know about biases that their specific locations have with respect to model data; maybe others have interests in climatology and climate change; some people might even be dabbling in machine learning.

Please share your thoughts on how you use/want to use historical weather data, and fundamentally: are you talking about historical observational data, historical model data, or both? I'm particularly interested if anyone has experience with machine learning in weather analysis, but that's just a personal curiosity. I'd like to start a more general dialogue about how we use historical data. And if we can clarify the applications of the data, who knows what we, as a community, could achieve?

John

Hi all, (@Gkikas-LGPZ @vicb @rittels @stitch @yves @jacobsjo @marekd )

I did a big update of windy-plugin-skewt over the weekend, was wondering if you'd like to check it out. You can now move the purple slider at the surface temperature to get a parcel ascent. Also, it's now possible to open the skewT and spot forecast simultaneously.

Would really appreciate any feedback or bug reports!

Perhaps those of us who loved the 3D mode were too silent about it! I would also love to see its return :)

Hi,

I've written a new plugin, called windy-plugin-skewt:

https://github.com/johnckealy/windy-plugins-skewt

I've successfully published it with npm, but when I try it out at https://www.windy.com/plugins, I get the following error:

TypeError: Plugin.instance is not a function

The package.json info displays, it just won't run. I realise that's not much to go on, but is there any chance someone at windy could have a quick look? It runs just fine locally using https://www.windy.com/dev. I'm hoping it will be a really useful plugin if I can get it going.

Thanks!

@Yves70 thanks! It's really good to know it's being used, much needed motivation to spend some time improving it (which I'll do when I can!) The README is actually just out of date, that's first on the list :)

Many thanks to @andrestotorica for sending me a PR for windy-plugin-skewt. I think it's working now, I'll try to find some time at some point to properly look it over. I also need to remove the sonde observations as I'm not running the server that processes them anymore.

Hi all, Yes I'm sorry i've had little to no time to keep maintaining the plugin when windy updates break it. Should be working again for now. @ianwightman what green line? All of the background thermodynamic lines are green. Parcel ascents are red.

@rittels Before I spend all day digging into this, do we know what broke everything? An update to Windy, or Leaflet or something?

@shashighedia

The red line is a parcel ascent. Drag the purple slider to the right, and this will change. The concept is that as the surface temperature increases, the energy of a rising parcel of also increases (known as CAPE). By comparing the red curve with the existing temperature profile, you can diagnose the instability (likelihood of showers, thunderstorms etc) of the atmosphere. In theory, you can even use this to predict the exact onset times of convective clouds, or even afternoon thunderstorms. Just google "radiosonde parcel ascent", "convective available potential energy", or "lifted index" to get started.

@ken I wrote SkewT. However, other work has kept me from giving it the love it needs, so I might have to suggest you look at the RadioSonde plugin instead. It's newer, and probably a little more reliable.

Feel free to give me more details in case I can find some time for it though, as I cannot reproduce exactly what you mean.

@EduardoSG

Sorry about that, I thought I'd fixed this already. Thanks for reporting the bug. I've just written a fix (v0.9.3).

Let me know if you still see problems.

Okay, added a fix for that in v0.9.2 🌨️🌞

@Yves70 @Gkikas-LGPZ

I just use the above equation in that calculation, it's just an estimate. Is there a better equation I can use? If not, suggestions on how to approach it? It's been a while since I did this stuff.

Hey @EduardoSG

sorry if I'm being slow, but what's the issue?

z = -H * log(P/Po)

so 500 hPa should be around 7000 m, no?

Wow.... Amazing work guys!