I've seen the Microsoft Aurora team make a compelling argument that weather is an interesting contradiction of the AI-energy-waste narrative. Once deployed at scale, inference with these models is actually a sizable energy/compute improvement over classical simulation and forecasting methods. Of course it is energy intensive to train the model, but the usage itself is more energy efficient.
There's also the efficiency argument from new capability: even a tiny bit better weather forecast is highly economically valuable (and saves a lot of wasted energy) if it means that 1 city doesn't have to evacuate because of an erroneous hurricane forecast, say. But how much would it cost to do that with the rivals? I don't know but I would guess quite a lot.
And one of the biggest ironies of AI scaling is that where scaling succeeds the most in improving efficiency, we realize it the least, because we don't even think of it as an option. An example: a Transformer (or RNN) is not the only way to predict text. We have scaling laws for n-grams and text perplexity (most famously, from Jeff Dean et al at Google back in the 2000s), so you can actually ask the question, 'how much would I have to scale up n-grams to achieve the necessary perplexity for a useful code writer competitive with Claude Code, say?' This is a perfectly reasonable, well-defined question, as high-order n-grams could in theory write code without enough data and big enough lookup tables, and so it can be answered. The answer will look something like 'if we turned the whole earth into computronium, it still wouldn't be remotely enough'. The efficiency ratio is not 10:1 or 100:1 but closer to ∞:1. The efficiency gain is so big no one even thinks of it as an efficiency gain, because you just couldn't do it before using AI! You would have humans do it, or not do it at all.
> even a tiny bit better weather forecast is highly economically valuable (and saves a lot of wasted energy) if it means that 1 city doesn't have to evacuate because of an erroneous hurricane forecast
Here is the NOAA on the improvements:
> 8% better predictions for track, and 10% better predictions for intensity, especially at longer forecast lead times — with overall improvements of four to five days.(1)
I’d love someone to explain what these measurements mean though. Does better track mean 8% narrower angle? Something else? Compared to what baseline?
And am I reading this right that that improvement is measured at the point 4-5 days out from landfall? What’s the typical lead time for calling an evacuation, more or less than four days?
To have a competitive code writer with ngrams you need more than to "scale up the ngrams" you need to have a corpus that includes all possible codes that someone would want to write. And at that point you'd be better off with a lossless full text index like an r-index. But, the lack of any generalizability in this approach, coupled with its markovian features, will make this kind of model extremely brittle. Although, it would be efficient. You just need to somehow compute all possible language before hand. tldr; language models really are reasoning and generalizing over the domain they're trained on.
Obviously much simpler Neural Nets, but we did have some models in my domain whose role was to speed up design evaluation.
Eg you want to find a really good design. Designs are fairly easy to generate, but expensive to evaluate and score. Understand we can quickly generate millions of designs but evaluating one can take 100ms-1s. With simulations that are not easy to GPU parallelize. We ended up training models that try to predict said score. They don’t predict things perfectly, but you can be 99% sure that the actual score designs is within a certain distance of said score.
So if normally you want to get the 10 best design out of your 1 million, we can now first have the model predict the best 1000 and you can be reasonably certain your top 10 is a subset of these 1000. So you only need to run your simulation on these 1000.
Sam Altman has made a lot of grandiose claims about how much power he's going to need to scale LLMs, but the evidence seems to suggest the amount of power required to train and operate LLMs is a lot more modest than he would have you believe. (DeepSeek reportedly being trained for just $5M, for example.)
I saw a claim that DeepSeek had piggybacked off of some aspect of training that ChatGPT had done, and so that cost needed to be included when evaluating DeepSeek.
This training part of LLMs is still mostly Greek to me, so if anyone could explain that claim as true or false and the reasons why, I’d appreciate it
I think the claim that DeepSeek was trained for $5M is a little questionable. But OpenAI is trying to raise $100B which is 20,000 times as much as $5M. Though even at $1B I think it's probably not that big a deal for Google or OpenAI. My feeling is they can profit on the prices they are charging for their LLM APIs, and that the dominant compute cost is inference, not training. Though obviously that's only true if you're selling billions of dollars worth of API calls like Google and OpenAI.
OpenAI has had $20B in revenue this year, and it seems likely to me they have spent considerably less than that on compute for training GPT5. Probably not $5M, but quite possibly under $1B.
So LLMs predict the next token. Basically, you train them by taking your training data that's N words long and, for X = 1 to N, and optimizing it to predict token X using tokens 1 to X-1.
There's no reason you couldn't generate training data for a model by getting output from another model. You could even get the probability distribution of output tokens from the source model and train the target model to repeat that probability distribution, instead of a single word. That'd be faster, because instead of it learning to say "Hello!" and "Hi!" from two different examples, one where it says hello and one where it says hi, you'd learn to say both from one example that has a probability distribution of 50% for each output.
Sometimes DeepSeek said it's name is ChatGPT. This could be because they used Q&A pairs from ChatGPT for training or because they scraped conversations other people posted where they were talking to ChatGPT. Or for unknown reasons where the model just decided to respond that way, like mixing up some semantics of wanting to say "I'm an AI" and all the scraped data referring to AI as ChatGPT.
Short of admission or leaks of DeepSeek training data it's hard to tell. Conversely, DeepSeek really went hard into an architecture that is cheap to train, using a lot of weird techniques to optimize their training process for their hardware.
Personally, I think they did. Research shows that a model can be greatly improved with a relatively-small set of high quality Q&A pairs. But I'm not sure the cost evaluation should be influenced that much, because the ChatGPT training price was only paid once, it doesn't have to be repaid for every new model that cribs its answers.
If its more energy efficient it is doing something different there is no guarantee that its more accurate long term. Weather is horrible difficult to predict and we are only just alright at it. If LLM are guessing at the same rate we are calculating but I am doubtful
Well that was a failed response opps. I am just cautious because while transformers get the random guessing right you can get the right answer statistically but fail on accuracy improvement long term. Clearly this model does better than the current model but extending it to be even better seems basically intractable besides throw more data at it but what if it derived the wrong model you simply cannot actually know
"Area for future improvement: developers continue to improve the ensemble’s ability to create a range of forecast outcomes."
Someone else noted the models are fairly simple.
My question is "what happens if you scale up to attain the same levels of accuracy throughout? Will it still be as efficient?"
My reading is that these models work well in other regions but I reserve a certain skepticism because I think it's healthy in science, and also because I think those ultimately in charge have yet to prove reliable judges of anything scientific.
> My question is "what happens if you scale up to attain the same levels of accuracy throughout? Will it still be as efficient?"
I've done some work in this area, and the answer is probably 'more efficient, but not quite as spectacularly efficient.'
In a crude, back-of-the-envelope sense, AI-NWP models run about three orders of magnitude faster than notionally equivalent physics based NWP models. Those three orders of magnitude divide approximately evenly between three factors:
1. AI-NWP models produce much sparser outputs compared to physics-based models. That means fewer variables and levels, but also coarser timesteps. If a model needs to run 10x as often to produce an output every 30m rather than every 6h, that's an order of magnitude right there.
2. AI-NWP models are "GPU native," while physics-based models emphatically aren't. Hypothetically running physics-based models on GPUs would gain most of an order of magnitude back.
3. AI-NWP models have fantastic levels of numerical intensity compared to physics-based NWP models since the former are "matrix-matrix multiplications all the way down." Traditional NWP models perform relatively little work per grid point in comparison, which puts them on the wrong (badly memory-bandwidth limited) side of the roofline plots.
I'd expect a full-throated AI-NWP model to give up most of the gains from #1 (to have dense outputs), and dedicated work on physics-based NWP might close the gap on #2. However, that last point seems much more durable to me.
> "it's more efficient if you ignore the part where it's not"
Even when you include training, the payoff period is not that long. Operational NWP is enormously expensive because high-resolution models run under soft real-time deadlines; having today's forecast tomorrow won't do you any good.
The bigger problem is that traditional models have decades of legacy behind them, and getting them to work on GPUs is nontrivial. That means that in a real way, AI model training and inference comes at the expense of traditional-NWP systems, and weather centres globally are having to strike new balances without a lot of certainty.
I suggest reading up on fixed costs vs variable costs and why it is generally preferable to push costs to fixed.
Assuming you’re not throwing the whole thing out after one forecast, it is probably better to reduce runtime energy usage even if it means using more for one-time training.
It's more efficient anyway because the inference is what everyone will use for forecasting. Researchers will be using huge amounts of compute to develop better models, but that's also currently the case, and it isn't the majority of weather simulation use.
There's an interesting parallel to Formula One, where there are limits on the computational resources teams can use to design their cars, and where they can use an aerodynamic model that was previously trained to get pretty good outcomes with less compute use in the actual design phase.
I mean that’s cute, but surely you can add up the two parts (single training plus globally distributed inference) and understand that the net efficiency would be an improvement?
These are available on Weatherbell[1] (which requires a subscription) now except for the HGEFS ensemble model which I'm guessing will probably be added later. AIGFS is on tropical tidbits which should be free for some stuff[5]. I believe some of the research on this is mentioned in these two[2][3] videos from NOAA weather partners site. They also talk about some of the other advances in weather model research.
One of the big benefits of both the single run (AIGFS) and ensemble (AIGEFS) models is the speed and (less) computation time required. Weather modeling is hard and these models should be used as complementary to deterministic models as they all have their own strengths and weaknesses. They run at the same 0.25 degree resolution as the ECMWF AIFS models which were introduced earlier this year and have been successful[4].
Edit: Spring 2025 forecasting experiment results is available here[6].
Really exciting to see NOAA finally make some progress on this front, but the AIGFS suite likely won't outperform ECMWF's AIFS suite any time soon. The underlying architecture between AIFS and GraphCast/AIGFS is pretty similar (both GNNs), so there won't likely be a model-level improvement. And most of ECMWF's edge lies in its superior 4DVar data assimilation process. AIGFS is still being initialized on NOAA's hybrid 4DEnVar assimilation process as far as I understand it, which is still not as good as straight up 4DVar unfortunately.
There are multiple efforts and a good number of VC working on AI DA system. DA is fundamentally a hand-crafted optimization process just like NN. I once reimplemented an EnKF in pytorch and it works amazingly fast. But our observations are so dirty and sparse. ECMWF tuned their system so well. NOAA definitely has potential being even better, but no hope any soon future IMHO.
Interestingly, while this model is based on a Google Deepmind AI weather model, it's based on a model from 2023 (GraphCast) rather than the WeatherNext 2 model which has grabbed headlines as of late. I'd imagine it takes a while to integrate and test everything, explaining the gap.
Google Research and Google DeepMind also build their models for Google's own TPU hardware. It's only natural for them, but weather centres can't buy TPUs and can't / don't want to be locked to Google's cloud offerings.
For Gencast ('WeatherNext Gen', I believe), the repository provides instructions and caveats (https://github.com/google-deepmind/graphcast/blob/main/docs/...) for inference on GPU, and it's generally slower and more memory intensive. I imagine that FGN/WeatherNext 2 would also have similar surprises.
Training is also harder. DeepMind has only open-sourced the inference code for its first two models, and getting a working, reasonably-performant training loop written is not trivial. NOAA hasn't retrained its weights from scratch, but the fine-tuning they did re: GFS inputs still requires the full training apparatus.
I am dearly hoping that they are using the current "AI" craze to talk up the machine learning methods they have presumably been using for a decade at this point, and not that they have actually integrated an LLM into a weather model.
Graphcast (the model this is based on) has been validated in weather models for a while[1]. It uses transformers, much like LLMs. Transformers are really impressive at modeling a variety of things and have become very common throughout a lot of ML models, there's no reason to besmirch these methods as "integrating an LLM into a weather model"
A lot of shiny new "AI" features being shipped are language models being placed where they don't belong. It's reasonable to be skeptical here, not just because of the AI label, but especially for the troubled history of neural-network based ML methods for weather prediction.
Even before LLMs got big, a lot of machine learning research being published were models which underperformed SOTA (which was the case for weather modeling for a long time!) or models which are far far larger than they need to be (e.g. this [1] Nature paper using 'deep learning' for aftershock prediction being bested by this [2] Nature paper using one neuron.
Yes, that is not in contention. Not all transformers are LLMs, not all neural networks are transformers, not all machine learning methods are neural networks, not all statistical methods are machine learning.
I'm not saying this is an LLM, margalabargala is not saying this is an LLM. They only said they hoped that they did not integrate an LLM into the weather model, which is a reasonable and informed concern to have.
Sigmar is correctly pointing out that they're using a transformer model, and that transformers are effective for modeling things other than language. (And, implicitly, that this _isn't_ adding a step where they ask ChatGPT to vibe check the forecast.)
The quoted NOAA Administrator, Neil Jacobs, published at least one falsified report during the first Trump administration to save face for Trump after he claimed Hurricane Dorian would hit Alabama.
It's about as stupid as replacing magnetic storage tapes with SSDs or HDDs, or using a commercial messaging app for war communications and adding a journalist to it.
It's about as stupid as using .unwrap() in production software impacting billions, or releasing a buggy and poorly-performing UX overhaul, or deploying a kernel-level antivirus update to every endpoint at once without a rolling release.
But especially, it's about as stupid as putting a language model into a keyboard, or an LLM in place of search results, or an LLM to mediate deals and sales in a storefront, or an LLM in a $700 box that is supported for less than a year.
Sometimes, people make stupid decisions even when they have fancy titles, and we've seen myriad LLMs inserted where they don't belong. Some of these people make intentionally malicious decisions.
It’s not an LLM, but it is genAI. It’s based on the same idea of predict-the-next-thing, but instead of predicting words it predicts the next state of the atmosphere from the current state.
> Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.
FGN (the model that is 'WeatherNext 2'), FourCastNet 3 (NVIDIA's offering), and AIFS-CRPS (the model from ECMWF) have all moved to train on whole ensembles, using a cumulative ranked probability score (CRPS) loss function. Minimizing the CRPS minimizes the integrated square differences of the cumulative density function between the prediction and truth, so it's effectively teaching the model to have uncertainty proportional to its expected error.
GenCast is a more classic diffusion-based model trained on a mean-squared-error-type loss function, much like any of the image diffusion models. Nonetheless it performed well.
The GraphCast paper says "GraphCast is implemented using GNNs" without explaining that the acronym stands for Graph Neural Networks. It contrasts GNNs to the " convolutional neural network (CNN)" and "graph attention network." (GAN?) It doesn't really explain the difference between GAN and a GNN. I think LLMs are GANs. So no, it's not an LLM in a weather model, but it's very similar to an LLM in terms of how it is trained.
I used GAN to mean graph attention network in my comment, which is how the GraphCast paper defines transformers. https://arxiv.org/pdf/2212.12794
I was looking at this part in particular:
> And while
Transformers [48] can also compute arbitrarily long-range computations, they do not scale well with
very large inputs (e.g., the 1 million-plus grid points in GraphCast’s global inputs) because of the
quadratic memory complexity induced by computing all-to-all interactions. Contemporary extensions
of Transformers often sparsify possible interactions to reduce the complexity, which in effect makes
them analogous to GNNs (e.g., graph attention networks [49]).
Which kind of makes a soup of the whole thing and suggests that LLMs/Graph Attention Networks are "extensions to transformers" and not exactly transformers themselves.
Oh yeah, GNN (graph neural network) is the common term, "graph attention network" is pretty confusing because a GAN is a totally different architecture.
(Well, not necessarily architecture. Training method?)
Same. I hope this was written by hardened greybeards who have dedicated their lives to weather prediction and atmospheric modeling, and have "weathered" a few funding cycles.
What does AI refer to here? Presumably weather models have been using all sorts of advanced machine learning for decades now, so what’s AI about this that wasn’t AI previously?
They're using a graph neural network. From the article - "The team leveraged Google DeepMind's GraphCast model as an initial foundation and fine-tuned the model using NOAA's own Global Data Assimilation System analyses".
> so what’s AI about this that wasn’t AI previously?
The weather models used today are physics-based numerical models. The machine learning models from DeepMind, ECMWF, Huawei and others are a big shift from the standard, numerical approach used for the last decades.
Thanks, I guess my assumption that ML was widely used in forecasting is wrong.
So are they essentially training a neural net on a bunch of weather data and getting a black box model that is expensive to train but comparatively cheap to run?
Are there any other benefits? Like is there a reason to believe it could be more accurate than a physics model with some error bars?
Machine learning _has_ been widely used in weather forecasting, but in a different way than these models. Going back to the 1970's, you never just take the output of a numerical weather model and call it a forecast. We know that limitations in the models' resolution and representation of physical processes lead to huge biases and missed details that cause the forecast to disagree with real world observations. So a standard technique has been to post-process model outputs, calibrating them for station observations where available. You don't need super complex ML to really dramatically improve the quality or skill of the forecast in this manner; typically multiple linear regressions with some degree of feature selection and other criteria will capture most of the variance, especially when you pool observation stations together.
> Are there any other benefits? Like is there a reason to believe it could be more accurate than a physics model with some error bars?
Surprisingly, the leading AI-NWP forecasts are more accurate than their traditional counterparts, even at large scales and long lead times (i.e. the 5-day forecast).
The reason for this is not at all obvious, to the point I'd call it an open question in the literature. Large-scale atmospheric dynamics are a well-studied domain, so physics-based models essentially have to be getting "the big stuff" right. It's reasonable to think that AI-NWP models are doing a better job at sub-grid parameterizations and local forcings because those are the 'gaps' in traditional NWP, but going from "improved modelling of turbulence over urban and forest areas" (as a hypothetical example) to "improvements in 10,000 km-scale atmospheric circulation 5 days later" isn't as certain.
Do these ML models replace the numerical approach completely? A lot of numerical methods are iterative. If the ML model can produce a good initial guess, it might make convergence of an iterative process quite a bit quicker…
> Presumably weather models have been using all sorts of advanced machine learning for decades now
This isn't actually true, unless you're considering ML to be just linear regression, in which case we have been using "AI" for >100 years. "Advanced ML" with NN is what's being showcased here.
I know someone pursuing a degree in meteorology at well known university for the subject and I asked that person if they are being taught about these and other AI weather models, about how they work, how to evaluate them for effectiveness, etc.
The answer: AI is not even covered, at least at the undergrad level. This is just a sample of one, so are any other universities educating future meteorologists on this subject?
No I haven’t but I think the lack of liquidity as a chicken and egg is a huge barrier to entry in these markets specifically. They are small right now but there are climate derivatives on the Chicago mercantile exchange so this isn’t a new concept I think.
No, not really. If you are just looking to work with the data you want to read about extracting from grib2 format. One of the faster ways off the ground is to use the Pywgrib2_s python package and iterate against the model files using python to extract the fields that are interesting. I have a container on docker hub that has pywgrib compiled with all its dependencies if you want to tinker.
How well do these predict extremes/outliers? Given that I expect these are more "ML" type models, these are somewhat limited to interpolation, rather than extrapolation?
Apple purports to still have this, but it is indeed less reliable than Dark Sky.
However, the author of Carrot for iOS implemented his own flavor of this* and it's remarkably decent.
* According to Gruber interview around the same time Carrot introduced an entire Broadway musical about the conflict between the Carrot AI and her Maker (the dev). Which, while made with AI, is rather more listenable than the typical weather app.
If you mean minute-accurate forecast for the next 4-6 years... That's called Nowcasting, and yes, it exists. Bing Weather have it, ACCU Weather as well. Rain viewer too. I believe Google already implemented on Pixel Weather at least.
IMO the best of these are Bing Weather and Rain Viewer, both provide rich maps showing where the rain it's going and all too. And how much.
My friend's kids would ride their bikes to school in the morning, and on rainy days, check Dark Sky to find the driest time window. It was usually quite accurate.
These look like staging MVP releases with a full rollout planned for the future. They are only including a few parameters at every 6 hours which is barely interesting to anyone with their feet on the ground.
Where I am the last couple of years, the EU model out performed the US model. The local stations tend to show both when sever weather is on its way to the area.
We know how the current admin views science and with the cuts to NOAA done this year, I expect that trend to continue and widen. At least where I am, we get to see both.
Apparently it seems to be impossible with these files and the best AI right now to answer the simple question, will it rain in midtown Manhattan tomorrow?
how about working with Weather Underground to validate predicted weather at ground level? Here in Southern CO would be a perfect place to try this. Weather Underground has thousands of volunteer backyard weather stations, including mine.
I understand that aviation safety is certainly a primary concern for NWS/NOAA but ground level forecasts are also very important for public safety.
Protip: Any time you read "AI" in a news article, substitute the phrase "faster, more numerous, and confidently incorrect." I don't think we need "confidently incorrect" weather models. Who is asking for this?
These models actually outperform traditional methods on many fronts, including accuracy a lot of the time. They are technically generative AI models, but they're definitely not LLMs.
This is big news. For decades, NOAA’s model has basically just been a huge Fortran physics simulation. Now they are making the leap to AI.
I suspect the nail in the coffin was the hurricane season, where NOAA’s model was basically beat by every major AI model. [0]
The GFS also just had its worst year in predicting hurricane paths since 2005. [1] That’s not a trend you want to continue.
[0] https://arstechnica.com/science/2025/11/googles-new-weather-...
[1] https://www.local10.com/weather/hurricane/2025/11/03/this-hu...
I've seen the Microsoft Aurora team make a compelling argument that weather is an interesting contradiction of the AI-energy-waste narrative. Once deployed at scale, inference with these models is actually a sizable energy/compute improvement over classical simulation and forecasting methods. Of course it is energy intensive to train the model, but the usage itself is more energy efficient.
There's also the efficiency argument from new capability: even a tiny bit better weather forecast is highly economically valuable (and saves a lot of wasted energy) if it means that 1 city doesn't have to evacuate because of an erroneous hurricane forecast, say. But how much would it cost to do that with the rivals? I don't know but I would guess quite a lot.
And one of the biggest ironies of AI scaling is that where scaling succeeds the most in improving efficiency, we realize it the least, because we don't even think of it as an option. An example: a Transformer (or RNN) is not the only way to predict text. We have scaling laws for n-grams and text perplexity (most famously, from Jeff Dean et al at Google back in the 2000s), so you can actually ask the question, 'how much would I have to scale up n-grams to achieve the necessary perplexity for a useful code writer competitive with Claude Code, say?' This is a perfectly reasonable, well-defined question, as high-order n-grams could in theory write code without enough data and big enough lookup tables, and so it can be answered. The answer will look something like 'if we turned the whole earth into computronium, it still wouldn't be remotely enough'. The efficiency ratio is not 10:1 or 100:1 but closer to ∞:1. The efficiency gain is so big no one even thinks of it as an efficiency gain, because you just couldn't do it before using AI! You would have humans do it, or not do it at all.
> even a tiny bit better weather forecast is highly economically valuable (and saves a lot of wasted energy) if it means that 1 city doesn't have to evacuate because of an erroneous hurricane forecast
Here is the NOAA on the improvements:
> 8% better predictions for track, and 10% better predictions for intensity, especially at longer forecast lead times — with overall improvements of four to five days.(1)
I’d love someone to explain what these measurements mean though. Does better track mean 8% narrower angle? Something else? Compared to what baseline?
And am I reading this right that that improvement is measured at the point 4-5 days out from landfall? What’s the typical lead time for calling an evacuation, more or less than four days?
(1)https://www.noaa.gov/news/new-noaa-system-ushers-in-next-gen...
To have a competitive code writer with ngrams you need more than to "scale up the ngrams" you need to have a corpus that includes all possible codes that someone would want to write. And at that point you'd be better off with a lossless full text index like an r-index. But, the lack of any generalizability in this approach, coupled with its markovian features, will make this kind of model extremely brittle. Although, it would be efficient. You just need to somehow compute all possible language before hand. tldr; language models really are reasoning and generalizing over the domain they're trained on.
Now that we’ve saved infinite energy all carbon tax credit markets are unnecessary! Big win for the climate! pollutes
Obviously much simpler Neural Nets, but we did have some models in my domain whose role was to speed up design evaluation.
Eg you want to find a really good design. Designs are fairly easy to generate, but expensive to evaluate and score. Understand we can quickly generate millions of designs but evaluating one can take 100ms-1s. With simulations that are not easy to GPU parallelize. We ended up training models that try to predict said score. They don’t predict things perfectly, but you can be 99% sure that the actual score designs is within a certain distance of said score.
So if normally you want to get the 10 best design out of your 1 million, we can now first have the model predict the best 1000 and you can be reasonably certain your top 10 is a subset of these 1000. So you only need to run your simulation on these 1000.
Heuristical branch-and-bound
It's definitely interesting that some neural nets can reduce compute requirements, but that's certainly not making a dent on the LLM part of the pie.
Sam Altman has made a lot of grandiose claims about how much power he's going to need to scale LLMs, but the evidence seems to suggest the amount of power required to train and operate LLMs is a lot more modest than he would have you believe. (DeepSeek reportedly being trained for just $5M, for example.)
I saw a claim that DeepSeek had piggybacked off of some aspect of training that ChatGPT had done, and so that cost needed to be included when evaluating DeepSeek.
This training part of LLMs is still mostly Greek to me, so if anyone could explain that claim as true or false and the reasons why, I’d appreciate it
I think the claim that DeepSeek was trained for $5M is a little questionable. But OpenAI is trying to raise $100B which is 20,000 times as much as $5M. Though even at $1B I think it's probably not that big a deal for Google or OpenAI. My feeling is they can profit on the prices they are charging for their LLM APIs, and that the dominant compute cost is inference, not training. Though obviously that's only true if you're selling billions of dollars worth of API calls like Google and OpenAI.
OpenAI has had $20B in revenue this year, and it seems likely to me they have spent considerably less than that on compute for training GPT5. Probably not $5M, but quite possibly under $1B.
So LLMs predict the next token. Basically, you train them by taking your training data that's N words long and, for X = 1 to N, and optimizing it to predict token X using tokens 1 to X-1.
There's no reason you couldn't generate training data for a model by getting output from another model. You could even get the probability distribution of output tokens from the source model and train the target model to repeat that probability distribution, instead of a single word. That'd be faster, because instead of it learning to say "Hello!" and "Hi!" from two different examples, one where it says hello and one where it says hi, you'd learn to say both from one example that has a probability distribution of 50% for each output.
Sometimes DeepSeek said it's name is ChatGPT. This could be because they used Q&A pairs from ChatGPT for training or because they scraped conversations other people posted where they were talking to ChatGPT. Or for unknown reasons where the model just decided to respond that way, like mixing up some semantics of wanting to say "I'm an AI" and all the scraped data referring to AI as ChatGPT.
Short of admission or leaks of DeepSeek training data it's hard to tell. Conversely, DeepSeek really went hard into an architecture that is cheap to train, using a lot of weird techniques to optimize their training process for their hardware.
Personally, I think they did. Research shows that a model can be greatly improved with a relatively-small set of high quality Q&A pairs. But I'm not sure the cost evaluation should be influenced that much, because the ChatGPT training price was only paid once, it doesn't have to be repaid for every new model that cribs its answers.
And an LLM can be more energy efficient than a human -- and that's precisely when you should use it.
If its more energy efficient it is doing something different there is no guarantee that its more accurate long term. Weather is horrible difficult to predict and we are only just alright at it. If LLM are guessing at the same rate we are calculating but I am doubtful
Well that was a failed response opps. I am just cautious because while transformers get the random guessing right you can get the right answer statistically but fail on accuracy improvement long term. Clearly this model does better than the current model but extending it to be even better seems basically intractable besides throw more data at it but what if it derived the wrong model you simply cannot actually know
That's precisely when, (insert hand wavy motion), we should use any of this.
This jumped out at me as well - very interesting that it actually reduces necessary compute in this instance
The press statement is full of stuff like this:
"Area for future improvement: developers continue to improve the ensemble’s ability to create a range of forecast outcomes."
Someone else noted the models are fairly simple.
My question is "what happens if you scale up to attain the same levels of accuracy throughout? Will it still be as efficient?"
My reading is that these models work well in other regions but I reserve a certain skepticism because I think it's healthy in science, and also because I think those ultimately in charge have yet to prove reliable judges of anything scientific.
> My question is "what happens if you scale up to attain the same levels of accuracy throughout? Will it still be as efficient?"
I've done some work in this area, and the answer is probably 'more efficient, but not quite as spectacularly efficient.'
In a crude, back-of-the-envelope sense, AI-NWP models run about three orders of magnitude faster than notionally equivalent physics based NWP models. Those three orders of magnitude divide approximately evenly between three factors:
1. AI-NWP models produce much sparser outputs compared to physics-based models. That means fewer variables and levels, but also coarser timesteps. If a model needs to run 10x as often to produce an output every 30m rather than every 6h, that's an order of magnitude right there.
2. AI-NWP models are "GPU native," while physics-based models emphatically aren't. Hypothetically running physics-based models on GPUs would gain most of an order of magnitude back.
3. AI-NWP models have fantastic levels of numerical intensity compared to physics-based NWP models since the former are "matrix-matrix multiplications all the way down." Traditional NWP models perform relatively little work per grid point in comparison, which puts them on the wrong (badly memory-bandwidth limited) side of the roofline plots.
I'd expect a full-throated AI-NWP model to give up most of the gains from #1 (to have dense outputs), and dedicated work on physics-based NWP might close the gap on #2. However, that last point seems much more durable to me.
"it's more efficient if you ignore the part where it's not"
> "it's more efficient if you ignore the part where it's not"
Even when you include training, the payoff period is not that long. Operational NWP is enormously expensive because high-resolution models run under soft real-time deadlines; having today's forecast tomorrow won't do you any good.
The bigger problem is that traditional models have decades of legacy behind them, and getting them to work on GPUs is nontrivial. That means that in a real way, AI model training and inference comes at the expense of traditional-NWP systems, and weather centres globally are having to strike new balances without a lot of certainty.
I suggest reading up on fixed costs vs variable costs and why it is generally preferable to push costs to fixed.
Assuming you’re not throwing the whole thing out after one forecast, it is probably better to reduce runtime energy usage even if it means using more for one-time training.
It's more efficient anyway because the inference is what everyone will use for forecasting. Researchers will be using huge amounts of compute to develop better models, but that's also currently the case, and it isn't the majority of weather simulation use.
There's an interesting parallel to Formula One, where there are limits on the computational resources teams can use to design their cars, and where they can use an aerodynamic model that was previously trained to get pretty good outcomes with less compute use in the actual design phase.
I mean that’s cute, but surely you can add up the two parts (single training plus globally distributed inference) and understand that the net efficiency would be an improvement?
[dead]
These are available on Weatherbell[1] (which requires a subscription) now except for the HGEFS ensemble model which I'm guessing will probably be added later. AIGFS is on tropical tidbits which should be free for some stuff[5]. I believe some of the research on this is mentioned in these two[2][3] videos from NOAA weather partners site. They also talk about some of the other advances in weather model research.
One of the big benefits of both the single run (AIGFS) and ensemble (AIGEFS) models is the speed and (less) computation time required. Weather modeling is hard and these models should be used as complementary to deterministic models as they all have their own strengths and weaknesses. They run at the same 0.25 degree resolution as the ECMWF AIFS models which were introduced earlier this year and have been successful[4].
Edit: Spring 2025 forecasting experiment results is available here[6].
[1] https://www.weatherbell.com/
[2] https://www.youtube.com/watch?v=47HDk2BQMjU
[3] https://www.youtube.com/watch?v=DCQBgU0pPME
[4] https://www.ecmwf.int/en/forecasts/dataset/aifs-machine-lear...
[5] https://www.tropicaltidbits.com/analysis/models/
[6] https://repository.library.noaa.gov/view/noaa/71354/noaa_713...
Really exciting to see NOAA finally make some progress on this front, but the AIGFS suite likely won't outperform ECMWF's AIFS suite any time soon. The underlying architecture between AIFS and GraphCast/AIGFS is pretty similar (both GNNs), so there won't likely be a model-level improvement. And most of ECMWF's edge lies in its superior 4DVar data assimilation process. AIGFS is still being initialized on NOAA's hybrid 4DEnVar assimilation process as far as I understand it, which is still not as good as straight up 4DVar unfortunately.
Came here to say this -- looks like the data assimilation is still done the "old fashioned" way. I wonder how long that will last?
There are multiple efforts and a good number of VC working on AI DA system. DA is fundamentally a hand-crafted optimization process just like NN. I once reimplemented an EnKF in pytorch and it works amazingly fast. But our observations are so dirty and sparse. ECMWF tuned their system so well. NOAA definitely has potential being even better, but no hope any soon future IMHO.
Interestingly, while this model is based on a Google Deepmind AI weather model, it's based on a model from 2023 (GraphCast) rather than the WeatherNext 2 model which has grabbed headlines as of late. I'd imagine it takes a while to integrate and test everything, explaining the gap.
Google Research and Google DeepMind also build their models for Google's own TPU hardware. It's only natural for them, but weather centres can't buy TPUs and can't / don't want to be locked to Google's cloud offerings.
For Gencast ('WeatherNext Gen', I believe), the repository provides instructions and caveats (https://github.com/google-deepmind/graphcast/blob/main/docs/...) for inference on GPU, and it's generally slower and more memory intensive. I imagine that FGN/WeatherNext 2 would also have similar surprises.
Training is also harder. DeepMind has only open-sourced the inference code for its first two models, and getting a working, reasonably-performant training loop written is not trivial. NOAA hasn't retrained its weights from scratch, but the fine-tuning they did re: GFS inputs still requires the full training apparatus.
I've been assuming that, unlike graphcast, they have no intention to make weathernext 2 open source.
That seems to be the case from what I've heard.
I am dearly hoping that they are using the current "AI" craze to talk up the machine learning methods they have presumably been using for a decade at this point, and not that they have actually integrated an LLM into a weather model.
Graphcast (the model this is based on) has been validated in weather models for a while[1]. It uses transformers, much like LLMs. Transformers are really impressive at modeling a variety of things and have become very common throughout a lot of ML models, there's no reason to besmirch these methods as "integrating an LLM into a weather model"
[1] https://github.com/google-deepmind/graphcast
A lot of shiny new "AI" features being shipped are language models being placed where they don't belong. It's reasonable to be skeptical here, not just because of the AI label, but especially for the troubled history of neural-network based ML methods for weather prediction.
Even before LLMs got big, a lot of machine learning research being published were models which underperformed SOTA (which was the case for weather modeling for a long time!) or models which are far far larger than they need to be (e.g. this [1] Nature paper using 'deep learning' for aftershock prediction being bested by this [2] Nature paper using one neuron.
[1] https://www.nature.com/articles/s41586-018-0438-y
[2] https://www.nature.com/articles/s41586-019-1582-8
Not all transformers are LLMs.
Yes, that is not in contention. Not all transformers are LLMs, not all neural networks are transformers, not all machine learning methods are neural networks, not all statistical methods are machine learning.
I'm not saying this is an LLM, margalabargala is not saying this is an LLM. They only said they hoped that they did not integrate an LLM into the weather model, which is a reasonable and informed concern to have.
Sigmar is correctly pointing out that they're using a transformer model, and that transformers are effective for modeling things other than language. (And, implicitly, that this _isn't_ adding a step where they ask ChatGPT to vibe check the forecast.)
“I hope these experts who have worked in the field for years didn’t do something stupid that I imagine a novice would do” is a reasonable concern?
Yes, it is a very reasonable concern.
The quoted NOAA Administrator, Neil Jacobs, published at least one falsified report during the first Trump administration to save face for Trump after he claimed Hurricane Dorian would hit Alabama.
It's about as stupid as replacing magnetic storage tapes with SSDs or HDDs, or using a commercial messaging app for war communications and adding a journalist to it.
It's about as stupid as using .unwrap() in production software impacting billions, or releasing a buggy and poorly-performing UX overhaul, or deploying a kernel-level antivirus update to every endpoint at once without a rolling release.
But especially, it's about as stupid as putting a language model into a keyboard, or an LLM in place of search results, or an LLM to mediate deals and sales in a storefront, or an LLM in a $700 box that is supported for less than a year.
Sometimes, people make stupid decisions even when they have fancy titles, and we've seen myriad LLMs inserted where they don't belong. Some of these people make intentionally malicious decisions.
It’s not an LLM, but it is genAI. It’s based on the same idea of predict-the-next-thing, but instead of predicting words it predicts the next state of the atmosphere from the current state.
It is in fact one of the least generalized forms of "AI" out there. A model focused solely on predicting weather.
"gen" stands for "generative". If you read the GenCast paper they call it a generative AI - IIRC it's an autoregressive GNN plus a diffusion model.
Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.
> Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.
FGN (the model that is 'WeatherNext 2'), FourCastNet 3 (NVIDIA's offering), and AIFS-CRPS (the model from ECMWF) have all moved to train on whole ensembles, using a cumulative ranked probability score (CRPS) loss function. Minimizing the CRPS minimizes the integrated square differences of the cumulative density function between the prediction and truth, so it's effectively teaching the model to have uncertainty proportional to its expected error.
GenCast is a more classic diffusion-based model trained on a mean-squared-error-type loss function, much like any of the image diffusion models. Nonetheless it performed well.
The GraphCast paper says "GraphCast is implemented using GNNs" without explaining that the acronym stands for Graph Neural Networks. It contrasts GNNs to the " convolutional neural network (CNN)" and "graph attention network." (GAN?) It doesn't really explain the difference between GAN and a GNN. I think LLMs are GANs. So no, it's not an LLM in a weather model, but it's very similar to an LLM in terms of how it is trained.
> I think LLMs are GANs.
They aren't, but both of them are transformer models.
nb GAN usually means something else (Generative Adversarial Network).
I used GAN to mean graph attention network in my comment, which is how the GraphCast paper defines transformers. https://arxiv.org/pdf/2212.12794
I was looking at this part in particular:
> And while Transformers [48] can also compute arbitrarily long-range computations, they do not scale well with very large inputs (e.g., the 1 million-plus grid points in GraphCast’s global inputs) because of the quadratic memory complexity induced by computing all-to-all interactions. Contemporary extensions of Transformers often sparsify possible interactions to reduce the complexity, which in effect makes them analogous to GNNs (e.g., graph attention networks [49]).
Which kind of makes a soup of the whole thing and suggests that LLMs/Graph Attention Networks are "extensions to transformers" and not exactly transformers themselves.
Oh yeah, GNN (graph neural network) is the common term, "graph attention network" is pretty confusing because a GAN is a totally different architecture.
(Well, not necessarily architecture. Training method?)
You're absolutely right! That was a category 5. Thanks for pointing that out.
Hopefully they weren’t all forced out this year. The NOAA had massive cuts.
NCAR is being dismantled as we speak.
I suspect the names of those perpetrating this kind of destruction will become synonymous with ignorance and intellectual cowardice.
Same. I hope this was written by hardened greybeards who have dedicated their lives to weather prediction and atmospheric modeling, and have "weathered" a few funding cycles.
inb4 it’s actually an intern maintaining a 3000+ line markdown file
I can see it now
What does AI refer to here? Presumably weather models have been using all sorts of advanced machine learning for decades now, so what’s AI about this that wasn’t AI previously?
They're using a graph neural network. From the article - "The team leveraged Google DeepMind's GraphCast model as an initial foundation and fine-tuned the model using NOAA's own Global Data Assimilation System analyses".
> so what’s AI about this that wasn’t AI previously?
The weather models used today are physics-based numerical models. The machine learning models from DeepMind, ECMWF, Huawei and others are a big shift from the standard, numerical approach used for the last decades.
Thanks, I guess my assumption that ML was widely used in forecasting is wrong.
So are they essentially training a neural net on a bunch of weather data and getting a black box model that is expensive to train but comparatively cheap to run?
Are there any other benefits? Like is there a reason to believe it could be more accurate than a physics model with some error bars?
Machine learning _has_ been widely used in weather forecasting, but in a different way than these models. Going back to the 1970's, you never just take the output of a numerical weather model and call it a forecast. We know that limitations in the models' resolution and representation of physical processes lead to huge biases and missed details that cause the forecast to disagree with real world observations. So a standard technique has been to post-process model outputs, calibrating them for station observations where available. You don't need super complex ML to really dramatically improve the quality or skill of the forecast in this manner; typically multiple linear regressions with some degree of feature selection and other criteria will capture most of the variance, especially when you pool observation stations together.
> Are there any other benefits? Like is there a reason to believe it could be more accurate than a physics model with some error bars?
Surprisingly, the leading AI-NWP forecasts are more accurate than their traditional counterparts, even at large scales and long lead times (i.e. the 5-day forecast).
The reason for this is not at all obvious, to the point I'd call it an open question in the literature. Large-scale atmospheric dynamics are a well-studied domain, so physics-based models essentially have to be getting "the big stuff" right. It's reasonable to think that AI-NWP models are doing a better job at sub-grid parameterizations and local forcings because those are the 'gaps' in traditional NWP, but going from "improved modelling of turbulence over urban and forest areas" (as a hypothetical example) to "improvements in 10,000 km-scale atmospheric circulation 5 days later" isn't as certain.
Do these ML models replace the numerical approach completely? A lot of numerical methods are iterative. If the ML model can produce a good initial guess, it might make convergence of an iterative process quite a bit quicker…
Reading the article could have helped with this.
AI refers to whatever would have been called "Machine Learning" five years ago.
This ambiguity resulted in some very funny drama on Bluesky: https://bsky.app/profile/nws.noaa.gov/post/3ma754dbtuj2t
Holy shit lmao. Like the wokest tumblr crowd focused into a laser for 2025. How do these guys get through life? Exhausting existence.
> Presumably weather models have been using all sorts of advanced machine learning for decades now
This isn't actually true, unless you're considering ML to be just linear regression, in which case we have been using "AI" for >100 years. "Advanced ML" with NN is what's being showcased here.
I know someone pursuing a degree in meteorology at well known university for the subject and I asked that person if they are being taught about these and other AI weather models, about how they work, how to evaluate them for effectiveness, etc.
The answer: AI is not even covered, at least at the undergrad level. This is just a sample of one, so are any other universities educating future meteorologists on this subject?
Are meteorologists even the right people to be training on how to produce and improve better modeling of weather?
Working on AI driven weather predictions to make money on prediction markets. The accuracy of WeatherNext 2 is astounding.
It may be a fools errand but makes for an extremely interesting research project. http://climatesight.app if you’re interested in climate markets.
Have you considered launching your own weather prediction market instead?
Parametric insurance, energy traders, etc could be good markets.
No I haven’t but I think the lack of liquidity as a chicken and egg is a huge barrier to entry in these markets specifically. They are small right now but there are climate derivatives on the Chicago mercantile exchange so this isn’t a new concept I think.
Could you tell me more? https://discord.gg/HPpN42SKQ
Is there a primer for reading these files?
https://www.nco.ncep.noaa.gov/pmb/products/gens/
https://www.emc.ncep.noaa.gov/emc/pages/numerical_forecast_s...
No, not really. If you are just looking to work with the data you want to read about extracting from grib2 format. One of the faster ways off the ground is to use the Pywgrib2_s python package and iterate against the model files using python to extract the fields that are interesting. I have a container on docker hub that has pywgrib compiled with all its dependencies if you want to tinker.
pywgrib https://www.cpc.ncep.noaa.gov/products/people/lxu/cookbook/a... containerized https://hub.docker.com/repository/docker/jmarks213/container...
It's far, far simpler for users to simply use eccodes[1], particularly as implemented in xarray[2].
[1]: https://github.com/ecmwf/eccodes [2]: https://docs.xarray.dev/en/stable/index.html
That is another viable path, no more no less.
How well do these predict extremes/outliers? Given that I expect these are more "ML" type models, these are somewhat limited to interpolation, rather than extrapolation?
All these years later and we still don’t have the minute-accurate forecasts that Dark Sky had before Apple shut it down. Living in the future sucks.
Apple purports to still have this, but it is indeed less reliable than Dark Sky.
However, the author of Carrot for iOS implemented his own flavor of this* and it's remarkably decent.
* According to Gruber interview around the same time Carrot introduced an entire Broadway musical about the conflict between the Carrot AI and her Maker (the dev). Which, while made with AI, is rather more listenable than the typical weather app.
Apple definitely broke something when they incorporated Dark Sky into their weather ecosystem and it isn’t nearly as good in my locality.
If you mean minute-accurate forecast for the next 4-6 years... That's called Nowcasting, and yes, it exists. Bing Weather have it, ACCU Weather as well. Rain viewer too. I believe Google already implemented on Pixel Weather at least.
IMO the best of these are Bing Weather and Rain Viewer, both provide rich maps showing where the rain it's going and all too. And how much.
Did you mean 4-6 hours?
No.
My friend's kids would ride their bikes to school in the morning, and on rainy days, check Dark Sky to find the driest time window. It was usually quite accurate.
These look like staging MVP releases with a full rollout planned for the future. They are only including a few parameters at every 6 hours which is barely interesting to anyone with their feet on the ground.
Does ai mean LLM here or just normal software?
Where I am the last couple of years, the EU model out performed the US model. The local stations tend to show both when sever weather is on its way to the area.
We know how the current admin views science and with the cuts to NOAA done this year, I expect that trend to continue and widen. At least where I am, we get to see both.
I wonder if the new models consider land use change and emissions from aggressive datacenter development and model training...
Apparently it seems to be impossible with these files and the best AI right now to answer the simple question, will it rain in midtown Manhattan tomorrow?
Take an umbrella if you're concerned.
What is possible is to know with near certainty the rough tonnage of water that will fall across a wide area grain region in an upcoming week.
Useful for the reliable production of grain (timing seeding, harvesting, spraying, etc) in the millions of tonnes.
how about working with Weather Underground to validate predicted weather at ground level? Here in Southern CO would be a perfect place to try this. Weather Underground has thousands of volunteer backyard weather stations, including mine.
I understand that aviation safety is certainly a primary concern for NWS/NOAA but ground level forecasts are also very important for public safety.
Whatever it is, it seems like it might be roughly competitive with ECMWF, the state of the art when it comes to global weather models: https://www.epic.noaa.gov/ai/eagle-verification/
A quick search didn't turn up anything about the model's skill or resolution, though I'm sure the data exists.
They run at 0.25 degree resolution (same as ECMWF AIFS models).
[flagged]
Neil Jacobs, Ph.D
This makes me skeptical that it isn’t just politicized Trumpian nonsense.
Protip: Any time you read "AI" in a news article, substitute the phrase "faster, more numerous, and confidently incorrect." I don't think we need "confidently incorrect" weather models. Who is asking for this?
These models actually outperform traditional methods on many fronts, including accuracy a lot of the time. They are technically generative AI models, but they're definitely not LLMs.
If it were LLM I’d agree, but that’s not the case here.