Costs

How we ran the NAM on AWS: what a run costs, what it ran on, and how to build your own.

What one 84 h run costs

The same model on Graviton costs about 39 % less per run than on x86, on Spot. A run also pays for its disk: a 300 GB gp3 volume is about $0.073 an hour, so about $0.40 more per run.

Pricing: x86 against Graviton, on-demand against Spot

us-west-2, Linux, 32 vCPU nodes. On-demand from the AWS Pricing API; Spot is the minimum to maximum across the four zones over the 24 hours before 2026-11-06 22:30Z. Hours per run = seconds per forecast hour × 90 h of integration (a product run starts 6 h early) plus 15 min of prep and post-processing; per-run cost is that times the price.

• Measured: c8g (200 s) and r7a (189 s). Scaled: m8g is the same Graviton4 chip as c8g, so it uses 200 s. c7g is Graviton3, which ran 39 % slower per forecast hour than Graviton4 in our test, so 278 s. c7a is the same AMD chip as r7a, so 189 s. A scaled row is arithmetic, not a run.
• c7a.8xlarge (64 GB) was killed by the out-of-memory killer at forecast hour 12 with restarts on, so the x86 run used r7a (256 GB). Restarts are off by default in our runner. That is what lets a 64 GB Graviton node do the job (43 GB peak). A c7a row assumes restarts off and has not been run.
• The measured runs above ran at slightly different Spot prices from today’s. Spot prices move by the hour and by zone.

For context: the GPU path

WOOF-NAM, the GPU rewrite of the NAM by Recast Systems, ran on a g7e.2xlarge at $3.36 an hour on-demand. A run took 54–59 minutes, about $3.30 for the run alone in our trial. The GPU Spot quota read zero, so there is no Spot figure. The GPU work is backlogged: WOOF-NAM’s grids are currently free, and its output lacks the snowfall and precipitation-type records (see Strategies we tried). Sources: docs/gpu-path.md, docs/gpu-woof-trial.md.

What a run-it-yourself setup costs per month

Two cases. Occasional: 4 runs a month and 50 GB of downloads. If you ran it daily: twice a day, about 60 runs a month and 500 GB of downloads. The second is arithmetic, not a promise: nothing here runs daily.

Inputs come from the AWS Open Data buckets in us-west-2 (noaa-gfs-bdp-pds, noaa-nam-pds) and cost nothing to read from a node in the same region. Spot capacity is a lottery: a reclaimed run is still billed and has to restart (see below).

What it cost to get here

Getting here cost about $83 of AWS compute and transfer: the build, the Graviton port, the hindcast cases and their reruns, and the GPU trials. Storage is not included. Engineering time is not billed and is not in any figure on this page. It is the larger cost.

How we run it on AWS

A short, reproducible walkthrough. Paths are in the repository.

1. Build the model

The model is NMMB/NEMS, the preprocessor NPS and the post-processor UPP, from the NAM v4.2 source, with ESMF 7.1.0r underneath. The first build used Intel oneAPI on x86. The Graviton port uses gfortran and OpenMPI from apt on stock Ubuntu 24.04 arm64. The build scripts are in build/ (build_all.sh, libs.sh, build_nems.sh, build_nps.sh, build_upp.sh), the source patches in build/patches/, and the story in docs/build-notes.md and docs/graviton-port.md. Thirteen patches; most are ESMF and compiler portability. Apply all of them, including nps-02-nemsinterp-write-sh2o.patch: without it NOAA’s own nemsinterp starts every run with no ice in the soil. Build on an EC2 node of the target architecture, not on a laptop.

2. Bundle it into S3

run/bundle.sh packs the executables, libraries, static tables and run scripts into one tar (9.3 GB for arm64) in the bucket. A runner is a stock Ubuntu AMI that downloads the bundle at boot, so there is no custom AMI to keep up to date.

3. Inputs

The GFS analysis and forecasts come from the AWS Open Data GFS bucket, and the NAM land state seed from the NAM archive bucket. run/recast.sh copies them server-side, runs ungrib, metgrid and nemsinterp, then the forecast and UPP, and run/products.sh writes the grid-218 awphys files and their .idx files.

4. A self-chaining Spot runner

run/launch.sh starts one Spot instance with the bundle in its user data. run/runner.sh runs the case, uploads results and terminates the instance. When it finishes it calls launch.sh --next to start the next job. Jobs sit in a plain text queue in S3. A job is claimed by a conditional put (If-None-Match: *) of a claim object, so two runners cannot take the same line, and the queue file is never rewritten. For occasional runs you can skip the queue and call deploy/04-run-case.sh.

5. Restarts off, and what a reclaim costs

With restarts off, a Spot reclaim loses the run and you start over: about $2.50 on a c8g node. With restarts on, writing a restart needed up to 77 GB of memory on x86, so you want a 128 GB node (m8g) at about $0.30 more per run. In one 16-hour stretch we lost nine runners to reclaims; about $7 went on runs that never finished.

6. Products and UPP

UPP runs alongside the forecast on each history file, and run/products.sh turns each into one grid-218 file in NOAA’s layout (454 records at f24). Post-processing needs about 21.6 GB of memory on top of the forecast. The design is in docs/tm06-design.md.

7. Permissions, and no alarms

One instance role (deploy/07-runner-policy.sh) lets a runner read and write the run bucket, launch the next runner and terminate itself, all scoped to a project tag. The launcher sets terminate-on-shutdown. For occasional runs we run no alarms. If you chain runs unattended, add a billing alarm and a check that no runner has been up for more than about eight hours.

Lessons that save money

• Check Spot placement scores before you pick a node. In us-west-2 the m8g score was 1 of 10 and c8g, c7g and m7g were 3, and we still lost m8g runners (four) and c8g runners (five) in 16 hours. Keep an ordered list of types and zones, fall through, and run the long job on demand if Spot keeps taking it.
• Disk is not free. A 300 GB gp3 volume with extra IOPS cost about $0.40 per run, which the early per-run estimates left out. Size the volume to the run and delete the history files when you are done.
• Restart interval. It must be a multiple of the 25 s time step. 99999 minutes is not, and the model dies at start.
• A terminated Spot request counts against your quota for a minute or two, so a launch straight after a termination can be refused.