Best Practices for Storing & Accessing Large Weather Datasets?

[xosotat358]

Hi Windy community! 🌪️

I work with large historical weather datasets (WRF outputs, ECMWF archives, etc.) and currently rely on 6TB SATA 6Gbps HDDs (5.9K RPM, 3.5") for local storage. While these drives balance cost and capacity well, I’m curious:

For those handling multi-TB weather models:

Do you prefer high-capacity HDDs (like 6TB SATA) or faster solutions (SSDs/NVMe)?

Any tips to optimize read/write speeds when accessing sequential data (e.g., time-series forecasts)?

Cloud vs. Local Storage:

How do you structure datasets—raw files on HDDs + hot data in cloud, or another approach?

Any experience with RAID setups for redundancy with SATA HDDs?

Windy Integration:

For users who export Windy data locally, what’s your storage workflow?

Would a dedicated Windy API for bulk downloads (to HDD arrays) be useful?

Example Use Case:
I archive 5TB/year of GFS data on 6TB HDDs, but querying years of records can be sluggish. Seeking smarter solutions!

Storage Specs for Reference:

HDD: SATA III 6Gbps | 6TB | 5.9K RPM | 3.5"

Use: Sequential reads (~150MB/s sustained) | 24/7 operation

Thanks for sharing your setups! 🖥️💾