Well structured - but only in small portions

For Spain, I wanted to bring the weather and climate data into Climate Chronicle via AEMET's open data service. At first glance, it looked encouragingly straightforward, because everything was already there: a documented REST API, station identifiers and monthly climate values. Exactly the kind of data offering where I initially assume that a country can be connected comparatively directly.

The actual implementation was then not the problem either. More difficult were access and, above all, the slow pace at which a historical inventory of around 900 weather stations can be built up. A fundamentally well-structured source thus became an export that was poorly suited to historical data, because progress was only possible in small pieces.
I did look for other options to obtain the data, but there appears to be only the API with the monthly values.

Without a registered API key, almost nothing works with AEMET OpenData. I had to sign up for the service, request a personal key and send it with the requests. The API works in two steps: the first response initially contains only a reference to a second URL. Only there do the actual monthly data reside. Technically this is a clean solution, but for a large historical download it practically means two requests per data window. I could not find a bulk download for all stations and years for this series; the data therefore had to be queried station by station and period by period. AEMET OpenData was my actual data source, while the API documentation defined the technical process.

The next limit is built directly into the time range of a request: a single request may cover at most 36 months. For long historical weather series, I therefore had to split the queries into three-year windows, skip empty periods and then merge the individual parts again.

In addition, the station inventory does not provide a start year for each station. Whether a station begins very early or only much later only becomes apparent during the download. Instead of blindly querying window after window from the year 1700 for every station, I therefore first search exponentially for the earliest period in which data actually exist.

AEMET limits downloads to 40 requests per minute and per key.

For ongoing operation, where I then only fetch a few months, this would not be a major problem.
For a one-time historical build with a little over 900 stations, three-year windows, start-year search and the two-step retrieval of reference and actual data, this limit quickly becomes the deciding factor.

With a single key, downloading the complete historical inventory would have stretched over many days. That is why I used 8 API keys in parallel. AEMET OpenData remained limited to the prescribed 40 requests per minute for each individual key.

In total, up to 320 requests per minute were theoretically possible.
That was my pragmatic solution for building the historical inventory once, at a reasonable scale.

I am aware that using several keys can look like a workaround of the rate limit. My aim, however, was not to load the service heavily on a lasting basis or to scrape the data commercially in bulk, but to assemble the historical inventory for Climate Chronicle once, in full. Once this base inventory is in place, only small follow-up downloads are needed for new years.

Apart from the technical limits, language was again a small point of friction for me. Documentation, interface and also many labels inside the API are in Spanish. Of course almost everything can be translated quickly today, but during research each extra translation step is a small interruption in the flow. When you are working through endpoints, parameters and response fields at the same time, this is more noticeable than you first expect.

Still, this is complaining at a high level. I am grateful to AEMET that the weather-station and climate data are provided in such a structured, open form at all. Registration, API key, two-step responses and small time windows make the path a little more cumbersome, but the documentation was usable for my purposes and access could be established comparatively quickly.

My downloader and the mapper, in a sense the two engines behind Climate Chronicle, were "Spanish" within a few hours 🇪🇸 ¡Olé! :D