Openly accessible - and still no walkover

For Sweden I first thought the integration would be very straightforward. SMHI provides the observations openly, without registration or an API key. That is genuinely pleasant: through the MetObs catalogue I can move transparently from parameters to stations, periods and individual files.

Official monthly values exist only for mean temperature and precipitation. That helps, but it is only two values. For maximum, minimum, sunshine, temperature-day counts and thunderstorm days I still have to calculate from daily or hourly data myself - exactly what has already created a lot of extra logic for other countries.

Not every weather station measures the same things. In the station list each station therefore has several identifiers, depending on whether temperature, precipitation or sunshine hours are measured there, for example. I first have to keep these identifiers apart and assign them to the matching data.

For the different measured quantities, data from different stations then have to be combined. That matters, because otherwise it is easy to get the impression that the data are equally well distributed everywhere. In fact the number of stations differs by measured quantity.

The mapper implementation deliberately separates these data by their technical origin. Parameter 22 supplies mean temperature and parameter 23 precipitation directly as official monthly values. From the daily extremes of parameters 20 and 19, temperature maximum, temperature minimum and summer, tropical, frost and ice days are calculated only when every day of a month is present.

The hourly sunshine values from parameter 10 are first summed into complete days and then into months. For thunderstorm days I use parameter 13; I describe the exact rule in the next section.

I like SMHI's quality information. Green values are checked, but yellow values must not be discarded across the board: for monthly values and daily extremes they can mean official aggregated values. The mapper therefore accepts G and Y, but does not treat missing or discarded values as null.

I do not integrate new snow. SMHI provides snow depth, but snow depth is not new snow. A difference between two snow depths would be misleading because of melting or drifting. I prefer an open metric here to an apparently complete but wrong number.

The language was again an extra research step for me. SMHI field names, parameter descriptions and quality notes are in Swedish. Terms such as present weather, representative month or quality first had to be placed securely before I could assign them correctly.

Downloading also brought technical pitfalls. The CSV files start with different header blocks depending on the measured quantity: first come details about the station and the period, and only then the actual values. The time columns are not uniform either. Monthly and daily data use time intervals, while sunshine hours and weather observations use separate date and time columns.

In addition, SMHI reports some CSV files technically as text/plain instead of CSV. The downloader still has to recognise these variants as data files, otherwise existing measurements would wrongly be skipped. That is not a fault at SMHI, but it makes a general import more work.

The two periods corrected-archive and latest-months bring another compromise. The archive holds the quality-controlled history, but ends before the newest months. latest-months closes that gap, but is only provisional. I merge both and always let corrected-archive win on overlaps. New data stay visible without provisional values overwriting final archive values.

The historical bulk download should also only run sequentially. Especially the many hourly sunshine values make the first import slow. That is understandable and fair to the source, but less convenient for processing. In practice I still find Sweden pleasantly fast despite the large volume. For this retrieval SMHI does not impose tight request limits or API-key hurdles. I can therefore load the files without artificial waiting times.

In the climate-data mapper, however, I hit a hard memory limit.
The SE holding comprises several thousand SMHI CSV files; taken together that is on the order of about 50 to 80 million data rows. The hourly and weather archives weigh especially heavily, with individual files already holding hundreds of thousands of rows. As long as the mapper first reads all raw rows completely and only then aggregates them, tens of millions of observations are held in memory at once - and even with 128 GB of RAM that repeatedly caused crashes, so I never finished the full series in one pass.

The solution was a special case for Sweden: a streaming import. Each file is aggregated immediately and then discarded, instead of holding the entire holding at once. The principle may help me later with other countries as well, because hourly values in particular can be huge, and it was first needed for Sweden.

My conclusion is mixed: the open, well documented SMHI source and the official monthly values are a very good foundation. At the same time the separate observing networks, the different formats, partly different headers per metric and the limited thunderstorm observation require careful handling of coverage. Sweden was basically integrable, but smaller hurdles kept appearing that I found unnecessary, so despite those smaller hurdles I still had to adapt my mapper and downloader quite strongly to Sweden.