The Indie API Forecasting Your Future

It is only appropriate that before we open a discussion about the future we start with a eulogy, of sorts, for the past. For the better part of the 2010s, my go-to weather app was Dark Sky. The clean, design-first, and hyperlocal experience was a joy to use. Using Dark Sky after trying to navigate cluttered sites like Weather.com was an experience akin to discovering Google's homepage after a semester of collegiate research frustration with Alta Vista. 

I wasn't the only one. Dark Sky attracted not only users in droves but also developers. The Dark Sky API was released along with the iOS app in 2012. There were other sources of weather data, but many developers found the same level of intuitiveness and ease evident in the UI extended to the API, as well. 

That design aplomb is probably what caught Apple's attention. It acquired Dark Sky in 2020. Over time, Apple quietly integrated many of its best features into its own weather app, and by 2022, both the iOS and Android versions of Dark Sky were discontinued. The API shutdown followed in 2023, breaking many popular integrations and leaving an ecosystem of orphaned apps and brittle hacks behind. 

RIP Dark Sky. If only we could have forecasted your own demise. 

Enter Pirate Weather: DIY Spirit in Action

However, where there is a will, there is a way. Pirate Weather began as a side project of Alexander Rey, a Civil Engineering graduate from Queen's University, Canada. 

Like many others, Rey had used the Dark Sky API in his own projects. When I asked him about how Pirate Weather began, Rey described it like this:

"The classic home project tale! I had a 'Magic Mirror' setup that pulled from Dark Sky for weather data. I also used and was always in awe of their minute-by-minute rain forecast, and found it to be incredibly accurate. When Apple acquired them, my idea for Pirate Weather started with the thought that it couldn't be that hard to roll my own provider, since the underlying data was all there. This turned out to be a wild underestimation of the challenge of running a weather API, but that's part of the adventure."

Pirate Weather mimics the original Dark Sky API structure, meaning developers could point their apps to a new base URL and, in many cases, resume their integrations. No complex migration, no reinventing the wheel. It's the kind of unsung benefit from a well-considered abstraction. 

"I always figured that there would be people with existing Dark Sky workflows that would benefit from a source that used the same syntax, since it's always a pain to swap over data sources. "

Sourcing the appropriate data was the first priority. While weather information was technically available through the National Oceanic and Atmospheric Administration (NOAA), it badly needed an intermediary. 

"I'm always surprised how hard it is to work with! NOAA is a world leader in numerical weather modelling, and their accuracy over an incredibly wide range of variables is incredible. So when I have to write a bunch of code to extract UV data because it's only published as a rolling averaged, or convert seemingly random integers into forecasted conditions, it still comes as a bit of a surprise."

The next problem to solve was how to store that data.

"The biggest initial challenge (and in my mind, still the crux of the API) was storing data in a way that scaled large enough to handle the amount of data required to produce a forecast, but still allowed very fast reads in a way that played nicely in the cloud. I initially used AWS Lambda and NetCDF, eventually moving to Docker containers and Zarr files, and in the future will switch to Kubernetes and Icechunk."

Then, perhaps counter-intuitively, giving people access easily turns out to be a whole deal, as well:

 "-auth and monitorization was an absolute headache until I stumbled across my (incredible) current provider (https://apiable.io). It's such a different framework than data processing I struggled with every aspect of it, and it was a great lesson about when it's best to outsource something to experts rather than trying to do it myself."

And with additional scale comes new challenges, with Rey having to revisit and reimagine the code along the way: 

"Solutions that handle one request per second can break down at 10, and ones that can handle 10 break down at 100. The amounts of data Pirate Weather has to process (~100 GB) are small enough that it can be done quickly and efficiently, but large enough that it's not easy."

While it has taken Pirate Weather four years to finally check off the last missing Dark Sky API feature, achieving total parity, the results speak for themselves: 80 million monthly requests to over 40,000 subscribers. It powers such varied experiences as MerrySky, Breezy Weather, and Weathergraph. 

Not too shabby for what started as a "home project". 

Data, Democracy, and the Stakes We Ignore

The weather isn't getting any simpler or more stable, which makes projects like Pirate Weather that much more critical. 

Weeks ago, flash floods tore through Texas, killing at least 120 people and causing billions in property damage. "Tornado alley", once exclusively associated with Oklahoma and parts of Kansas and Texas, appears to be shifting eastward from Ohio to Tennessee. And, Canadian wildfires - now an increasingly annual occurrence - show no signs of politely containing their smoke within national borders, despite U.S. representative protests.

Rey understands the importance of having accurate, real-time data:

"I think of environmental data (like weather forecasts, but also monitoring data) the same way that some people think of roads or police- a means to an end that allows higher order things to flourish. So much of the modern world relies on weather (Aviation! Agriculture! Sports! Electricity! I could go on forever), but because of the data and computing demands, it's a problem that can only be effectively solved at scale. Good data is the lifeblood for so many applications, and without access to this sort of information, everything becomes exponentially harder for everyone."

However, providers, such as the National Weather Service and NOAA, face increasing budget cuts and staff layoffs. This is just as their work becomes more essential. And while Pirate Weather enables some incredible experiences, it is built on the data they provide. 

Rey is already seeing problems ahead in the NOAA pipeline:

"I already run into issues trying to download the occasional forecast, and if the infrastructure isn't maintained, that's obviously going to be an issue. Data quality is an insidious problem, since little degradations here and there don't seem to impact the quality of the output, until you take a step back and realize things don't work as well as they used to."

Making weather data harder or more expensive to access isn't just an inconvenience; it's a direct hit on our collective resilience in the face of large-scale, global change. 

Support the Indie Infrastructure That Supports You

Alexander Rey didn't build Pirate Weather for fame or fortune. He built it because there was a gap, and he had the skills to fill it. The result? A public-good weather API that quietly keeps other people's projects running without fanfare, marketing departments, or multi-million dollar product roadmaps.

But even DIY efforts need care. And time. And, ideally, more than $100 a month in GitHub sponsorships. When I asked what would happen if Pirate Weather suddenly had ten times the support it had today, Rey was clear:

"It'd be ten times better! The API is always improving as people contribute bugs, suggestions, and solutions for things that they notice in their corner of the world; so the more community there is the more things advance."

"For Pirate Weather, I'm really excited about adding in some new data sources and getting map data publishing now that we've wrapped up text summaries. There's also some really exciting advances on machine learning forecast models, which have potential to increase accuracy in some situations."

If you're using Pirate Weather, benefiting from something built on top of it, or if you simply appreciate the idea of this kind of open infrastructure continuing to exist, consider supporting the project. Toss in a few bucks. Share it with your team. 

And don't stop there. Take a look at the other scaffolding you're standing on. That open-source library you always npm install and forget about? The tiny tool you piped together for a cron job three years ago and haven't touched since? The quiet elegance of a web app that works exactly as it should?

There's a good chance someone (or a small team) is keeping that thing alive.

Mike Masnick recently wrote the piece, "Stop Begging Billionaires to Fix Software - Build Your Own". In it, he makes the case that we should stop waiting for tech sugar-daddies to ride in on white Teslas and subsidize useful things. 

We are living through a period of instability - an uncomfortable liminal space bridging what was and what will be. In these times, when everything else is unsure, the most radical thing you can do is build something generous. Or support the people who already have.

Because the future isn't just forecasted, it's built… sometimes by one person, in their spare time, for everyone.

Milestones

• Kong releases its AI-powered OpenAPI description builder. You can try it out now for free.
• I greatly enjoyed a recent episode of Developer Voices covering TaxiQL. In this podcast focusing on integration issues, the host and guest discuss what it takes to sidestep taxonomy unification in large orgs by narrowly focusing on the per-field semantic intent. I assure you, that's much more fascinating than it reads on the screen. 
• I wrote about HTMX more than a year ago. Lo and behold, Drupal is now using it.  
• Slack introduced new API rate limits. (Presumably due to AI training data harvester abuse?) Unfortunately, these limits have had some side effects.  
• There are rumors of "significant" Tyk layoffs. 
• Finally, there have been ongoing reports of Tea app data breaches. From what I can glean, the exploit seems to be a standard case of Broken Object Property Level Authorization in the API. In other words, the issue wasn't about authentication (the users were properly authenticated with their API keys). Still, the authorization logic was flawed, allowing an authenticated API request (or requests) to access the entire production database.

Uffda.

Wrapping Up

And that's a wrap on this edition of Net API Notes. Thank you for reading. Your comments, both here and on LinkedIn, Slack, and Mastodon, are appreciated. Like what you see? Tell a friend. And if you're not subscribed, sign up! 

You can also follow Net API Notes on the Fediverse by following @index@netapinotes.com.

Enjoy your August, and I'll catch you again soon in the next edition.

Till then,

Matthew (@matthew in the fediverse and matthewreinbold.com on the web)