API Auth: From No-Auth to Google's Gamut

It's Halloween! After several recent integrations, I reflect on the things that truly scare me in the API space. No, it's not shadow IT or micromanaging goblins. What truly sends a shiver up my spine are unexpected changes in API authentication.

I don't actively seek out new APIs to tinker with for giggles. My weekends are no longer multi-hour tinker time. When I get the opportunity, I try to get something working, confirm the automation is firing, and move on. But every so often, I get those dreaded emails like the one from AirTable this spring, the ones that say, "We're deprecating API keys".

AirTable isn't alone. From HubSpot to Asana, from JFrog to 7Shifts, it's a pattern I've seen repeatedly. Companies start with simple authentication in pursuit of growth, but, as they grow, so does the security complexity necessary.

In this edition of Net API Notes, I will summarize several ways that organizations attempt to secure their API access, discuss how organizations outgrow their initial, rudimentary efforts, and the kind of complexity necessary in a company with the size and scope of Google.

Changing Needs, Changing API Security Strategy

API security isn't a "1" or a "0". While some might think it's either OAuth 2.0 or bust, the variety of actual implementations is far more nuanced. For simplicity, I tend to think not only in terms of the mechanisms at play but the degree of interplay they have with the customer's account management.

Level 0: No Authentication

An API without authentication is one that anyone can call. These APIs are great for quick smoke tests of new tools or product demos. The Star Wars API (SWAPI) is an example of an API that does not require authentication.

While these APIs can be fun, they are usually little more than toys. An API provider might attempt some rudimentary rate limiting based on the caller's IP address to avoid malicious actors and DDoS attacks. However, this can be easily foiled.

There may have been a time, even as recently as a decade ago, when teams building APIs for internal consumption might have sought to avoid the authentication overhead. However, given the modern realities of privilege escalation and many CISOss "zero trust" responses, even the wide-open internal service is (mostly) a thing of the past.

Most APIs need some level of authorization to:

• Prevent unauthorized access to sensitive or private data
• Ensure that only authorized users can create, modify, or delete data
• Rate limit the number of requests a user or system can make to prevent abuse or overuse
• Facilitate charging users based on their API usage
• Maintain a log of who accessed what data and when, aiding in accountability and debugging should things go wrong
• Provide personalized content or services based on the authenticated user's profile or preferences

That leads us to many API provider's first attempts to secure their API:

Level 1: Basic Authentication

Enter the world of usernames and passwords. If it works for account management on the web and in our native apps, it should work for API access as well, right?

Well, not quite. While the basic authentication officially defined in rfc7235 is a step up from nothing, it has some pretty egregious problems.

For starters, using the username and password for API access conflates account management and authentication. If, for whatever reason, someone forgets their login information and needs to reset their password, we now have to update and synchronize a code push to update the API's consuming code. No bueno, even if you do have an optimized CI/CD pipeline.

Further, the username and password information is only encoded, not encrypted, in this scheme. The movement to all-HTTPS, all the time does help obscure sensitive account credentials. But it would be akin to having to recite your social security number to unlock your car - it is information that can be used in many places besides opening your car door, and you'd be sharing it with who knows who all the time.

Because of those pitfalls, basic authentication is not recommended.

Level 2: Bearer Authentication via API Keys

API keys are the next logical step. They're separate from account usernames and passwords; resetting your account password does not impact the generated keys, and vice versa. Further, multiple team members can have their own account access, with their secure practices and churn rates, while maintaining a single means of access between accounts.

Many, many, many small-to-midsize companies maintain an API key approach to protecting access to their API. However, there are cases where greater security may be needed. That's where tokens come in.

Level 3: Token-Based Authentication

On the surface, tokens and keys are very similar. Visually, they both resemble long strings of random characters. They're both usually appended in an authorization header including the 'Bearer' keyword. However, one of the most significant differences between keys and tokens - as Graham was kind enough to point out to me in a recent chat - is their duration. Tokens are short-lived and can be refreshed automatically, making them more secure - it is difficult to steal a key and perform a malicious act when the key is constantly changing.

Of course, if we are manually updating tokens in a similar fashion to keys, it wouldn't happen. What schemes like OAuth 2 do is define an automated process for renewing a token. The process is akin to:

• A client makes a request to a resource server using a token.
• The API provider recognizes the token is either expired or invalid and instructs the client to renew their token.
• The client sees the token renewal requests and submits some alternative, longer-lived credentials, such as a consumer key and a client secret. In other cases, a unique refresh token used only for these purposes is provided.
• Regardless of what is shown as proof, the API returns a new access token if it likes what it sees.
• The client then recalls the same resource as before but with the new token.

The developer calling the API may not even realize this "dance" is happening behind the scenes; I've long maintained that the #1 reason that API SDKs and libraries exist is to abstract away the exact details of the provider's OAuth 2 specific renewal implementation.

This process is a good thing; the API provider can trigger credential renewal on a schedule or when an access token is suspected to have been compromised. And if implemented correctly, the API caller can accommodate this renewal without deploying new code.

But we're not done yet.

Level 4: Token-Based Authorization

Thus far, we've only talked about authentication protection for our APIs; in other words, when a request comes bearing something, whether a key or a token, an API provider checks that it is valid and, if so, allows it to go about its business. This process doesn't ask whether the caller is authorized to do what they're asking to do.

In lower-level schemes I've described thus far, authorization does happen - it is just that all authorization becomes custom code. By taking our previous, short-lived token from level 3 and imbuing it with claims, the API receiving the request can decide whether to allow the request to proceed, given the provided authorization logic.

Schemes like OpenID define several standard scopes that can be used to generate identity arguments. Creating custom scopes for an API is possible. Laying in claims via a signed piece of data like a JWT is also possible (and worthy of its own dedicated article).

This level does necessitate a higher degree of complexity. It is also something that individual API developer teams should refrain from implementing in a project-by-project or siloed manner. When we start talking scopes and claims in provided tokens, API providers will need some centralized coordination mechanism to ensure the work is done consistently - something that makes sense for larger companies, but may be overkill for those small and midsized firms I alluded to before.

Level X, where 'X' stands for eXtreme: A Google APIs example

It could be argued that there are several additional, intermediate API securing approaches that could be layered in, from the introduction of formal Identity and access management (IAM) to administrative workflows for enabling API products. To illustrate what lies at the end of that and the complexity that it entails, it is worth a brief discussion of what it is like to get started with an API at Google.

Google has a multifaceted approach to not only API security but cloud account access. And it has to, given the diversity of a client base that ranges from individual randos (Hi!) to sprawling enterprises.

Assume, for example, you want to do something programmatically with YouTube, a Google product. You don't start by going to the site or service you wish to use (YouTube, in this case). Instead, you go developers.google.com. You start with creating (or logging in) via a relationship with the overarching Google entity, not the specific service you initially sought.

It is an initial surprise that contradicts many developers' initial assumptions. But centralizing the initial interaction allows Google to manage and monitor an account's standing at a macro level.

The downside for the potential API caller is that, even once logged in, there are still several more steps. You'll find yourself setting up a Google Cloud project that encapsulates all the configuration options from a different project, say with Google Ads or Waze. With everything under the same macro roof, it is necessary to create rooms to encapsulate all the stuff more neatly.

Next, you'll enable the APIs you want associated with the project. Depending on what those APIs are and the security they implement, you may need to define OAuth consent screens, generate OAuth 2.0 credentials, and configure the Identity and Access Management (IAM) system. This granular permissions approach is incredibly powerful - and an end-state concern that is difficult to configure correctly before we've even had a chance to make our first API call.

And maybe that's all manageable if I lived within Google's ecosystem - everything would be under the same umbrella and work the same way. However, the unfortunate reality is that I can go to a different Google product's API, like Fitbit or Google's LLM, PaLM 2, and have a completely different API access experience. (For an example of using the PaLM API, check out Raymond Camden's recent blog post.) These somewhat independently operated silos add to the overall complexity of working with Google's APIs.

In essence, Google's approach to API security reflects its vast scale, numerous products, and diverse user base. While the intricate setup and configurations ensure robust security and flexibility, they also present a steep learning curve for API callers. It's a delicate balance between providing advanced functionalities for large enterprises and ensuring accessibility for individual developers. And as with any tightrope act, it's not without its precarious moments.

In Conclusion

API security is not an "on" or "off" proposition. As customer needs grow and evolve, so does the security sophistication. Whether you're an API developer, product manager, or architect, staying aware of consumers' composition is vital.

And if you're one of those API consumers that receives a "we're deprecating your current form of access" email? Well, congratulations! Your dependency is growing up!

Milestones

• Tesla, after years of its car APIs being unofficially co-opted for a variety of enthusiast usages, is now launching its developer API. As Erik Wilde points out on LinkedIn, one must sign up for a developer account before being able to read anything.
• ActivityPub (and Mastodon) have been getting a lotta love recently. But what about other networks, like BlueSky? Well, it turns out that anyone's posts are available to anyone through the unauthenticated API. You don't even need a BlueSky account! So there's that.
• In issue #223, I linked to a report that Kong had pulled local storage during a recent "update". After community outcry, they've apparently reversed course and have "brought back 100% local storage with no cloud synchronization". More context is available on Kong's GitHub.
• Optic is taking Beta signups for LintGPT, an OpenAPI linter that lets one write spectral rules in natural language. This one intrigues me quite a bit.

Wrapping Up

A big thank you goes out to paying subscribers. Not only do they ensure that Net API Notes editions remain free of paywalls and advertising, but they also fund larger and more exploratory projects. One of those things has been the API job listings database. With several thousand records collected this past quarter, I've attempted to form an analysis approach to tease insights out of the sheer amount of data collected. In the past week, that has meant passing the rather wordy descriptions through an LLM to summarize and infer important attributes we'd want to know about.

One of those things is the nature of the work situation: whether the positions are in office, remote, or some hybrid combination. The most straightforward way would be to loop over the database, line by line, looking for those exact keywords. The next level of sophistication would be to search not only for those keywords but also all known synonyms ("in-person", "flexible", "work anywhere"). Soon after, we might write regular expressions to capture phrasing that implies a type of work situation ("[work|be]+ \d days [a week]*[from|in]+ the office"). At this point, the brittleness of detection can quickly spiral out of control.

Applying an LLM to infer the work arrangement is an excellent alternative that not only saves us some regex, but leads to results like this:

I'd like to continue to build out both the data and the analysis through the rest of this year and begin publishing deeper dives in 2024 - a long term effort possible because of this newsletter's paying subscribers.

Paying subscribers get a customized audio podcast of these notes. They also receive ¡APIcryphal!, a new Net API Notes sub-series recapping the (mostly) true stories from API's extensive history. The first issue, on the Bezos' API Mandate Memo, is now available. In keeping with this edition's theme, November's API apocryphal story is OAuth 2.0, Eran Hammer, and "The Road to Hell". It has been so much fun to research and write thus far.

To become a paid subscriber, head to the subscription page. For more info about what benefits paid sponsorship includes, check out this newsletter's 'About' page.

Till next time,

Matthew (@matthew in the fediverse and matthewreinbold.com on the web)