How to Solve the API Variant Problem (Part 1)

In a previous, recent edition of Net API Notes, I discussed how "economic headwinds" were impacting API producers and API platform owners at the Fortune 500 companies I consult with. One of the biggest priorities of the moment is increasing the ROI on existing API investment, increasing overall operational efficiency, and ensuring all ingress and egress is adequately secured. API leaders are either starting or re-examining their catalog and portal efforts.

The thing about compiling an API catalog at a large enterprise - even an incomplete one - is that seemingly duplicate or redundant APIs almost immediately appear. My coworkers and I call this the "variant API" problem. Given the current emphasis on cost reduction and efficient operation, reducing these variants might seem like low-hanging fruit; get some people in a room, agree to use a single, "canonical" API, and declare a win at the next division all-hands. 

Unfortunately, few things are that simple. Determining whether to consolidate variant APIs into a single API product is a complicated issue that has many parts. As a result, my approach will require multiple parts. In part one (this edition that you're reading), I'll illustrate the variant API problem with examples. Then, I'll discuss why this phenomenon is so common (and no, it is not due to lazy devs). 

In part two (issue #234, out in March), I'll explain why most attempts to solve the variant problem are too crude to deliver the promised impact. I'll also outline a back-of-napkin approach for determining whether your API ecosystem actually has a variant issue - or whether consolidation is an expensive distraction from deeper, more pressing problems. 

Let's get started. 

What Are API Variants? 

API Variants occur when two or more organizational APIs offer the same, or nearly the same, business capability. Rarely is it the case where two APIs are identical - if that were the case, consolidation would be a breeze (pick one!) and this would be a very short series. Instead, these variants often express mostly the same core information but differ in several crucial aspects. When known only by the API producers and (usually) their point-to-point consumers, they are an annoyance. 

However, when placed in the catalog with their other unearthed variants, the near duplicity goes from annoyance to Achilles' heel. API consumers are burdened with interpreting and navigating the different versions, adapting their code to traverse the variations, and maintaining this hard-fought understanding as the mapped APIs change over time.

The net is that variants increase development time, reduce developer productivity, and increase the odds that mistakes will happen.

There are several ways these APIs may differ:

Content

• Data inclusion or exclusion - Newer versions of the API might include additional fields not present in APIs made earlier. Conversely, older APIs may contain data that was later deemed unnecessary or restricted due to privacy or security concerns. 
• Data format - How data is structured in the response can differ. For example, newer versions might use deeper or more complex nested objects, while older APIs over the same data exhibit a flatter structure.
• Data Granularity - The level of detail provided for the same information might vary. Newer APIs might offer more granular detail (e.g. breaking down addresses into street, city, and postal code), while older offerings have less specific representations (e.g., a combined address field). 

Interaction

• Authentication methods - Different APIs expressing the same capability might employ various methods for authentication, such as basic authentication, OAuth tokens, or API keys. This can confuse developers who are versed (or have the shared libraries) in dealing with one and not the others.
• API term naming - Are "users" and "accounts" the same thing? What about "customers"? Good naming is the single biggest way to boost API comprehension and the hardest things to do in computer science. The inconsistency often exhibited across variant APIs creates an additional comprehension tax on developers.
• Response format - The response data format can differ between similar APIs - I'm not just talking about JSON vs XML. Consider the case where one API returns images as a Base64 encoded string within a body of JSON while another API returns a hyperlink to the image's CDN location.

Support

• Versioning scheme - While some APIs might use a versioning scheme that manifests itself in the endpoint URL, others might implement header versioning. If header versioning is implemented, what is the assumed behavior when the caller provides no specifics? Is the latest, greatest of the API returned or is the oldest supported version of the API? If developers are left to infer what happens based on their work with other APIs, they may be in for an unpleasant surprise. 
• Deprecation communication - Unfortunately, internal API programs often leave deprecation and sunsetting communication solely up to the teams rather than boilerplate essential info into repeatable templates, easily executable by the platform. Leaving it up to the teams to discover means APIs with nearly identical functionality may still have vastly different comm strategies.
• Intended use cases - Different versions might be intended for specific use cases. Still, this information might not be readily available, leading to developers to choose the wrong version for their needs. This could be as simple as an API only being budgeted to support a team's internal TPS requirements, not the hundreds of thousands of requests needed in an enterprise integration.

A Simple Variant Example

I think I've yet to work with a company that didn't have some form of a "users" API. But even with this seemingly straightforward, simple concept, we can run smack into terminology and structural differences. 

User API #1

User API #2

To save you a diff, the discrepancies include:

• Terminology: API 1 uses /users/{userId} while API 2 uses /profile/{userId}.
• Interaction: API 1 has an optional includeEmail parameter, while API 2 has an optional fields parameter that accepts a comma-separated list of desired attributes.
• Data granularity: API 1 returns separate firstName and lastName fields, while API 2 combines them into a single fullName field. 

NOTE: Neither OpenAPI snippet is objectively "wrong". Both (in isolation) are perfectly fine implementations of a developer's interface preferences. The variant problem arises when a consuming audience has to compare and contrast between them. 

Why Do API Variants Happen? 

If variants create such obvious problems, why do we find them everywhere? A bad management take would be that variants' omnipresence is due to "lazy" developers or that "it's just a training issue". While that may be an easy excuse, the truth is due to more significant, systematic issues, like:

• Legacy systems - Companies don't just happen overnight. Each is a complex, nested amalgamation of changing markets, expertise, and previous software fads. These systems provide business value, but their interfaces reflect the priorities at the time they were built - priorities that may be different from today's.
• Acquisitions and mergers - Similar to legacy systems, combining two different organizations often means reconciling different competing interface norms.
• Third-party integrations  - Companies might proxy third-party services for internal usage, providing an abstraction for user logging and access management. These third parties may manifest their interfaces differently than the "house style", causing some teams to create additional wrappers for convenience or necessity. 

The final, and perhaps biggest, reason variants happen is due to the degree to which modern software development practices prioritize team independence. In the pursuit of minimizing coordination costs between teams, organizations have created fertile ground for redundant, duplicate efforts to take root. 

When growth is paramount and investment capital is readily available, it is easy for organizations to look the other way. However, API leaders seek to weed that garden in times like these. I'll talk about why these efforts often fail, along with a back-of-napkin approach to being smarter in approaching the problem in the next issue. 

Milestones

• P0, an AI-based security startup, secured $6.5 million in seed funding. P0 secures APIs by using Large Language Models (LLMs).
• API security startup Wib was acquired by F5 for "tens of millions of dollars". 
• Google Settles a 2015 Google+ API Data Leak Lawsuit for $350 million.
• Postman's Head of Enterprise Growth goes viral for a cringeworthy, ragebait post, the stupidity of which, given the subsequent follow-ups, appears to have been the attention-getting, growth-hacking point. I bemoan LinkedIn's prioritization of toxic business-bro hustle culture over authentic relationships and meaningful professional events because of vapid stunts like this. 

Wrapping Up

Due to Substack concerns in December, I resolved to move Net API Notes to a new newsletter platform. I completed the work to port things to Ghost managed hosting in mid-January. Now, with a month's worth of data to review, the impact is sobering. 

On Substack I would reliably get at least one or two new signups a day, it has taken me the entire month to match that total on Ghost. If you combine that with the natural churn (people signing up with their work email and then subsequently leaving those jobs, for example), I've started 2024 down more than 4%. I'll be hard-pressed to match last year's 50% growth.

On the one hand, the value generated by Substack's various goading has always been debated. Opening the notification email to discover your newest signup is also subscribed to a thousand other publications gave me mixed emotions. I was happy to have them but lacked confidence that what I was sharing was actually being seen. On the other hand, it is a shock to return to the "traditional" signup method (that is, to inspire so much organic traffic that a token person commits to filling out a form with their contact information). Nazis aside, the way Substack cross-promotes one publication's audience to another is an impressive leveraging of network effects for communal benefit. 

I really, really hope some bright, enterprising product manager within Ghost's managed offering is thinking about addressing these system dynamics.  I'm happy to have a conversation to talk through these issues anytime.

Oh well. This is my high horse, and I'm committed to riding it for the foreseeable future, even if most would say it's moving backward. I was never under the illusion that publishing would pay the bills. But a steady singup drip was enough validation to power through most weeks. Going forward, I need to recalibrate my expectations. 

If you'd like to become a paying subscriber and support this work, check out the subscription page. A monthly or yearly pledge helps Net API Notes remain ad-free and ensures previous commentary, analysis, and insights stay available for all. 

That's all for now. Till next time,

Matthew (@matthew in the fediverse and matthewreinbold.com on the web)