Net API Notes for 2021/10/06 - Issue 177 - Amazon's Inconsistency Invites Imitation

Occasionally, when compiling the notes, I come across something that warrants a more extended look. This time around, that means taking a moment to talk about Amazon and the effects its cultural decisions have on API design throughout industries.

Net API Notes is a regular, hand-curated digest of impactful news and analysis for busy API practitioners. Are you reading this on the web and not subscribed yet? Sign up today and be the first to get ad-free, actionable info delivered weekly to your inbox.

NOTES

AWS AND THEIR INCONSISTENT API DESIGN INVITES IMITATION

STRAT / DESIGN / DOC / DEV & TEST / DEPLOY / SECURITY / MONITOR / DISCOVERY

Luc van Donkersgoed had an absolutely thorough dunking of Amazon in a piece entitled (deep breath) "How AWS dumps the mental burden of inconsistent APIs on developers". Published at the end of September, this isn't your average quibbling. Luc shows his work.

"What I found was that there’s no consistency between different AWS services, between related AWS services, and even within a single service’s API."

What follows is example after example of puzzling, sometimes infuriating, always consistently frustrating design inconsistencies. From the piece:

"CloudFront has Get, List, and Describe calls. The List operations all return a dictionary with an Items key, which is a list:"

"This is a divergence from every other API, all of which have the resources on the top level. For example, here’s S3 ListBuckets:"

And on and on and on, so on and so forth.

How could one of the most significant API success stories, a company with seemingly all the money in the world, end up in this sorry state of affairs? To understand that, let's go back to the beginning.

THE BEZOS APPROACH TO BUSINESS

The Bezos mandate is one of the most apocryphal, yet retold, stories in all of net API-dom. What we know is based on a Steve Yegge rant. In that retelling, Bezos allegedly sent a note to the entire company sometime around 2002. In that memo, Bezos dictated that all data and functionality could only be accessed via service interfaces going forward; anyone that didn't do it would be fired.

This was a big deal. While APIs are an architecturally obvious way to enable small, autonomous teams to scale today, that was not the case in 2002. It set a precedent of minimal top-down oversight as long as the customer was being served. Other than that, let the best team win.

Bezos further elaborated on the Amazon cultural approach in his 2016 memo to shareholders. In it, Bezos contrasted "Day 1" and "Day 2" companies. While Day 2 companies were doomed to failure, he urged Amazon to hold fast to Day 1 qualities: being customer-obsessed with quick decision-making.

It isn't easy to make quick decisions if information has to travel to the top of the org chart for decisions to, subsequently, flow down. Compared to other organizations, Amazon teams had a large amount of autonomy and clear direction to do whatever was necessary to please their customers.

What could go wrong?

TOO MUCH OF A GOOD THING

As of this writing, AWS has more than 200 products and services. Some of it suffers from feature creep. Some of it overlaps (do you prefer your relational database in Amazon RDS or Amazon Aurora flavors?). And straightforward storage and compute services compete for shelf space alongside talking to satellites in orbit, exploring quantum computing, and selling machine-learning powered race cars. That there are (apparently) 34 different ways of running containers on AWS.

It is more than a dashboard and comprehension problem. The same issue extends to the way customers are billed:

"AWS is clearly a company built out of small teams; Amazon talks about this constantly in the context of 'two-pizza teams, no larger than you could feed with two large pizzas.' ... The two-pizza philosophy clearly leads to what presents as a per-service, if not a per-feature, margin target. I find what this suggests to be relatively abhorrent: AWS is giving each service team margin targets rather than viewing profitability as a cohesive cross-org strategic metric."

The problem is that a cohesive, cross-org metric would require coordination across groups. Coordination impinges on team autonomy, even if a only little, slowing decision-making. What AWS would gain from a better API, UI, or billing cohesion they would lose the shocking speed with which they launch features.

From the same article quoted above:

"While the concept of decentralized service teams has been one of the biggest reasons for AWS’s success since inception, it's created a lot of whitespace–the gap between teams–that continues to get bigger the more AWS grows."

WHAT HAPPENS NEXT

A decade ago, I assumed that companies that repackaged AWS capabilities in slick, simplified packages were not long for this world. I thought that at some point customer growth would slow and Amazon would refocus their considerable resources on assimilating those that had already done the developer experience (DX) grunt work.

Having spent a decent chunk of time studying how corporate cultures work in recent years, I no longer have such presumptions. Inconsistent APIs and cluttered dashboards are a purposeful byproduct of their culture, which values decision-making speed over coordination overhead. Announcing dozens of new products at AWS:Reinvent, some that overlap or behave differently than what came before, is not a bug, it is a cultural feature.

It is an example of autonomy taken to an extreme. While it isn't necessarily the path I'd walk, I can respect Amazon for knowing who they want to be and sticking with it. It certainly works for them.

Or, at least, I didn't have a problem until inconsistent their API design practices went from being just an Amazon thing to being an industry thing. Last week, Cloudflare announced their R2 cloud storage, which competes with Amazon's S3. From Cloudflare's press release last week (emphasis mine):

"Cloudflare R2 Storage includes full S3 API compatibility, working with existing tools and applications as built."

The R2 and S3 APIs are the same to entice customers to make the switch. But, by emulating the design decisions of a specific culture, we risk propagating the downsides, as well. I'll end with a quote from the Luc van Donkersgoed article that started this note:

"Inconsistent APIs cost more time to implement, and they increase the mental load on developers. But they might also reduce the quality of the applications using them. Worst case scenario, this can lead to a bug in production, with serious time and effort required to fix it. And that cost adds up."

Amazon is gonna Amazon. But we need to ask whether we want that design consistency (or lack thereof) replicated across the wider industry.

MILESTONES

• There was a data breach at AI Dungeon, an online game. It appears the cause was a poorly secured GraphQL implementation.
• SmartBear released their 2021 State of Software Quality survey. Findings included that API developers are increasingly responsible for testing duties.
• Hey! Stephen Mizell's book on measuring API complexity is out now. Give it a look!

WRAPPING UP

Last week I presented "We made an API description, now what?" at ASC 2021. The slides and an extended transcript are now posted online. A collection of the mentioned links is available on Twitter.

I also recently became manager of the LinkedIn API and Web Services Professional Group. If you're looking for a like-minded community of API creators, builders, maintainers, give the group a look.

Finally, thank you to my Patreons! Their support helps keep this newsletter is free of advertising, information selling, or paywalls for everyone's benefit. They are fine people, one and all.

Till next time, Matthew

@libel_vox and matthewreinbold.com

While I work at Postman, a 2021 Gartner visionary company, the opinions presented above are mine.