Why APIs (and I) Persist

In a previous role, I sat across from my boss's boss, a VP of software engineering. The purpose of the meeting was to define the strategic initiatives for the following year. I was excited. After a period of rampant API growth, we had a moment to plan some audacious next steps. There were pain points, but I had ideas on how to address them.

The VP listened politely, asking the occasional clarifying question. But as I proceeded through my list, I noticed nothing seemed to be landing. Finally, they stopped me after I outlined the third or fourth API process-related improvement.

"But we've already done API transformation. What's next?"

Whether we acknowledge it or not, software engineering is as beholden to political forces as any other department or function. Deft leaders can recognize the animating energy of a new trend or technical buzzword and convert that into increased headcount, budget, or resources. For this particular VP, the problems that I identified may have existed. However, "doing the same API thing, but more" was no longer currency spendable with their peers.

As a consultant, I've had the opportunity to peek behind the curtain at several other companies. Unfortunately, what I see there is playing out very much like that meeting with the VP. All too many businesses treat their APIs as a project to be done rather than a culture to be grown.

As I write this, we're starting the second half of 2023. That is as good of a time as any to reflect on where we are as API practitioners. After 15 years in the space, I want to explain why we're still only scratching the surface of web or Net API potential. I'll explain how API transformation differs from frequent technology fads and how I would reassure my younger self after meetings with folks like that VP.

Fashion Animates the Industry, For Good and Ill

Stewart Brand described Pace Layer Theory in his 1999 book Clock of the Long Now. An extension of the earlier "shearing layers" architectural concept, the theory describes how different components of a system change at different rates. (For a deeper dive than the summary I present here, I detailed Pace Layering's relationship to APIs in a 2018 talk.)

One might assume that moving fast, whenever and wherever possible, is best. However, the combination of fast and slow moving parts gives a system its resiliency. As Stewart says:

"The fast parts learn. The slow parts remember. The fast parts propose things, the slow parts dispose things. The fast and small instruct the slow and big with accrued innovations and occasional revolutions. At the same time, and we don't respect this as much as we should, the big and the slow control the fast and the small with constraints and with constancy. All the attention is paid to the fast parts. But all the power is in the slow parts.

"Each layer needs to respect each other's pace. If commerce is too dominate in a society, it can jerk governance around harmfully. Or it can reach down and disrupt culture and nature by going too fast while not having the patience to deal with infrastructure problems. The flip side is where the Soviet Union tried to run everything at governance pace (five year plans, for example) and they destroyed other layers."

As a professional whose job is to spot, understand, and explain changes across the technology landscape, I know how easy it is to become disorientated by the constant spin from press releases. There's no end of whitepapers to read or conferences to attend. I have a stack of books that I need to get to… eventually. For some, especially aging engineering leaders who no longer code regularly, keeping up can feel overwhelming. Despite that, they feel an incredible pressure to do so.

Seeing the world through a Pace Layering lens helped me see through that "keeping up" façade. Most of the discussion on a site like Hacker News is ephemeral - technology fashion. Occasionally, some of that will net a nugget of a good idea, which begets a product. Make a product good enough, and it subsequently starts being incorporated into company infrastructure. That is around the time when I start applying my limited time and energy to understand.

However, that doesn't mean fashion is trivial. Whether we're talking about clothes or an architectural approach, fashion animates and excites. It gives a new story to tell, a justification for why things will be different this time. I've watched as the same individual started as a "DevOps Engineer", was later retitled as a "Site Reliability Engineer (SRE)", and now rocks a "Platform Engineer" label on LinkedIn. They do approximately the same job. But the changing title reflects how the organization attempted to capture software's zeitgeist to do the needful.

Silly? At one time I would have agreed. Now, however, I'm more inclined to give the benefit of the doubt; somewhere a savvy leader was trying to do the right thing amidst stiff budgetary competition. And utilizing fashion was one tool at their disposal.

Net APIs Are Not a Fad

The way we create software has changed significantly over the past two decades. To name a few:

• Git (and version control systems in general) greatly impacted developer workflow. While other VCS existed before (my first job out of college used Visual SourceSafe), Git's usage transformed not only code versioning, the entire practice toward code process and promotion.
• Continuous Integration/Continuous Development (CI/CD) started by reducing the batch size between changes and, subsequently, reducing the probability of massive, unexpected failures due to "big bang" releases.
• Mobile Computing (empowered by native apps and responsive web design) met people where they were, requiring new approaches to support the diverse form factors and environments.
• Cloud Computing saw the delay and overhead costs inherent with server provisioning and maintenance, specifically for bursty workloads, and sought to abstract those to specialized operators.
• Containers (and orchestration) addressed the "works on my machine" problem, allowing dependencies to be packaged and deployed together in something more svelte than a VM.
• "Big Data", the precursor to today's AI/ML, only became possible with the ability to store, recall, and analyze massive amounts of data, taking information out of databases and applying additional rigor (and more data) to turn it into insights.

Reviewing that brief list, we see several big ideas that started out fashionable. and maybe only performed at well-known tech powerhouses. Over time, however, these ideas became enshrined into products (commerce) and passed into every company's infrastructure, not just the tech giants. Some of these became so important to a company's business value that they have become centrally governed concepts.

That said, those big ideas of the last two decades would only have had the impact they have had with a larger cultural shift in IT shops both big and small. Mainly, the shift toward a distributed systems architecture made possible by APIs.

• Git's adoption in software development (particularly in Github, GitLab, or Bitbucket manifestations) is due to its ability to integrate with numerous tools and platforms, creating a beneficial ecosystem.
• Similarly, CI/CD is only as prevalent as it is today with the automation that APIs make possible.
• Mobile Computing wouldn't be possible without the ability for apps and sites to communicate with APIs elsewhere.
• Cloud Computing, without programmatic control via APIs, just recreates manual datacenter management (but without the pride of ownership).
• Containers, without their APIs, lose all of their scaling advantages.
• And "Big Data" needs APIs to facilitate integration between different platforms, tools, and models - otherwise we're back to the days of batch scripts FTP'ing CSV files to each other.

Even the most monolithic software produced today is most likely dependent on APIs for compiling, testing, hosting, and deployment. APIs are not JIRA tickets to be done but a modular approach adopted throughout the industry's culture. Success at any upper layer necessitates a high-degree of execution at the API cultural layer.

Has the cloud transformation stalled? Does the mobile experience suck? Speaking from experience, there is a high likelihood that somewhere in those hairballs is subpar API execution.

Advice to A Younger Me

The VP declaring the API transformation done at the time felt like a value judgment on what I felt was necessary. With hindsight, I recognized that exchange was less about me and much more about the kind of project thinking prevalent in organizations. It's the approach that recognizes how trends can be beneficial but lacks the introspection for which things operate at a deeper, more profound level.

"The power of a system comes more from the relationships among programs than from the programs themselves." - Kernighan and Pike, The Unix Programming Environment

APIs rewire the communication paths within a company. As such, they've become a cultural approach to software development, making so much more of these other ideas possible. Companies that have sustained, long-term success are those that treat APIs accordingly.

And as such, any stress I might have had about being irrelevant or superseded by the next big thing was wasted energy. While fashion constantly changes, cultural concerns change on a time scale measured in decades. The APIs created now for fulfilling business needs, internal or external, will still be the dominant paradigm for the foreseeable future. Rest easy tonight; there will be plenty of work for some time to come.

Milestones

• There's a new version of the Postman State of the API Report. The 2023 installment includes responses from more than 40,000 developers. New this time around are insights on API monetization (43% say APIs generate at least a quarter of company revenue) and generative AI (60% say they're using it in their job).
• Istio, an open-source service mesh originally developed by Google and IBM on top of Lyft's Envoy proxy, has "graduated" from the Cloud Native Computing Foundation's incubation status. Bit of non-news news for tech's seasonal slow period.
• From the 'something more useful' department, Adrian Cockcroft has re-uploaded a 300+ "megadeck" of all the microservice and cloud content he used in his 2016 Netflix workshops. It had been previously available on Slideshare, but given how ad-ridden that site has become, I welcome the repost to Speakerdeck. It is also available on Adrian's GitHub account.
• Speakeasy, an early-stage startup, has received a $7.6 million seed investment. When given an Open API description, Speakeasy will "use AI" (I'm reflexively using the finger quote gesture as I edit this) to create SDKs in several languages. As I've said, "It's not AI unless it's from the uncanny valley region of Big Tech. Otherwise, it's just spicy autocomplete." As I mentioned in my most recent talk, we'll see a host of regular 'ole algorithms dressed up in new, investor-fueled finery. Tools like Swagger Codegen have been around since 2016. Proceed accordingly.
• Have you ever thought to yourself, "I simply do not have an enamel pin of my favorite HTTP status code?" Well, your search is over! For your heart's desire, I present the "418 I'm a Teapot" enamel pin. (For the record, I'm not affiliated with the store. It just gave me a good laugh and I hope it does for you too.)

Wrapping Up

Thank you paid subscribers. In addition to working on my book, I've also been surveying other email newsletters; in particular, I'm curious about what kinds of add-ons or benefits their readers find valuable. There's a variety of ways to create a tighter and more rewarding feedback loop. Let me know if you have suggestions, things you would like to see, or something you'd like me to try with Net API Notes.

That's all for now. Till next time,

Matthew (@matthew and matthewreinbold.com)