Marketplaces, Portals, Catalogs, and More - Clarifying Your Organizational Approach to API Platforms

I review a lot of API-related writing for research purposes in any given week. I recently came across a piece that argued that exact terminology wasn't important, as long as your API program (and constituent components) achieved a core set of goals related to consumers and producers.

That attitude is precisely the reason why the industry still mostly conflates "Swagger" with "OpenAPI", even though the renaming happened in 2016. Words matter. As comedian, screenwriter, and actor Kumail Nanjiani says:

"To all saying 'they're just words': words are literally what separates us and animals. To downplay their importance is avocado penguins."

[Or, in other words, communication suffers.]

In an API program, we have the nasty tendency to conflate a variety of terms:

• Platforms
• Marketplaces
• Portals
• Catalogs
• Gateways
• API Management

In issue 209, I used the word platform without first clarifying what I meant. In this edition of Net API Notes, I want to share the model I've developed over the years for terms like 'platform'. It helps me organize how I approach problems and define solutions in the stack. Further, the model can be a useful BS detector when a vendor's claims lack conceptual clarity.

Disclaimer

If you have a different definition of these terms or a different model of how they relate, great! How we successfully navigate the world results from the complex interaction of our experiences, skills, preferences, and incentives. Awareness of, and commitment to, conceptual clarity is a laudable thing. A gold star is you!

Sharing this model does not imply other models are wrong; after all, British statistician George E. P. Box said, "all models are wrong, but some are useful". My goal in sharing is to provide a model for those that may not have one or as a catalyst for discussion where there are differences.

The Touchpoints of an API Program, Defined and Related

Platforms

Digitally, a platform is a broad term for something that allows producers and consumers to interact with each other. I'm purposely abstract at this level to accommodate everything from video game consoles to video streaming services. It includes the broadest set of interaction concerns, from defining the core interaction, who is either side of that interaction, and what incentives them. That is both all there is and an aspect so important that it should necessitate involvement at the highest business levels.

A platform facilitates interaction between two or more sides, and the platform steward's responsibility is to make that interaction more efficient. There are social media platforms, like Instagram or LinkedIn. StackOverflow and Quora are knowledge platforms. YouTube and Spotify are media-sharing platforms.

Marketplaces

As we move to subsequent layers, what we discuss gets more specific. A digital marketplace is a digital platform that adds the ability to buy and sell to the core interaction. This monetary transaction includes some form of payment processing and additional considerations like service strategy development, copywriting, seo, accounting, and marketing. Airbnb and Etsy are examples of digital marketplaces. Rapid (formerly known as RapidAPI) is an example of an API marketplace.

Portals

I mentioned that successful marketplaces have numerous components. One of the most important is a portal. A portal is a web-based destination that enables objects of the core interaction to be discovered. On an API platform, an API portal would, predictably, offer APIs available for integration in a secure and (usually) self-service manner.

If there is a marketplace strategy, the portal is the touchpoint by which it is enacted. Those responsible for the portal are concerned with optimizing the user experience (UX) on both the producer and consumer sides (developer experience, or DX, when those audiences are developers).

Catalogs

The portal may have numerous features: standardized documentation, profile management, and performance reporting to facilitate the user experience. However, when people talk about portals, they almost always, inevitably, visualize a specific piece: the catalog. The catalog would be a list of APIs available in an API platform.

A catalog can exist without a portal. I've seen enterprise environments where a spreadsheet with all the API endpoints within a domain sufficed. While a spreadsheet fulfills the needs of an API catalog, it lacks the UX "creature comforts" and features of a portal, the coherent strategy of the marketplace, and thoughtful interaction optimization of platform ownership.

Gateways

Related, but outside my platform/marketplace/portal/catalog breakdown are gateways. Gateways are unique to the API platform stack. An API gateway is software that sits between API requestors and the APIs to fulfill those requests. It performs various tasks that would be tedious or inconsistent if implemented independently, like authentication or quota management. Together with an API portal, an API portal, we have an API management layer.

Our terminology gets bungled, in part, because of how vendors manifest these components. An API management provider, like Apigee, offers many different versions of its API management product. Those installations often include not only a gateway but a portal, as well. When employees refer to "our Apigee installation", they might be referring to the totality of their API management, their gateway, the portal, the business's marketplace ambitions, or the company's platform infrastructure! Conflating things in such a way may be expedient when a single team manages the entire stack of platform concerns. However, with scale comes the separation of concerns, and failing to distinguish between the various terms is a "boundary smell", similar to a "code smell". Without clear language, it is unlikely we will have clear roles and responsibilities.

"So, should we have an 'Apigee Team'? Or do we need a 'Gateway Team' and a 'Portal Team' in addition to a 'Platform Team'?"

Having clear responsibilities is essential. However, problems occur when team identities fixate on the tool rather than the outcomes. As Sam Newman describes in his piece, Don't Call It A Platform:

"There is a more subtle concern with single-issue teams like this though. It speaks to confused motivations, a lack of focus. Their job becomes about managing the tool. Are they ever going to be in a position to think about alternatives? To realise a better way to solve the problems at hand? These are teams where they have only one tool, one hammer - and all problems better be a nail. Or else."

A "portal team" is prone to optimize for their activity - after all, it is how they are measured. A portal team will only ever deliver a portal, even when there may be better options for facilitating the core interaction of the platform. For many API platforms, especially internal ones, the desired outcome should be better/easier/faster/more-secure delivery fulfillment. There may be many ways of doing that. But it is unlikely other opportunities will be explored when their titles hem them into a single output, like "platform team".

Why Does This Matter?

Defining terms may seem arbitrary, even antithetical, to the "real work" of pushing code to production. In previous jobs, I've been "volun-told" to sit in definition efforts; the bizarro spacetime voids where the time spent warps inversely proportional to the amount of business impact.

That said, I also have seen how vital having clearly defined and well-understood terms are for effective collaboration and communication. The higher the level of abstraction, the more effort we have to put into guaranteeing we've established a shared understanding. Without a shared understanding of critical concepts and terminology - in everything from decision-making to documentation - confusion and misunderstandings can arise, leading to errors and delay.

Clear definitions also aid in clarifying roles and responsibilities. When it is a small, tight team responsible for everything API related, the terminology is less important; questions for feature feedback all get routed to the same people. However, as the work grows and different teams emerge to handle the growing responsibilities, language becomes an essential means of establishing boundaries and routing communication to the right people.

Likewise, when we begin conflating team outcomes with team outputs, we don blinders to the larger solution space. Let's organize teams that sel-identify as delivery support and fulfillment and less about "platform engineering".

Milestones

• Twitter, Twitter, Twitter. After ending 3rd party clients in a most undignified fashion, the powers that be first announced all free API access was going away with only a week's notice. At the time of the announcement, pricing wasn't available. When it was, the lowest tier was priced at $100 a month, sending academic researchers, auto-posters, deleters, hobbyist bot makers, and those still using the 'Log in With Your Twitter Account' into a (justified) tizzy. Dave Winer further identified a mismatch between how Twitter thought about "apps" and how they were actually used in practice. Then Twitter announced that there would be a new form of free access "for bots providing 'good' content". And then they had to delay that whole circus another week as the changes introduced were causing service interruptions. Oh, and Musk fired one of the last remaining principal engineers for suggesting Musk's declining view count wasn't caused by a bug. That last one has nothing to do with the API; I'm just rubbernecking this dumpster fire.
• Fern is a new, open-source format for defining "schema-first" REST and RESTish APIs. It joins Cadl, a "language for describing cloud service APIs", as something I'll have to check out.
• IBM acquired StepZen, developers of a GraphQL server.
• Meanwhile, there's speculation that the Open Graph protocol is dying. Reasons include the spam overrunning the Facebook OGP discussion group, the developer mailing list being broken, the Google documentation links to a dead Google+ page, and the GitHub page has been archived. What's going on? (Source 1, Source 2)
• Lastly, Rivers Cuomo, from Weezer, writes his own Python code to call Spotify and Last.fm APIs. He uses the information to generate playlists for their live concerts. His code can be found on GitHub.

Wrapping Up

That ended up being longer than I expected. When I switched up formats after issue 200, I knew more effort would be required. Sometimes that means I can't publish as often as I like. Having folks who sponsor the email means I can put in the time necessary to get it right rather than having to rush something out to meet an advertising commitment. If you're interested in supporting this endeavor for the continued enrichment of your peers, check out my Patron page.

That's all for now. Till next time,

Matthew (@matthew and matthewreinbold.com)