AI Agents Don't Break Your APIs - They Just Stop Compensating for Your Docs

This past June, a paper came to my attention that I immediately shared with my peers. It's titled "Making OpenAPI Documentation Agent-Ready: Detecting Documentation and REST Smells with a Multi-Agent LLM System".

Its most important lesson from authors Lima, Pinheiro, and Menezes at Brazil's Sidia Institute of Technology is not about AI agents, MCP, or the future of software integration. Rather, the paper speaks volumes about the sorry state of our documented API present. 

The most consequential insight? AI agents reveal how much existing API consumption has always depended on contextual knowledge that never made it into the API contract. APIs that appear stable, mature, and perfectly usable may only work because the people consuming them compensate for everything the docs fail to say.

That is a lot to unpack. So let me unwind how a new type of API consumer is exposing a problem that programmers have spent decades learning to work around. 

AI agents can fail to consume even otherwise stable, mature APIs

The API environment described in the paper is relatable: 16 production APIs containing roughly 600 endpoints. These services were actively used. They had OpenAPI descriptions which were structurally valid. When MCP appeared, their engineers assumed it would be a simple matter of converting these existing API descriptions into agent-addressable tools. 

The paper's authors chose to conduct an experiment. They created two MCP servers, each exposing the same ten endpoints. The difference was that one server used the existing OpenAPI description. The other MCP server utilized API descriptions that had been "enriched";  descriptions specifically modified to elaborate the endpoints' purpose, inputs, security requirements, output structure, and operational behavior. 

With the servers configured, they asked agents to perform a series of tasks, most of which required utilizing multiple services. 

What they found was striking:

• With the original documentation, about 70% of the tasks failed during planning.
• Only 10% of the expected endpoints were correctly identified. 
• None completed successfully end-to-end. 
• With the enriched documentation, valid plans were produced for 90% of the tasks, 80% of the expected endpoints were selected, and every selected endpoint executed successfully.

There are qualifiers aplenty here: this was a proof of concept, the tasks were run only once, and there was no control for randomness, making those numbers more illustrative, not statistically sound. 

However, the exercise still said something profound:

The APIs had not changed. The business logic wasn't different. These were the same endpoints, mechanically. The difference in success was driven solely by the nature of the contract presented to the consumer.

So more words more gooder?

Richer API documentation produced significantly better outcomes. So why were the API descriptions deficient in the first place?  

I don't have access to the OpenAPI descriptions used in the paper's analysis. However, based on decades of looking at API descriptions, I suspect what was published was adequate for consumption by people already familiar with the environment. What I mean by that is that the people creating these descriptions made a tradeoff (conscious or not) about the LOE they'd expend. If context was assumed to be "known", it was probably left out for the sake of expediency. This context may include:

• Internal terminology
• Local conventions
• Source code references
• Experience with previous implementations
• Peers who could answer questions
• Recognition of undocumented constraints
• Educated guesses when interpreting generic responses

The paper documents what many of us recognize in our own operating environments: the OpenAPI documentation is only one component in a much larger, more complex operating culture.

One specific example from the paper: several endpoints returned a generic structure containing status and data, but the data field could contain arbitrarily shaped JSON. In some cases, even the original API authors could not confidently explain what a response might contain without reopening the source code. 

If I had a nickel for every time that "polymorphic responses" in returned payloads have caused problems, well… I'd still be doing this. Inflation sucks. However, the point here is that if the meaning of a response can only be recovered by inspecting the implementation, then the contract was never truly self-describing. People are going to have a hard time with that, never mind agents.

But while both groups will struggle, our fashionable meat sacks have become adept at piecing together solutions from incomplete information. We recognize an acronym because we've heard it in a meeting. For example, we know a field labeled id means a project identifier rather than a user identifier. We remember that a parameter that is tagged as optional is actually required for a particular task. And we intuit that a "successful" response is only an intermediate step in a larger workflow.

The OpenAPI description authors don't expend the additional effort to codify those things because, thus far, those gaps have been bridged by tenured people, professional relationships, and institutional memory. And, failing that, someone schedules a meeting.

An agent has no inherent access to this context. Worse, they expose the deficiency because they fail faster and more conspicuously. But while they put this problem in sharp relief, they weren't the ones who created the problem.

The problem is systemic

With the experiment results in hand, the team looked to expand their analysis. However, manually studying approximately 600 endpoints posed a logistical challenge. What the team needed was a way to automate the identification and suggested remediation of the remaining API surface area.

Their solution was to build Hermes, a custom multi-agent LLM system designed to detect recurring documentation and REST-related "smells". Each agent was prompted to examine an endpoint against a different set of criteria. Upon completion, each agent's findings were consolidated into a report cataloging suspected problems and recommending improvements. 

When Hermes was run against their production APIs, it identified 2,450 smells across the 600 operations - or an average of about 4 findings per endpoint. Every endpoint had at least one potential problem:

• 100% of the portfolio had response-related deficiencies
• 90% suffered from sparse or generic documentation
• 88% had poorly explained inputs
• 68% exhibited unclear security information
• 53% had questionable path or method choices

I should pause here and emphasize that the word smell here does not automatically mean something is broken. Rather, it suggests that something needs another look - a closer inspection. A strangely modeled endpoint, for example, is not necessarily unusable. Rather, it may reflect legitimate constraints, inherited decisions, or tradeoffs that made sense at the time of creation. 

The smells that Hermes identified did not prevent the environment from functioning. 

However, functioning is not the same as being self-describing. 

The APIs often provided technically valid response schemas, but the accompanying description offered little more than "Successful Response". A consumer might recognize the outer structure of the returned JSON but still have to guess how the inner response should be interpreted under different conditions.

The same problem appeared in request inputs: a parameter might be declared as a string, but fails to mention whether that value should be a UUID, an internal account number, an email address, a date in a proprietary format, or one of five undocumented status codes.

Security definitions followed a similar pattern. An OpenAPI document might correctly declare bearer authentication while saying nothing about how credentials were obtained, which permissions were required, or whether access to the operation depended on a particular organizational role.

These are not exotic requirements invented for AI agents. They are the kinds of things a new developer needs to know as well. The difference is that a person can compensate. An agent encounters type:string and has to make it up on the spot.

Reviewing these findings with the API producing teams revealed further insights. Developers largely agreed with findings tied to explicit, verifiable omissions such as unclear parameters, responses, and security requirements. However, they also tended to dispute complaints about short descriptions or nonstandard REST design. Their justification was that the existing documentation was sufficient for their existing environment or that they were beholden to legacy constraints.

Which is understandable. They weren't being lazy; they were just making rational decisions while only considering their local context.

And that was also precisely what prevented those APIs from becoming reliably accessible to agents. 

The solution is not to rewrite everything

Based on these findings, the authors' organization now had a dilemma:

If all 600 endpoints had missing context, did the documentation for all 600 endpoints need to be rewritten to make them agent-ready? 

Thankfully, the answer is no. 

To do so would represent an incredible investment - something that teams have limited amounts of. Instead, the team narrowly focused on the specific tasks they hoped the agents would be able to do.

The answer is not to launch a new, portfolio-wide mandate for teams to stuff paragraphs of text into every field (the "fix it in documentation" strategy). Nor was it to wrap all 600 endpoints with MCP simply because the organization technically could (the "kitchen sink" strategy).

Instead, the organization featured in the paper returned to narrowly define the tasks where an agent would provide value. In doing so, they identified 18 automation scenarios that made sense. These scenarios depended on only 42 of the available endpoints. Rather than remediating their entire API landscape, the team focused on making a smaller set of operations sufficiently clear, predictable, and fully self-descriptive. 

The difference is substantial. The authors estimated revising 600 endpoints would require approximately 385 engineering hours. Addressing the 42 endpoints for the identified scenarios was estimated to only take 42 hours - a much more defensible amount.

And that brings me to my last point: an API portfolio doesn't become agent-ready because every endpoint passes the same universal documentation checklist. Readiness is contextual. It depends on the type of consumer, task to be performed, and the particular set of permissions and constraints. 

That grounding - the context to how an API will be used - matters because not every smell identified by Hermes required immediate correction. I've seen too many well-intentioned API improvement initiatives launched with great fanfare, only to be dashed upon the rocks of "all or nothing" edicts. An unconventional HTTP method may be constrained by a legacy implementation that is not worth rebuilding. A short description may be perfectly adequate for an operation that will never be exposed outside its existing internal use. Some endpoints may not belong in an agent's toolbox at all. 

For an organization considering a similar effort, the sequence should begin with scenarios rather than specifications. 

First, define the outcome the agent is supposed to produce. Not "use the customer API", but something concrete: retrieve an eligible customer account, validate its current status, submit a particular change, and confirm whether the change was accepted. 

Then identify the smallest set of API operations required to complete that scenario. Include not only the obvious transactional endpoint, but also the supporting operations needed to discover identifiers, retrieve valid values, check permissions, interpret errors, or confirm final state. 

Only then should the organization inspect whether those contracts communicate what an unfamiliar consumer needs to know:

1. Selection: When should this operation be selected instead of another?
2. Constraints: What does each input mean, and what boundaries apply?
3. Auth: Which credentials, roles, or scopes are strictly required?
4. Side Effects: What state changes will the operation cause?
5. Success State: What does a successful response actually mean?
6. Error Handling: What errors can occur, and how should the consumer recover?
7. Completion: How can the consumer verify the task is finished? 

Where those answers exist only in source code, old examples, team memory, or the head of the person who built the integration six years ago, that is the documentation gap to remediate. 

Wrapping up

If you take away two takeaways from all this, they should be the following. 

First, don't assume that every existing API should be agent accessible. Exposing more operations can increase ambiguity, make tool selection harder, and expand the surface over which permissions and side effects must be accounted for. 

The second is not to treat agent readiness as a one-time documentation campaign. If the team we've been talking about repairs their 42 endpoints but continues creating new APIs with generic descriptions, unconstrained inputs, and opaque responses, the organization will simply exacerbate the same problem. 

The paper concludes by noting the organization did not use Hermes solely as a cleanup detector. It revised its doc standards and incorporated automated assessment into its API review process. What began as a diagnostic experiment became part of API governance. 

Ultimately, the path to an agent-accessible API future does not start with MCP. It starts by confronting the undocumented assumptions embedded in our API present. Agents did not create those gaps; they simply made them much harder to ignore.

Stay cool, get some summer break in if you get the chance, and I'll catch you in the next edition,

Matthew (@matthew in the fediverse and matthewreinbold.com on the web)