What Happened When GitHub's AI Copilot Wrote My OpenAPI Description

In Issue 168, I took a look at GitHub's Copilot AI. Advertised as "Your AI pair programmer", the AI - trained on publicly available source code uploaded to GitHub - would suggest what code someone should write in real-time and directly in the editor. In that issue, I expressed concern with just how many API keys the AI was inadvertently surfacing. After all, the AI couldn't tell which code was a quick and dirty tutorial or example with embedded API keys and which code demonstrated proper security practices; it just saw an attempt at calling an API and returned its best guess.

More subtly, I was confused as to who Copilot was for. New developers don't have the practical experience to judge and possibly correct what Copilot suggests. Like the unfortunate developers from Nissan, beginners would use whatever they found uncritically. The suggestions require expertise to evaluate. But why would a skilled developer want to "babysit" Copilot generation if they had the expertise to just write their own code? I struggled with that paradox.

However, that paradox really only exists if we're talking about production code. And, pragmatically, there's a host of code required for modern development that is (1) not production code and (2) tedious to write. Test cases, particularly unit tests, can be pretty repetitive.

Creating an OpenAPI definition is another possibility. The prevailing wisdom is to create an OpenAPI description of an API, share it with stakeholders, and rapidly iterate before coding. However, that is much easier said than done. Technical product leadership opts out upon seeing the JS or YAML schema, delegating what looks like a programming task to the programming people. And these developers, in most cases, aren't specification gurus; how could they be when they only interact with the OpenAPI specification on a project basis once every 6-18 months? That is not a recipe for retention, let alone best practice mastery.

Can GitHub Copilot do better?

MY SET-UP FOR GITHUB COPILOT OPENAPI GENERATION IS BARE BONES

To provide real-time suggestions, Copilot runs as a plug-in/extension in several major IDEs: Visual Studio Code, Visual Studio, JetBrains, and Neovim. Adding the extension to my instance of Visual Studio Code on a new, unused laptop was straightforward. The only other extension I installed was Stoplight's Spectral extension. I turn off a handful of Spectral's default rules - tag and licensing requirements, for example - to try and keep samples uncluttered.

In short, Copilot helps write the API description and Spectral spot checks what it (and I) produce.

GITHUB COPILOT NEEDS CONTEXT

With excitement, I started a new OpenAPI YAML file... and nothing happened. Many of the videos and GIFs of Copilot in action emphasize a real-time modal pop-up containing suggestions. Perhaps that is a setting somewhere I needed to enable, but throughout my experimentation, I never got that experience.

However, having a modal interjecting whether I needed it or not would have been distracting, if not annoying. Instead, I was grateful to be able to call upon Copilot at my discretion by typing Ctrl + Enter. This opened another pane that showed suggestions. And this is where we need to talk about context.

At first, I thought my Copilot was on strike; the suggestion pane remained blank. However, as I learned, Copilot relies on context for clues on what it should suggest, and that's not just the file you're currently working on. That context also includes neighboring or related files within a project. Copilot also uses any URLs of repositories or file paths it finds to identify even more clues on what it should suggest. This context is then used to offer anything from single lines to whole code blocks.

Because I was on a clean machine, and my public repos don't have OpenAPI descriptions in them, Copilot was having trouble getting its bearings.

What I needed was some seed material. Last year, I codified several common API behaviors in easily reusable OpenAPI templates in a Postman workspace. I copied several of these OpenAPI references to the same file folder as the OpenAPI description I was trying to write, and, Voila! now I had suggestions.

WRITING OPENAPI DESCRIPTIONS WITH COPILOT IS A COLLABORATION

My use case was to attempt a simple API for quotes I've collected and used for my Twitter Quote of the Day bot. Each quote has two fields, "quote" and "attribution".

After the initial elements, I created a quick "GET" method on a "quotes" collection. I then made a "POST" method and asked Copilot what was next. The suggestion included a summary, operationId, tags, request body, and the response structure that referenced my previously created 'quote' instance schema! It was also surprising how Copilot took my cheeky GET description of "got you some quotes, yo." and echoed by a modified "Added your quote, yo." in the POST.

That was kind of amazing.

The suggestion wasn't perfect. For example, Copilot suggested a 200, rather than 201. There also wasn't a 4xx code.

When I attempted to add a 400 error, I appreciated the presence of the description but was surprised that there wasn't a content block.

When I went to add an error object for my newly created 400 status, I was surprised by the different options (see below). Some only were simple strings. Others returned message strings and a code in several formats.

Returning to the GET, I wanted to create some pagination, preferably with limit and offset affordances. However, the suggestions provided after I typed parameters weren't great.

And now, we need to revisit context. In that set of codified best practices, I defined a robust limit and offset pagination example. Curious, I copied that template to a new YAML file in the same directory. I then polled Copilot again and got a much more robust definition: minimums, maximus, and defaults.

CONCLUSIONS

There's a reason GitHub's product is called Copilot and not autopilot. As shown above, there is still the need for a person to provide judgment and correct where necessary.

However, not having to Google exact syntax, and being able to double-click and customize from best practice, is huge. I'm not in OpenAPI descriptions daily. A lot of the nuance and formatting, especially around nested arrays and objects, doesn't stay with me. I forget the expressive details possible in parameter schemas. With a great set of reference examples Copilot can surface, this minutia is now just a double-click away.

I've written before on the importance of leveraging the environment (in this case, the IDE) to nudge better behaviors during development. Copilot takes those static reference materials and puts them, in near real-time and at the point of creation, at the fingertips of developers. API COE's, enterprise architects, and even API product leaders - basically, anyone who needs greater consistency and cohesion from their API portfolio - can benefit. Copilot, in conjunction with thoughtful, consistent templates placed in the proper context, could improve and align API design.

GitHub Copilot is $10/month or $100 per year. People that sign up for an annual subscription get two months for free. You can sign up for a 60-day free trial and can cancel anytime before the trial ends to avoid being billed.

MILESTONES

We have more detailed info on the Optus API breach mentioned in the previous newsletter. From reporter Jeremy Kirk, we learn that the customer API (1) required no authentication and (2) used an incremented number as a primary record identifier.

Further, it has been implied that the API was made available for integration testing. Rather than fake data to conduct testing (which, granted, is nontrivial), real customer data was made available in the hopes that nobody would notice for the duration of the test.

Yikes.

WRAPPING UP

If you're a Patron, Thank you! My Patrons ensure these emails remain free of ads, paywalls, or information. Because of their generosity, I can also do things like sign up for GitHub Copilot account. If you're interested in joining them, go to my Patron page and sign up.

That's all for now. Till next time,

Matthew

@libel_vox matthewreinbold.com

While I work at Concentrix Catalyst, a strong signal in a world of simulacrum, the opinions presented above are mine.