Docgen: Enhance your code with OpenAI’s GPT in your Jetbrains IDE, a cautionary tale

Innovation is at the core of Profiq’s mission. Arguably, the most innovative development in technology recently is ChatGPT, developed by OpenAI. In an earlier article, we explored OpenAI’s Codex model, which we used for code generation and editing. In this follow-up article, we share our experience leveraging OpenAI’s models programmatically via its API, for creating a plugin for Jetbrains IDEs.

Querying OpenAI from the IDE

We’ve been using ChatGPT from the browser in various ways to aid with our daily software development and QA. When the interaction with ChatGPT involves frequent code generation and editing, using the browser interface becomes cumbersome, as there’s a lot of copying and pasting between the browser and the editor. As many of us at profiq are avid JetBrains users, we wondered whether GPT models could help us directly in the editor with OpenAI’s API? A number of plugins already exist that aim to integrate ChatGPT. There’s Github Copilot, which is more of a completion tool than a prompt-response assistant. Besides Github Copilot, we could not find any plugin that inserts generated code directly in the editor buffer without needing to switch to the browser. So, we decided to write one.

Which useful daily task could a GPT model help with? It’s well known that most developers are not very fond of writing documentation for their code. Could GPT help with that? Of course! By generating code documentation, it could save time, and let developers focus on actual problem-solving. Thus, Docgen, a Jetbrains IDE plugin for generating documentation, was born.

Getting started with the plugin

When we started developing the plugin, OpenAI’s service of choice for code generation and code editing was Codex, so we used the Codex API. Since we did not have much Java expertise or prior experience with IntelliJ Platform plugin development, we turned to ChatGPT for help with writing the plugin itself. It guided us through development one step at a time, gave us useful code suggestions, and helped us write the plugin considerably faster. Our goal was to create a simple MVP that would generate documentation for code in the current editor buffer, including formatted docs for individual functions. Although this sounds like a complex task, Codex was up to it. After a few days, we had a working version of the plugin that, triggered by a keyboard shortcut, was able to generate code documentation for various languages known to Codex. Everything was bright and sunny.

Change is the only constant

Satisfied with our MVP, we submitted it for review on the Jetbrains Marketplace on March 21, and awaited an email response with feedback. We received a different email from OpenAI than we expected—effective March 23, all support for the Codex API would be discontinued, and customers would need to transition to a different model. Yes, OpenAI gave developers 3 days to transition from Codex to another model. Three days! Well, at least we got the notice before the plugin’s release.

Fortunately, since we were making only one request to the Codex API in the plugin’s source code, the implementation change needed was not too dramatic. First, we replaced the Codex model with GPT-3.5-turbo, the most sophisticated model at the time. As GPT models are conversational, the main programming challenge was to parse the documentation string from the response and insert it into the right position in the current buffer, instead of simply replacing the entire buffer content as before. A bit tricky, but doable. There was another challenge, though. While Codex always returned the generated documentation string in an appropriate format for the given programming language, GPT-3.5 did not. No matter how we wrote the prompt, it returned an explanation of what the inputted code does in plain English instead of the desired documentation string. Bummer.

Narrowing the scope

One consequence of the necessary model change was, we had to give up the idea of a plugin that would document source code in any programming language known to the model. Since Python is one of the two most popular languages at profiq (along with Javascript), we made the prompt Python-specific:

Generate high-quality docstring for the following 
Python function including function signature:

{{ function code }}

With this prompt, GPT-3.5 actually does return a valid docstring (along with an explanation in plain English), which we can parse and insert directly after the Python function’s signature. One could, of course, modify the prompt to generate documentation for a different language, such as, Java:

Generate Javadoc for the following code:

Unfortunately, since GPT models return a conversational message with an explanation in plain English in addition to the code itself, the documentation string always needs to be parsed from the message. Since each language has its own documentation format, we’d need to implement a custom parser for each language. Our goal, however, was to create an MVP, so the plugin does not include any additional parsers for other languages for now.

In summary, we ended up creating and publishing a plugin that can generate and insert documentation in docstring format into a selected Python function. Despite our initial ambitions for a more general documentation plugin for many languages, we are happy to have a tool we can use inside our IDE without switching to the browser and copy-pasting.

A word of caution

We’ve learned a few lessons while building on OpenAI’s services and APIs. This year has definitely been a turning point in the AI domain. Current AI models are evolving at a rapid pace, and we expect the models and their capabilities to significantly improve in the months to come. Governments worldwide are currently trying to catch up and limit and control the AI (r)evolution. As we’ve seen with Codex, AI platforms rise and fall. We recommend using caution when building on top of the existing APIs, as they may change unexpectedly.

If you’ve read this article until the end, you might be curious and want to try the plugin yourself. It’s available for download from Jetbrains Marketplace free of charge: https://plugins.jetbrains.com/plugin/21294-docgen

Leave a review if you like the plugin or run into any issues with it. We hope you find it useful!