I didn't know it did that - Postman Postbot

Last week I was updating the public workspace for Momento in Postman when I saw a little helmet icon in the corner of my screen. I hadn't seen that icon before, and being curious by nature, I stopped what I was doing and clicked on it. Much to my surprise, it opened up Postbot - the native AI helper inside of Postman.

I've been a Postman user for a long time and Postbot is relatively new, I always forget it exists. So I took this as an opportunity to procrastinate on my work task and do a bit of exploration. I wanted to see how good it actually was. I knew Postbot was supposed to help you build your tests and answer questions about your requests, but I didn't know how involved I could get or how much time it would save me in real life. I started off by asking it to visualize some data for me by turning a JSON list into a bar graph grouped by location. Then I followed up by asking it to create a JSON schema for the response and using it in the documentation and tests for validation.

It did surprisingly well on both tasks, so it was time to really put it through its paces. I gave Andres Moreno, my enabler friend who loves APIs as much as I do, and Sterling Chin, engineering manager over Postbot, a call and we hopped on a live stream in the Believe in Serverless community to figure out exactly how far we could push it.

Before I continue, I want to share this is not a paid endorsement. I'm just really excited about a new-ish feature in an app I know and use on a daily basis.

What Postbot does well

In the live stream, we went through several key areas of Postman, putting Postbot to the test to see just how much time it can save developers and how much work you can offload to it.

Test writing

I can't tell you how much time I've spent in Postman over the years writing tests. Not necessarily difficult or tricky ones, but simple validation tests or tests that set chaining variables to flow into the next request. It's a lot of typing the same thing over and over again with minor changes in each request. With Postbot, that all changes.

I found great success typing phrases like "create a test that validates there is at least one record with an id of X" or "verify the response does not have any records where the person is older than Y." Postbot would look at the response of the request I'm on, get context of the schema, then write JavaScript and tests that do exactly what I asked. If there was a problem with the test or I needed to make an update to the criteria, conversation history is preserved so I could follow up with "fix that test to verify no people are older than Z." The test is updated in place without needing to touch anything 🔥

It also has autocomplete based on the name of your test! Similar to inline coding assistants you see today like Copilot and Code Whisperer, all you need to do is intuitively name your tests and the code for it will show up automatically. Hit Tab to accept the suggestion and you're done!

Schema creation and validation

While I'm a firm believer in API-first development and fully building your API spec before anything else, I do realize many people prefer a code-first approach and might not have a strongly defined specification at all. Turns out that might not be a problem anymore.

You can tell Postbot to "create a JSON schema for the response and add it to the documentation with meaningful definitions for each property" followed by "now add a test that validates the response against this schema" to fully document and validate your payloads. I was genuinely impressed by the quality of the produced schema and how quickly I could get a starting point for meaningful documentation. If you have flexible objects in your schemas defined by a oneOf, anyOf, or allOf, you might not get the best results right now, but I assume that will improve over time as LLMs continue to get better.

The main takeaway here is to use it as a starting point and make sure you double-check the schemas for minor details.

Visualizations

I always forget the visualizer exists. When I do remember it's a thing, I always struggle to get it working properly. It's super flexible in what it can do, but when something is flexible that usually means it's complex, too.

If you want to take your JSON payloads and turn them into meaningful graphs and charts, Postbot can do that for you in seconds. Not only can you ask it for exactly what you want, but you also have a wizard that will walk you through options if you don't know what you're looking for. Bar charts, line graphs, and pie charts are some of the options you can generate with a single command.

I did find it difficult to update the visualizations in chained commands. So if you ask it to "create a bar chart of the results" followed by "now group it by city/state" then "update the colors to various pastels", it struggles a bit. Your best bet is to figure out exactly what you want and ask it in a single, detailed command.

Documentation

Ah, documentation. The thing that developers hate writing yet are so angry if it doesn't exist. Meaningful documentation goes a long way with APIs and it's much more involved than simple schema definition. Explaining context and building story around use cases turn flat, code-generated documentation into engaging, effective tools for developers. This can be hard to do for many of us! While developers tend to be creative, it's not often the storytelling type of creativity.

Since Postbot uses generative AI to come up with responses, you can seed it with anything and have it come up with a consistent story across all of your requests. To me, this is probably the most significant yet underused capability of Postbot. Using it to elevate your documentation is a game-changer, and I expect it will open up a swath of possibilities down the road.

Fixing things

I'll be honest, when I saw "fix tests" as a suggestion in Postbot I glanced right over it. Seeing something as generic as "fix it" doesn't typically fill me with optimism. I see something like that and I usually look right past it. That's too vague to do anything meaningful.

But it works!

Per Sterling's suggestion, I intentionally wrote broken code and ran a request. After it broke, I told Postbot "fix the tests". It was able to analyze my code, find my error (I changed a variable from responseData to responseData2), and update it to use the correct variable name. So it worked for syntax/code errors!

I also tried another test that's all too real. I wrote a status code test to verify the response was a 201 Created. My endpoint actually returns a 200 upon success instead of a 201. When my test failed, I asked Postbot to fix it and it updated my test to check for a 200 OK instead of a 201. I've had so many copy/paste test failures like this over the years, it's really cool to see something that can find and fix them for me automatically.

What it does not do

One of the fun things I did on the live stream was simply see where my assumptions were wrong. I was tempted many times to ask Sterling "what would happen if I..." but decided to just try it myself. I learned real quick I had several assumptions that were flat out wrong.

Work with API definitions

Postbot doesn't do anything with API definitions. I tried asking it things like "Does this specification meet OAS best practices" and "Update the endpoint descriptions to be more meaningful" to no avail. It doesn't understand the spec as context. Seems like it might be limited to requests only for now.

Create or work with other resources

You can't ask Postbot to create a new collection for you. Or a request. It's intentionally not a creation tool. It's intended to be used as a pair programmer to work on a single request at a time. With that in mind, you also can't ask it to update a request you don't currently have focused on screen. The conversation history in Postbot is scoped to a specific request, so keep your conversations focused on the tab you have in front of you.

Undo

While I was trying to update my visualization to render bar graphs in various pastel colors for Easter, something went wrong and put my request in a bad state. My tests were full of syntax errors and the visualization completely broke. I had definitely stumbled across a bug, which is fine - everything has bugs. But the part that irked me was that there is no undo button. I pressed ctrl + Z (or command + Z for you Mac folk) to get back into a working state, but nothing happened. This led me to realize that everything Postbot does is destructive. Kinda scary phrased that way, but technically it's true. So be careful when prompting, because it's possible to lose your work.

Answer random questions

Postbot is designed to help you build things inside of Postman. It knows how to write and fix tests, create documentation, and build rich visualizations. It's not meant for answering things like "What is the weather today?" It will be the first to tell you that! If you ask it a random question, you'll get back a "I can only help you with Postman and API related queries" response. But you can ask it questions around APIs, like "What status code should I return on a POST" or "How should I implement idempotency on a PUT" you'll get meaningful answers and links to documentation or blog posts on Postman's website.

I had incorrectly assumed that Postbot only worked when asking it to change something on a request. Knowing it can also be a great reference for API-related questions is a huge win!

What I learned

Postbot is a big deal. If you haven't used it before, it's easy to shrug off as Postman's answer to the AI boom. Now having used it for real, I realize that this can fundamentally change how I use Postman. It not only has me moving faster when it comes to API creation, but it has me creating more complete, thorough tests and documentation.

Some "insider information" I discovered on the stream was that Postbot is powered by OpenAI. I wasn't able to get more info from Sterling on this, but I'm always willing to speculate. It appears to me that Postbot has a layer of Postman-specific context that it gathers and sends to OpenAI when someone types a message. The response from OpenAI is then processed by this layer and either turned into an action, like updating tests or other request information, or into a simple answer to your question. Based on the quality of the generated code and Postman-specific influence, I'd guess that they are using LangChain with GPT-3.5 and a Postman Knowledge Base for a RAG query. But again, it's just speculation.

I get periodic errors with extensive use of Postbot. Randomly I'll get a response back that tells me there was an unexpected error and that I should try again. When this happens, I send the same message again and usually get a working response. Remember, there's a ton of stuff going on when you hit that Enter button. Errors happen - especially in new tech. Try it again if you get an error. If you get an error again, try once more.

Postman seems to be investing a lot of time and energy into Postbot and I can see why. This is quickly becoming an enabler for both technical and non-technical users. I expect to see it doing things like fetching requests from the public API network and building workflows in the not too distant future. It's an exciting time to be a developer and an API builder.

Personally, I'm sold. While it's not perfect, it certainly beats doing everything yourself. So go give it a shot and tell me how you use it. I'm always looking for new, creative ways to build.

Happy coding!