Automated API testing with Newman in Azure DevOps
Testing software is an essential part of developing an enterprise application. It has always been an argument to what extent we should test our applications, however, the “why” behind testing software is accepted by almost any developer on this planet. Today we are going to shed some light on how we can automate testing our APIs using Postman and its testing tool called Newman and see how we can integrate this super useful tool into Azure DevOps.
I am using a sample public rest API which is provided by the Postman official website in this link.
You can also just go to the Postman Explore tab and search for Continuous Quality with Postman. Then you are able to fork the project and start using it in your own workspace.
Disclaimer:
In this article, all the steps and examples are done on macOS. However, this should be the same on any OS since Postman and Azure both have the same look on all platforms.
Imagine you have an API endpoint on your backend server and you want to test your application’s API. What we normally do is check manually if it’s working correctly. We’ll start with a small GET request:
To get started, first, We’ll send a GET request from The Power of Libraries folder called Chai:
It works! So let’s write some test cases!
To start writing some test cases we have to open up the Tests tab.
You can see the pm global variable is here to begin writing your tests. The good news is that you can write your test in JavaScript language and there are some pre-written tests in this collection. An easy way to ensure our tests are working is to start with the simple ones. This means our response has a response of 200 HTTP status codes:
To see if the tests are passing, click on Test Results in the below panel. As you can see the test are labeled PASS in our result.
You can always use the Postman console to debug your tests. To open the console and monitor the requests you are sending you can use this button on the bottom left corner of the Postman.
This screenshot is taken after the above request finished successfully. You are always able to see the logs in here.
Now we can see that our API is working and the test is passed successfully. We are not going to show you how to write tests because the Postman test console is built on the Jest test framework. Our goal here is to show you where you can write these test cases and where you can utilise them in an automated way. So, let’s see how we can integrate our Postman test cases into our CI/CDs in an automated fashion.
Introducing Newman:
Newman is an open-source project for Postman which enables you to run Postman tests via the CLI. It enables you to run and test a Postman Collection directly from the command line. It’s built with extensibility in mind so that you can integrate it with your CI/CD pipelines. So let’s see how Newman works. First, we need to install 2 major packages:
- Yarn or NPM (For the NPM installation, you can use https://nodejs.org/en/download/ and for the Yarn installation, you can use https://classic.yarnpkg.com/lang/en/docs/install)
- Newman using yarn add global newman or npm install newman -g.
And that’s it! All you needed to do to install Newman! But before we configure the CI/CD pipelines, we want to take a look at how we can run it on a local machine. To do so, you can export our Postman collection in a file first. (if you have downloaded the JSON collection you don’t need to do this step)
Next, hit Export and choose a file location. I name the file Postman.postman_collection.json.
This might be different due to the different Postman versions.
Now go back to the terminal and navigate to this file directory. Then run this command on the CLI:
newman run Postman.postman_collection.json.
The runner should start running the test cases. This should be the final result:
Nice! Newman runs the Postman tests we had. Now, it’s time to see how we can integrate these steps on Azure DevOps pipelines.
Run Newman on Azure DevOps:
To begin making the Newman setup on the Azure DevOps pipelines, first, let’s create a new project by clicking on this button:
Next, give this project an informative name, I named it testAPIwithNewman and hit Create.
After we create the project, we should push the exported JSON collection file to the server. If you are using Azure Repos, you can clone via your IDE/code editor or push the file via the command line. The instructions are super easy to use. You can simply go to the Repos menu.
Now it’s time to go to the Pipelines page and use Azure pipelines for our Newman tests.
If this is the first time you are creating a new pipeline you will see this page and you can click on the Create Pipeline:
Azure DevOps will ask you for your git configurations. Although I have used Azure Repos, let’s click on the Use the classic editor to create a pipeline without YAML at the end of this list.
It will redirect you to the page where you can choose a git source, project, and its branches. We have not defined any environments, so we can just go for the master branch. I have used Azure Repos so I will select Azure Repos git and hit continue.
Scroll down in the list and choose Empty pipeline.
It will create a new pipeline with an agent for us. We can give it a name, I’ll go with testAPIwithNewman-CI and select a pipeline OS for the image. I’ll choose ubuntu-latest.
Now select Agent job 1 under the pipelines that we just configured.
And give this agent job a name and choose ubuntu-latest under the Agent specification section. You can also define a job time out for your agent.
Now it’s time to create our Newman test job. In order to do that you have to click on the plus button on the right side of the selected agent:
you can search for the Newman extension in the search box on the side pane that shows up.
Select the Newman the cli companion for Postman. I have already installed the extension, and hitting the Add button will bring you to the Newman extension installation process in your Azure pipelines.
Hit Get it free and you will be redirected to this page which basically downloads the extension in your Azure DevOps. Hit the Download button and wait for it to be installed.
After the installation select the added job and you can see the configuration on the right pane that pops up. First, we can give it a name. I’ll go with Newman Test. You are also able to provide a URL for the JSON collection, but since we have pushed it into the server, we can use the file for our pipeline as well. Select the postman JSON collection file by hitting on this button under the Collection File Source section and select our collection:
If you have any environment file you can set it under the environment file section. We are not going to cover this part in this blog, but for demonstration purposes, we should select the same file here under the environment file section.
Next, you should enable continuous integration which is not enabled by default. To do so, we have to go to the triggers tab and check out the Enable continuous integration checkbox. This will make sure every time you push a commit to the git repo, the pipeline triggers.
Next hit the save & queue button.
Easy as that you now have a ready and running Newman API testing pipeline for your project!
Let’s make sure it works!
Now you can see the pipeline has been started and if you click on the running job you will be redirected to this page where you can see which command generated which logs. In this case, we definitely want to see our logs about the failing tests.
Now, if you commit a new change to the git repo, the automated test pipeline will start immediately.
There you go! Now you can have a safe API development by simply adding Postman tests using Newman!
Conclusion:
In this article, we demonstrated how you can test APIs using Newman which enables you to run your tests by the command line for running Postman collections, And also how you can integrate this tool in Azure DevOps to have a reliable test stage for our CI/CDs.
Who am I
A passionate, curious full-stack developer with more than 8 years of experience building apps and websites using PHP & Javascript with a bit of perfectionistic inside. Enthusiastic about state-of-the-art techs and practices. Away from my screen and keyboard, you can find me playing basketball or DJing as a hobby.