Karate vs. Cucumber: Comparing Frameworks for Automated API Testing

Companies are increasingly adopting innovative practices for their software projects, to improve the quality in the development and delivery of a project.

This implies leaving behind traditional methodologies and adding agile development as a new way of carrying out projects.

This innovative, agile ecosystem, where there is continuous integration and deployment, and which involves frequent iterations, requires more deliverables than the traditional one, since the product grows in functionality with each iteration and is built by integrating the customer.

When talking about agility, it is common that the first word we think of is Scrum, since it is one of the most used methodologies in agile development. However, there are many others that are less used or uncommon, such as XP, Kanban, Lean.

These methodologies make it possible to achieve better results, and to respond to changes more quickly and flexibly, integrating customers throughout the development process. The philosophy of these methodologies is based on the Agile manifesto.

Regarding the application of agile development methodologies, you should consider building the product under these fundamental principles:

• The software should keep it as simple as possible.
• You must have quick feedback.
• structure must allow for incremental change.
• That the software is quality work.

Under these principles and thinking about the nature of software development, where changes are incrementally and continuously integrated, it is key to maintain quality and functionality.

How to maintain the quality and functionality of the software?

You can do this through automated testing.

Automated tests help in this continuous process of changes and deliveries, because once you automate the test set, you can see at an early stage that a change caused some functionality to be lost in the software or modified part.

Frameworks for Automated Testing

Karate and Cucumber they are two frameworks for automated testing and we want to show you which one provides the best results in terms of usability and compatibility.

Cucumber and Karate are based on the BDD (Behavior Driven Development) methodology, which is an improvement of TDD (Test Driven Development), a test-based development methodology. BDD, unlike TDD, includes the business and allows both the technical and business teams to relate through a common language called Gherkin, used to describe software behavior. It uses five general statements:

• Feature: tells us the feature of the software.
• Scenario: It tells us in which scenario the test will be carried out.
• Given: It tells us that “since” there is a scenario, the test will be carried out.
• When: tells us that “when there is a condition” the test will be performed.
• Then: tells us that “then” an expected behavior will occur in the test.

Now that you've seen how the gherkin language works, we can review the frameworks we'll use to do the automated tests.

Cucumber

For this exercise, we'll be using Cucumber with JVM. If you are using a language other than Java, there are others integrations. We'll use Maven for dependencies. The following is added to the pom.xml:

Cucumber Spring integration

To add cucumber-jvm:

‍

Then, we added the Junit and cucumber dependency:

Finally, we added, the Spring dependency:

‍

Then you must configure the REST service. For this example, we will use a API REST with a list of available games.

Cucumber Steps Definitions

To run the tests Cucumber with Junit, you need to define the steps, create an empty class with the @RunWith annotation (Cucumber.class) and @CucumberOptions to define the options we'll use.

Then, on the road “src/test/resources/features/” you must write the files that contain the characteristics of the tests written in the Gherkin language.

This scenario makes a GET call to the service that delivers the list of games and validates that the information is displayed correctly.

After having this scenario created in Gherkin, you must code the steps of each feature in Java, for this purpose create a class that contains the same steps so that the steps are recognized.

NOTE: We must use Rest-Assured to validate API responses.

Now you have everything you need to run the test. Run the test with Junit in the RunTest class that we created. If everything is correct, the tests ran successfully.

To view the report you can use cucumber-reporting.

Now we move on to Karate.

BDD Karate

Karate is a simple testing framework like Cucumber and has tools included that make it very effective. This is how they describe it originators:

“Karate is the only open source tool that combines API test automation, simulations, performance testing, and even user interface automation in a single, unified framework. The BDD syntax popularized by Cucumber is a neutral and easy language even for non-programmers. Powerful JSON and XML statements are built in, and you can run tests in parallel for greater speed.”

For this example, we will not use all the tools, but we will use several to perform the same exercise as with Cucumber and make the comparison.

First add the following dependencies to the pom:

‍

Then create a class Runner to add the Karate libraries and then run the tests from there. In the class, the method that executes the tests must have the annotation @Karate .Test:

Then create the file Feature which will contain the scenarios and the steps to be executed:

‍

Then run the tests. This will automatically generate an html report:

Image: Results with Karate

Now that we know both frameworks, we can draw conclusions!

While both frameworks work in a similar way, Karate is simpler to use than Cucumber. Let's look at the advantages of both frameworks:

Karate benefits:

• With Karate you don't need to be an expert in programming to implement it.
• Unlike Cucumber, you can run tests simultaneously.
• At Karate we validate the answers directly in the file Feature and it even allows us to compare answers, since it works natively with Json files.
• By including certain tools, it automatically generates reports to view the test results, while Cucumber needs to be configured and a dependency must also be added to allow the results to be viewed with a graphical interface.
• Fewer configurations are needed to achieve the same results.
• We don't need to include other additional libraries to do the same job (Rest-Assured, Serenity BDD, cucumber-report).

Cucumber benefits:

• Cucumber can be run from Maven or from Junit using the same configuration.

• Cucumber allows you to use different libraries or frameworks that interpret the results to be displayed in pie charts with a much friendlier and better understanding interface.

• You can use the reports generated by cucumber in project management tools such as Jira via the Xray integration.

• You can use Cucumber with Selenium, if your tests go beyond the backend and you also need to test the frontend.

Karate or Cucumber: Which is Better?

Both tools are powerful and the one you choose will depend on each case and the way you need to view the results. If what you need is to provide information at a managerial level, with a better graphical interface, and better interpretation of the results, Cucumber It is the best alternative if you complement it with Serenity BDD or another component to visualize results. If what you need is to graph the results for a technical area, Karate It's the most agile alternative, because you'll get the same results with fewer steps and complexity.

Ready to optimize your API tests with the best tools?

At Kranio, we have experts in test automation who will help you implement frameworks such as Karate and Cucumber, improving the quality and efficiency of your developments. Contact us and discover how we can promote your technological projects.