Test Automation Solution with Xray and Cucumber

Xray is a complete Test Management tool for Jira. It is a full-featured app that does not require any other software in order to run. Xray supports the entire testing life cycle: test planning, test design, test execution and test reporting.

On the other side, Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in a natural language, while the implementation of each specification’s phrase (i.e. step) is done in code (e.g. Java, Ruby or some other language).

In this article we give an explanation of the steps about how you can create Automated Cucumber test with the help of Xray and how you can import the automated test results back in JIRA.

 

Create Cucumber Test with Xray

 

One of the ways to create an automated Cucumber using Xray is to navigate to the Jira user story that you are planning to test.

 

JIRA User Story, test automation

Figure 1. JIRA User Story

 

By clicking on Create new Test button, the form for creating a test case is opened. In order to create Cucumber Xray test, under Test Details tab, for Test Type choose Cucumber and from Cucumber Type drop-down choose Scenario. (The option Scenario Outline is used when you want to create instructions that contain variables).

 

Cucumber Test Creation with Xray

Figure 2. Cucumber Test Creation with Xray

 

The next step is to populate Cucumber Scenario with test steps in Gherkin language.

 

Cucumber Test Creation – Creating test steps in Gherkin Language

Figure 3. Cucumber Test Creation – Creating test steps in Gherkin Language

 

Under the Link issues tab you can see that the test is automatically linked to the user story from where the test was created.

 

Cucumber Test Creation - Associate test to User Story

Figure 4. Cucumber Test Creation – Associate test to User Story

 

By clicking on “Create” button the Cucumber test will be created.

 

Xray Cucumber Test in JIRA

Figure 5. Xray Cucumber Test in JIRA

 

Test Execution and Importing Test Results in JIRA

 

Xray enables exporting Cucumber tests to be exported as .feature file that further can be used in your automated tests. In order to export .feature file, from the created test choose More -> Export to Cucumber.

 

Exporting .feature file from Xray Cucumber Test

Figure 6. Exporting .feature file from Xray Cucumber Test

 

Further, you can import the exported .feature file in your test automation project, for example implemented in Eclipse.

 

Exported .feature file in Eclipse

Figure 7. Exported .feature file in Eclipse

 

After implementing the step definition file, using for example Selenium Web Driver, for the given feature file, and executing Cucumber test in Eclipse, Cucumber.json file with results will be generated under the target folder in your Eclipse project.

 

Cucumber.json file with test result after test execution in Eclipse

Figure 8. Cucumber.json file with test result after test execution in Eclipse

 

Using Xray, the test results from the generated json file with the test results can be imported back in JIRA, and associated with the corresponding test, by manually importing the json file or by using API that is provided by Xray.

 

Importing Cucumber automated test results in JIRA through UI

 

In order to import the generated result from the automated tests, in the Test Execution* associated with the Cucumber test, click on More -> Import Execution Results and import the generated json file in JIRA. (* Test Execution is custom JIRA issue type introduced by Xray, that can be described as an assignable, “schedulable” task to execute one or more tests for a given version/revision along with its results. For more information on different Xray custom JIRA issue types, please refer to https://confluence.xpand-addons.com/display/XRAY/Terms+and+Concepts)

 

Import Cucumber.json file with test results in Test Execution

Figure 9. Import Cucumber.json file with test results in Test Execution

 

Once the execution results are imported in JIRA, Test status is updated. In our case the test has failed, so correspondingly its status is updated in the associated Test Execution.

 

Updated Test status after importing .json with test results

Figure 10. Updated Test status after importing .json with test results

 

Execution details, including the error that was thrown during automated test execution, can be accessed as depicted on Figure. 10

 

Test Execution Details

Figure 11. Test Execution Details

 

Importing Cucumber automated test results in JIRA through API

 

Xray provides specific REST endpoints for different test execution results depending on the testing framework that is used for test automation. In the table below are listed some of the REST endpoints, while the full list can be seen on: https://confluence.xpand-addons.com/display/XRAY/Import+Execution+Results+-+REST

REST endpoints

In a real-world scenario, importing automated tests results in JIRA through Xray provided APIs will be executed in conjunction with CI/CD tools, like Jenkins, however in this article, for demonstration purposes we will use JMeter to perform POST operation on the REST endpoint for Cucumber test result output format:  /rest/raven/1.0/import/execution/cucumber

 

JMeter script for importing results through Xray REST API

Figure 12. JMeter script for importing results through Xray REST API

 

After invoking REST API, through JMeter, corresponding Test Execution associated with the Test is updated, showing Test as Pass:

 

Test execution results in JIRA after importing through API 

Figure 13. Test execution results in JIRA after importing through API 

 

Conclusion

 

Xray plug-in for JIRA provides ideal solution for test management on all projects where JIRA is used for tracking requirements and bugs. Besides it supports management of test cases, test planning, test execution and requirements coverage with its built-in support of Cucumber and other test automation frameworks enabling support whether is needed manual or automate testing.

 

Aleksandra Sekulovska

Sr. Technical Consultant