Xray integration with Robot testing framework

In this article we are going to describe the integration between Xray test management plugin for JIRA and Robot test automation framework utilizing Jenkins. We will elaborate the configurations that need to be done, so results from automated execution of tests implemented in Robot framework can be transferred back to Xray/JIRA where tests are documented. All this processes can be implemented with Jenkins job and accomplished without any manual intervention.

Introduction

Xray is one of the most popular test management tools for JIRA. Xray supports the entire testing life cycle:  test planning, test design, test execution and test reporting. Xray supports both manual and automated tests, including full support for Cucumber tests. It has built-in support for multiple test automation platforms, like JUnit, TestNG, NUnit, Robot framework, it also provides easy integration with CI platforms like Jenkins, Bamboo or TeamCity. For more information on Xray please refer to its official documentation: https://confluence.xpand-it.com/display/XRAY/Xray+Documentation+Home

Robot framework is a well-known test automation framework based on keyword-driven paradigm. The framework has a rich ecosystem around it consisting of various generic test libraries and tools. For more details on Robot framework, please refer to its official website: http://robotframework.org/.

Jenkins is the most popular open-source Continuous Integration (CI) server which provides hundreds of plugins that enable integration with many different systems and/or frameworks, including Xray and Robot framework.

This article is organized in three main sections: the first organization of test cases in Xray on a sample projects will be presented, then the necessary Jenkins configuration that provides execution of the Robot tests will be explained and finally the three different methods for importing results of Robot automated tests back in Xray/JIRA will be elaborated.

Sample JIRA project supported with Xray

Let’s assume that the system under test is the e-commerce site http://automationpractice.com and in Sprint 1 there are 5 user stories.

Sprint 1 user stories for the sample project

Figure 1: Sprint 1 user stories for the sample project

For each user story Tests in Xray are created. For example, for user story XR-32 three test cases in Xray are created:

Xray Tests associated with one of the user stories of the sample project

Figure 2: Xray Tests associated with one of the user stories of the sample project

Test Plan is Xray provided custom JIRA issue type that ensures better visibility and management over the Tests and can be considered as a “container” of Tests related to particular version of the system under test. Not going into details  on how Test Plan is created (for more details on creating Test Plan using Xray, please refer to: https://confluence.xpand-it.com/display/XRAY/Test+Plan), let’s assume that a Test Plan is created with all the tests from user stories of Sprint1:

Sample Xray Test Plan

Figure 3: Sample Xray Test Plan

Test Execution is another Xray provided custom issue type, which tracks multiple Test Runs, i.e. it is composed of multiple Test Runs. From the Test Plan, it’s possible to create one or more Test Executions for all or just a subset of the Tests contained within the Test Plan (e.g. only those Tests that are still failing). We are not going to go into details on how Test Execution is created (for more details on creating Test Execution using Xray, please refer to: https://confluence.xpand-it.com/display/XRAY/Test+Execution), for the sample project one Test Execution is created associated with Sprint 1 Test Plan.

Sample Xray Test Execution

Figure 4: Sample Xray Test Execution

Installing and configuring Xray Jenkins plugin

Xray provides a plugin for Jenkins that enables importing results from automated test execution back in JIRA. Xray Jenkins plugin can be downloaded from: https://confluence.xpand-it.com/download/attachments/35356981/xray-for-jira-connector.hpi?version=1&modificationDate=1532097682950&api=v2

In order to install Xray Jenkins plugin navigate to Manage Jenkins > Manage Plugins and select Advanced tab. In the Upload Plugin section, click upload and select the location where you have downloaded xray-for-jira-connector.hpi file.

Installing Xray Jenkins plugin

Figure 5: Installing Xray Jenkins plugin

Further, you need to configure Xray plug-in to communicate with JIRA. For that purpose, navigate to Manage Jenkins > Configure System > Xray for Jira configuration. Field Server Address is populated with the URL of the Jira Server where Xray is running, while Username and Password should be JIRA Administrator’s credentials:

Configuring communication between Jenkins and JIRA where Xray is running

Figure 6: Configuring communication between Jenkins and JIRA where Xray is running

Configuring Jenkins job for automated test execution and reporting back results to Xray-JIRA

The Jenkins job that will be created is a Freestyle project, named as Xray Robot Integration Demo:

Creating Jenkins job

Figure 7: Creating Jenkins job

For the purpose of this article, eight tests in Robot framework for already defined Tests in Xray were developed. In this case custom workspace will be used and the specified directory is the location where Robot tests are placed.

Configuring Jenkins job – specifying workspace

Figure 8: Configuring Jenkins job – specifying workspace

For executing automated tests Windows batch command can be used:

Configuring Jenkins job – Robot tests execution

Figure 9: Configuring Jenkins job – Robot tests execution

And finally, in order to transfer results from Robot tests execution to Xray, as “Xray: Results Import Task” that comes with the Xray Jenkins plugin, it needs to be defined as Post-build action.

Configuring Jenkins job – Post-build action

Figure 10: Configuring Jenkins job – Post-build action

After creating the Xray: Results Import Task, in the field JIRA Instance you can choose to which JIRA the server results will be pushed. Xray plugin allows to configure multiple JIRA servers under the Xray for JIRA configuration in Jenkins configuration. In this case only one JIRA server is configured, and the displayed JIRA instance is defined as Configuration alias (please refer to Figure 6). Format drop-down, lists reports from different testing frameworks that can be imported in Xray.

Xray: Results Import Task

Figure 11: Xray: Results Import Task

In our case  Robot XML format will be used, and import task needs to be configured with the following parameters:

  • Import to Same Test Execution – When this option is checked, if you are importing multiple execution report files using a glob expression, the results will be imported to the same Test Execution
  • Execution Report File (file path with file name) – location and name of the Robot Framework tests output file. Location should be specified as relative to the defined workspace
  • Project Key – key of the JIRA project. This is a required parameter, if it is not specified, test results will not be imported to JIRA
  • Test Execution Key – JIRA issue key of the Test Execution where results will be imported. If this parameter is not specified, then after importing results a new Test Execution will be created in JIRA
  • Test Plan Key – JIRA issue key of the Test Plan. If this parameter is not specified, then the imported Test Execution will be not associated with any Test Plan
  • Test Environments – list of Environments (separated by “,”) where tests are executed.

Xray: Results Import Task – configured for Robot framework

Figure 12: Xray: Results Import Task – configured for Robot framework

On the other side, configuration needed for Robot tests in order to be associated with a particular Test in Xray/JIRA is very straightforward, i.e. tag should be added to Robot test case that match key of the Test in JIRA. For example, in the figure test case bellow “Create Account” is tagged with XR-2, while “Place Order paid with bank wire” is tagged with XR-27, where XR-2 and XR-27 are keys of already created Tests in Xray/JIRA.

Associating Robot test cases with Test key in Xray/JIRA, using tags

Figure 13: Associating Robot test cases with Test key in Xray/JIRA, using tags

In the following text, three different ways of importing automated test results back in Xray/JIRA will be presented.

1. Running Jenkins job with defined Test Execution

Let’s examine how imported Robot test results, will look when in “Xray: Results Import Task”, Test Execution Key is defined. In order to illustrate this let’s assume that before Robot tests run there is an already created Test Execution in JIRA X-50, and import task is configured as on Figure 12.

After Jenkins job has finished, and Robot results are imported in JIRA automatically Test status is updated in the corresponding (in this case XR-50) Test Execution. Also for Test Environment label is set to value “QA”, as defined in the import task.

Test Execution created before automated test run and after importing Robot test results

Figure 14: Test Execution created before automated test run and after importing Robot test results

2. Running Jenkins job without defined Test Execution

Another option is automated Robot tests to be run from Jenkins without creating Test Execution in JIRA before. For that purpose the field Test Execution Key in “Xray: Results Import Task” should be left blank.

After Jenkins job has finished, if we navigate in JIRA to the associated Test Plan, we can see that an additional Test Execution (XR-51) is automatically added:

Test Plan with the associated two Test Executions

Figure 15: Test Plan with the associated two Test Executions

3. Importing tests and results from automated Robot test run, when Tests are not defined in Xray

Interesting feature of Xray is that Tests can be created in JIRA, based only on importing tests results from Robot automated test run. Consider that our tests are created only in Robot and there are no corresponding Tests created in Xray/JIRA.

In order Robot test cases to be mapped to particular User Story in JIRA, Robot test cases should be tagged with User Story which those are testing. For example in the image bellow, Robot test case “Create Account” verifies User Story XR-1, while Robot tests “Place Order paid with bank wire” and “Place Order paid with check” verify User Story XR-26.

Associating Robot test cases with User Story key in JIRA, using tags

Figure 16: Associating Robot test cases with User Story key in JIRA, using tags

In this case, “minimal” configuration for the import task in Post-build action in Jenkins job will be used, i.e.only JIRA project key and environment will be configured, which is optional.

Xray: Results import Task - minimal configuration when Tests are not created in Xray/JIRA before automated test run

Figure 17: Xray: Results import Task – minimal configuration when Tests are not created in Xray/JIRA before automated test run

After tests are executed by Jenkins, new Tests are created, that are automatically associated with User Stories, based on the tags that were added in Robot test cases. For example, in a case of a user story XR-32, three Tests are automatically associated from executed Robot test cases.

"Automatically" created and associated Tests after Robot tests execution

Figure 18: “Automatically” created and associated Tests after Robot tests execution

Also, after tests run, new Test Execution is created with all executed Robot tests. (Please note, that Test Type in this case is “Generic”. Xray supports three different Test Types: “Generic” – supports automated agnostic test that is executed by an external tool, “Cucumber” – enables writing Tests in Gherkin language and “Manual” – enables writing tests in standard Step Description/Expected Result format for manual execution)

Created Test Execution after Robot tests run

Figure 19: Created Test Execution after Robot tests run

From Test Execution, it can be navigated to the Execution Details for each of the Tests. For example, if we navigate to Execution Details of the Test XR-53, we can see the test status of each of the keyword contained in executed Robot test.

Execution details with keywords from the executed Robot test

Figure 20: Execution details with keywords from the executed Robot test

Comparing the above mentioned three methods for importing automated tests results, it looks that the most applicable is the method 2, because it automatically creates Test Execution in Xray/JIRA upon execution of the automated tests and associate results with corresponding Test Plan. On the other side, comparing method 2 with method 3, the drawback of method 3 is that it doesn’t create any Test Plan associated with the automated test run. The main advantage of method 3 – automatic creation of Tests in XRay/JIRA, in most situations will not be applicable, because most probably there will be already defined Tests for manual testing, before implementation and execution of automated tests.

Conclusion

Xray test management plugin for JIRA,  provides all necessary artifacts for entire testing life-cycle in a case of manual testing, it provides strong support for automated testing with its support of importing results from different test automation frameworks, like Robot testing framework. This possibility, avoids the need of “manual” importing results from automated tests. All in all, Xray simplifies the entire test management, monitoring and reporting, because it provides one central point – JIRA where all relevant information of the testing for a particular project can be found.