Integrating Xray with JMeter functional tests

This article describes an approach for importing results from JMeter functional tests in JIRA based test management tool – Xray by utilizing Jenkins CI platform.

Introduction:

Xray is a popular test management tool add-on for JIRA. Xray supports manual and automated tests and a complete testing lifecycle: test planning, test designing and test execution. Xray has native support for Cucumber tests and also support for importing the test results from other testing frameworks like Junit, TestNG, NUnit, Robot Framework, Behave, Calabash and etc. However, Xray doesn’t have built-in support for importing JTL files that contains test results generated by JMeter, and as a solution for this we are proposing converting JTL files in JUnit format as part of the ANT script that executes JMeter functional tests.

Ant script for JMeter tests run

 

In order to execute JMeter tests from Jenkins in our example, we will use ANT script. Below a used ANT script  is described.

Define project with destination directory and set global property “jmeter.home” with the path to the JMeter installation folder:

Define class path with the location of the “ant-jmeter-1.1.1.jar”:

Create jmeter ant task that will point to the class “JMeterTask”:

Clean the directories with the results and the log file and create a new directories for storing the new results:

Execute single or multiple JMeter tests by using the “<testplans>” tag, set the format and the output directory for the results. You can also specify JVM arguments to the JVM launched to run Jmeter.

Generate junit-xml report from the JTL results file by using XSLT transformation.

Because Xray has no built-in support for JMeter reports in .jtl format, the results need to be converted to “junit-xml” format with XSLT in “junit.xsl” file:

Xray configuration

 

For the purpose of this demonstration, let’s consider that in Xray a Test Set named as “Create Order” that contains Xray Tests of type Generic is created . For more info on specific issue types introduced by Xray, please refer to: https://confluence.xpand-it.com/display/XRAY/Terms+and+Concepts).

Actually, each of the Tests in this Test Set corresponds to JMeter implemented http requests that simulates order creation in e-commerce site. Please note that Xray Tests should have the same “Summary” as the name of the http requests from the JMeter script.

 

Further, from this Test Set in our configuration, we will create Xray Test Execution that will keep results of particular JMeter tests run:

Jenkins job setup

 

For importing JMeter results from Jenkins, Xray Jenkins plugin needs to be installed. For more details on installing and configuring Xray Jenkins plugin, please refer to: https://confluence.xpand-it.com/display/XRAY/Integration+with+Jenkins.

 

In our Jenkins job, the first step will be execution of the Ant script described in the first section “Ant script for JMeter tests run”:

In order to import JMeter results in Xray add “Xray: Results Import Task” in post-build actions step.  The fields that need to be populated are described below.

  • JIRA Instance – reference to the JIRA server configured with Xray, as it is setup in Manage Jenkins > Configure System > Xray for Jira configuration
  • Format – select JUnit XML
  • Execution Report (file path with file name) – Add relative path to the JUnit .xml test results that is generated with the ant script
  • Project Key – Add key to the JIRA project.
  • Test Execution Key – Add “Test Execution” issue number.

After Jenkins job is executed, results from JMeter tests run is “automatically” transferred to previously created Test Execution in Xray: