JenkinsTestResultsInXML

Differences between revisions 3 and 4
Revision 3 as of 2012-01-10 15:50:40
Size: 3699
Editor: brad-figg
Comment:
Revision 4 as of 2012-01-10 15:53:15
Size: 3741
Editor: brad-figg
Comment:
Deletions are marked like this. Additions are marked like this.
Line 34: Line 34:
{{{
Line 37: Line 38:
}}}
Line 43: Line 45:
{{{
Line 44: Line 47:
}}}
Line 46: Line 50:
{{{
Line 49: Line 54:
}}}
Line 54: Line 60:
{{{
Line 56: Line 63:
}}}
Line 65: Line 73:
* output1.xml - from a run of xfstests under autotest, 66 test cases, all succeeded  * output1.xml - from a run of xfstests under autotest, 66 test cases, all succeeded
Line 67: Line 75:
* output2.xml - from a run of xfstests under autotest, 2 test cases, one failed  * output2.xml - from a run of xfstests under autotest, 2 test cases, one failed

This document and code samples are concerned with how to get your test results into a format which can easily be digested by Jenkins.

The source and example files referenced can be found here

The Problem

Although Jenkins will display a simple pass/fail for a Jenkins job, this is usually not sufficient, because:

  • a set of tests in a job may return success if it completed, even when individual tests may have failed
  • the Jenkins results page for a job will not show the status of individual tests

One way to solve this is to get the test results into a format which Jenkins understands. The JUnit xml format is a good choice for this.

The JUnit test xml output format

The specification in .xsd format: http://windyroad.org/dl/Open%20Source/JUnit.xsd

More information: http://www.junit.org/node/399

In this context, "Error" means an error completing the test or job, and "Failure" means a test which completes but fails.

My implementation

I used the generateDS.py utility to generate a python wrapper API for the xml data, directly from the .xsd file. This file is named JUnit_api.py, and is used by the examples I've provided.

To process test results when there are multiple sub-tests within a job, you have to parse whatever the test output is and collect data in the following areas:

Properties

Properties apply to the entire test job, and can include things like the test command line, system configuration, installed package list, or anything else which has been collected which is of interest.

In the xml file, these appear like this (for example):

<property name="version" value="Linux version 3.0.0-14-generic-pae (buildd@palmer) (gcc version 4.6.1 (Ubuntu
/Linaro 4.6.1-9ubuntu3) ) #23-Ubuntu SMP Mon Nov 21 22:07:10 UTC 2011
"/>

Test Cases

Each test is reported as a test case. Each test case contains information If a test has an error or failure, then it has additional information about that.

An example of a successful test case:

<testcase classname="autotest" name="xfstests.228.ext4" time="30.000000"/>

An example of a test case failure:

<testcase classname="autotest" name="xfstests.241.ext4" time="24.000000">
    <failure message="Test xfstests.241.ext4 is Not Applicable" type="Failure">dbench not found</failure>
</testcase>

Summary Information:

The testsuite item in the xml file contains summary information, which typically has to be gathered and accumulated while parsing all the test cases. The summary includes the number of tests, number of errors and failures, time taken by the test, etc.

<testsuite tests="2" errors="0" name="Autotest tests" timestamp="2012-01-10" hostname="Ivy
" time="157.000000" failures="1">

Implementation

The JUnit_api.py file implements objects which encapsulate the xml file format, making it relatively easy to create objects and set properties, then export the entire xml file. The hardest part is parsing the results of your tests.

I've provided two examples. One (ltp2JUnit) was done for the ltp test suite, and parses a long log from stdout. The other (results2junit.py) was done for autotest, and reads a bunch of individual files containing data gathered for the test (which become properties), and files for each test case. These are fairly convoluted examples because of the complexity of the results presented by the tests.

Two output xml files are provided:

  • output1.xml - from a run of xfstests under autotest, 66 test cases, all succeeded
  • output2.xml - from a run of xfstests under autotest, 2 test cases, one failed

SteveConklin/JenkinsTestResultsInXML (last edited 2012-01-19 12:00:39 by 85-210-149-110)