UTAH Compatible Test Cases

Any binary that can run on Ubuntu, can be considered a Test Case as far as UTAH is concerned. If you add the right control files and structure your test suite in directories neatly, UTAH will run whatever you want. This is both good and bad, because it means we are counting on self-discipline to develop new test cases to a good standard, even if legacy test cases are not as good as they could be.

Directory Structure

This is what a UTAH test suite looks like:

.
├── testsuite1
│   ├── t1
│   │   ├── Makefile
│   │   ├── t1.c
│   │   └── tc_control
│   ├── t2
│   │   ├── test.py
│   │   └── tc_control
│   ├── ts_control
│   └── tslist.run
└── testsuite2
└── master.run           # OPTIONAL - Could be located elsewhere

Runlists Hierarchy

A runlist is a collection of test cases that we want to run in one go. The syntax for runlists is:

master.run file contains:

timeout: 123       # OPTIONAL - overrides timeout for every test case included by this runlist
testsuites:
  - name: testsuite1
    fetch_cmd : git clone/pull wherever
    include_tests:               # OPTIONAL
      - t1
      - t2
      - t3...
  - include: master.run.amd64.   # OPTIONAL - this runlist will be included

  - name: security/testsuite2
    fetch_cmd : bzr branch/pull wherever
    exclude_tests:               # OPTIONAL
      - st4
      - st5

Each test suite has a default runlist and a control file that contain as follows:

tslist.run file contains:

- test: t1                             # directory
  command: python test1.py a1 a2 a3    # OPTIONAL - command to run the test from within t1
  run_as: nobody                       # OPTIONAL - user that runs the test

- test: t2                             # directory/binary
  command: test2.sh                    # OPTIONAL
  overrides:                           # OPTIONAL (Subject to change) array of control properties to override
  - timeout: 200                       # timeout in seconds

ts_control file contains:

build_cmd:      make
timeout:        300
ts_setup:       tsetup/setup.sh      # test case t1 is the set up for
                                     # the whole test suite
ts_cleanup:     tcleanup/cleanup.sh  # clean up test case

To avoid excessive typing and repetition in runlists and control files the following rules apply:

• tslist.run's test is relative to the test suite. So if the suite is called sample_tests and the entry in tslist.run for test is test_one then the path to the test folder will be sample_tests/test_one.

• tslist.run's command is relative to the testcase’s folder. So the above test with a command of python test_one.py will result in this effectively:

      cd sample_tests/test_one && python test_one.py

• ts_control is optional.

• if build_cmd, or ts_setup fails no tests in the suite will be run.

• if build_cmd, or tc_setup fails the test will not be run.

• ts_cleanup will be run whether or not ts_setup, build_cmd, or any tests fail.

• tc_cleanup will be run whether or not tc_setup, build_cmd, or the tesjk fail.

Each test case, with all the code required for it to compile or run, lives in a folder, and contains also a control file that includes its documentation amongst other things:

tc_control file contains:

build_cmd:      make    # OPTIONAL - or scons or build.sh
description:
( tc_id: TC-INSTALL-ALTERNATE-01)
 dependencies:
 actions: |
   1. Action 1
   2. Action 2
 expected_results: |
   1. Expected result 1
   2. Expected result 2
type:           build_only/userland/kernel (?)
command:        python test1.py
timeout:        100
tc_setup:       t1/t1 setup    # OPTIONAL - in case the test case has a setup
tc_cleanup:     t1/t1 cleanup  # OPTIONAL -in case the test has a cleanup

For each test case that is a binary or a script, we will give them the option to define a setup and a cleanup function if they wish so, and it is the test code developer’s responsibility to add the right parameters to their code so that the harness can run their setup and clean up functions.

In terms of settings precedence:

Test case control files shall contain the default parameters the test case author deems necessary to run the test case. Test suite control files can override values such as timeout and command to meet other needs. The master.run file has the highest precedence and can override these values to meet the runlist author's needs.