Selecting the right plugin
There are mainly two different kinds of test cases: manual and automated test cases. The former require a human intervention to decide if the test was successful or not while the latter just requires a script that executes the tests and provides a return code that will be interpreted by checkbox. Following the unix convention, the script will return 0 on success and any other code on failure.
Manual test cases are implemented by the manual plugin, while automated test cases are implemented by the shell plugin. So when adding a test case that needs someone to decide if the output was correct or not, use the manual plugin; and, when the test is fully automated, use the shell plugin and checkbox will take care of collecting the results for you.
Please note that checkbox executes manual tests before automated tests so don't be afraid to mix both kind of test cases in the same test suite (i.e. the test file in which a set of test cases have been written) since human intervention will be needed only at the very begining and, after that, the device under test may be left unattended while automated test cases are executed.
name: manual-test plugin: manual description: Ask the user to follow some procedure and mark if the output was the expected one
name: automated-test plugin: shell command: some-script description: This description won't be displayed to the user
Using command output
Not only automated tests require a command line to be added so that checkbox knows how to run the test case, but also manual tests could benefit of such a command line to provide dynamic information in the test description. To take advantage of this functionality, just make sure that the command writes useful information to the standard output and add the $output placeholder in the description. checkbox will take care of capturing the output and displaying it properly in the description. Please note that the output of the command is also captured for the automated test cases. However, since the output cannot be displayed to the user, it's stored in the report so that it can be checked later not only the return code of the command but also any other information that could have been added using the standard output.
When manual test case command is run, the interface provides a Test option to repeat the command execution in case the conditions of the test cases changed and the user wants to test again to make user about the result. In case the command is to be executed only once, there is a way to disable the Test option. Please refer to this tip for more information