Ubuntu Wiki

Firmware Test Suite (fwts)

The tool fwts comprises of over fifty tests that are designed to exercise and test different aspects of a machine's firmware. Many of these tests need super user access to read BIOS data and ACPI tables, so the tool requires running with super user privileges (e.g. with sudo).

Running fwts with no options will run through all the batch tests that require no user interaction and by default append test results into a log file called "results.log". However, if require, one can specify specific tests to run rather than the default set.

There are 6 classes of tests in fwts:

• Batch
  • These run automatically by default and require no user intervention. However, one can select these with the -b or --batch option.
• Batch Experimental
  • This tests require no user intervention and contain code that may give false positive results - much akin to beta quality code. One can select these tests with the --batch-experimental option.
• Interactive
  • These tests require user interaction, such as pressing keys, closing laptop lids, disconnecting/connecting power supplies etc. Selected with the --interactive option.
• Interactive Experimental
  • These are beta quality interactive tests. Selected with the --interactive-experimental option.
• Power Management
  • These run suspend (S3) and hibernate (S4) power management tests and may hang the machine, hence they are not in the default batch test category. Select these tests using -P or --power-states options.
• Utilities
  • These are not strictly tests, but will gather system information useful for kernel developers to debug a system. So if a test fails, it is useful to run these and append the data to the results log to help diagnose and debug issues with the firmware. These tests can be selected with the -u or --utils options.

One can mix and match these tests to run the different classes of tests in one go, e.g.:

sudo fwts --batch --interactive --power-states

To run just one or a few tests rather than the default set of batch tests, one can specify the test name too. To see the names of the available tests use the --show-tests or --show-tests-full options. Currently, with version V15.11.00 there following tests are available:

ACPI tests

Batch tests

Batch Experimental tests

EBBR tests

Interactive tests

Power States tests

SBBR tests

Utilities

Unsafe tests

UEFI tests

ACPI Spec Compliance tests

One can specify one or more of these tests, for example:

sudo fwts klog checksum wakealarm s3

..and even mix and match with classes of tests, e.g.

sudo fwts klog --power-states --batch-experimental

One can run exclude tests from being run with the -S and --skip-test options. List the names of the tests in a comma separated list and fwts will skip over and ignore tests tests. For example

sudo fwts --batch --skip-test=nx,oops,smbios

or

sudo fwts --interactive -S lid,power_button

Finally, the -a and --all options allow one to run *all* the tests.

Getting Help

fwts has two methods of getting help. Use -?, -h or --help options to display a short summary of available options. Alternatively, a manual is availble using:

man fwts

You can also view the manual page here.

S3 and S4 Test Specific Options

While most tests have default modes of operation that cater for most use cases, some tests can be driven in different modes to perform intensive soak testing. The S3 and S4 tests have a range of options to allow one to do multiple repeats and to configure the delay times when awake and the sleep times when suspended or hibernated. There are also options to pass quirks to the pm-suspend and pm-hibernate scripts that initiate the suspend/hibernates. The options are as follows:

• --s3-quirks
  • Comma separated list of quirk arguments to pass to pm-suspend.
• --s4-quirks
  • Comma separated list of quirk arguments to pass to pm-hibernate.

for example:

sudo fwts s3 --s3-quirks=--quirk-s3-bios,--quirk-dpms-suspend

see man pm-utils for a list of available quirks.

• --s3-multiple=N
  • Run S3 tests multiple times, e.g. --s3-multiple=250
• --s4-multiple=N
  • Run S3 tests multiple times, e.g. --s4-multiple=30

By default the multiple tests is set to 1, i.e. just one cycle. These options are designed for repeated S3/S4 soak testing.

• --s3-sleep-delay=N
  • Sleep N seconds between start of suspend and wakeup, e.g. --s3-sleep-delay=30
• --s4-sleep-delay=N
  • Sleep N seconds between start of hibernate and wakeup, e.g. --s4-sleep-delay=60

By default, fwts allows plenty of time for a suspend and a hibernate to occur before triggering a wakeup alarm to come out of suspend/hibernation. However, one can tune the sleep delay to match the suspend or hibernate time on one's hardware. Note that this time is measured from the start of the suspend/hibernate to the time that the wake alarm triggers. In retrospect, these options should have been called "wakeup-delay".

• --s3-device-check
  • Check differences between device configurations over a suspend cycle.
• --s4-device-check
  • Check differences between device configurations over a hibernate cycle.

These options enable a simple device status check. Device status/configuration for devices such as Bluetooth, Wireless, Ethernet, etc are gathered before the suspend/hibernate and then checked against the status/configuration on wakeup to see if any devices fail to re-configure. Because some devices take a while to get back to their original state (such as wifi that need to re-association), this tests waits 15 seconds after wakeup before doing the status/configuration check. This means these options cannot be used with the --s3-min-delay, --s3-max-delay and --s3-delay-delta options.

Note we add a default of 15 seconds to allow wifi to re-associate. Cannot be used with --s3-min-delay, --s3-max-delay and --s3-delay-delta.

The final options controls a delay time between waking up and going into the next suspend/hibernate. One can specify a min/max range of times to delay and an increment that is added to each cycle up to the max time. Then the delay wraps back to the min time and restarts again. The main use for this is to add some variable delay which may trip bugs in drivers that have timing bugs, for example, Wifi drivers may oops when being suspend while they are re-associating. Hence selecting appropriate min/max times can sometimes trigger race conditions in drivers when going into suspend/hibernate. The options are as follows:

• --s3-min-delay=N
  • Minimum time in seconds between suspend iterations, e.g. --s3-min-delay=10
• --s4-min-delay=N
  • Minimum time in seconds between hibernate iterations, e.g. --s4-min-delay=10
• --s3-max-delay=N
  • Maximum time in seconds between suspend iterations, e.g. --s3-max-delay=20
• --s4-max-delay=N
  • Maximum time in seconds between hibernate iterations, e.g. --s4-max-delay=20
• --s3-delay-delta=N
  • Time to be added to delay between suspend iterations. Can be a fraction of a second.
• --s4-delay-delta=N
  • Time to be added to delay between hibernate iterations. Can be a fraction of a second.

These options cannot be run with the --s3-device-check/--s4-device-check options.

Example:

sudo fwts s4 --s3-min-delay=1 --s4-max-delay=20 --s4-delay-delta=0.25

gives a delay ranging from 1..20 seconds, and adds 1/4 of a second per iteration to the delay.

Logging

fwts has a fairly complex logging mechanism designed to capture and summarise test pass/fail status messages. Arguably it may gather too much data, but the underling principles behind the design are as follows:

• To log test status, e.g. Pass, Fail, Skipped (not applicable on the hardware), Aborted (due to some error), Info (such as data from utilities)
• To capture error conditions to allow a developer to diagnose failures or warnings
• To provide contextual advice feedback. These lines are prefixed by "ADVICE" tags.

By default, fwts will append it's output to the log file "results.log". The -f and --force-clean options will start the log from fresh, clearing an previous log messages. One can redirect the log data to a different file using the -r and --results-output. If the output file name is stderr or stdout then the output will be directed to these file descriptors. Finally, the - option will redirect the output to stdout.

Examples:

sudo fwts -f -r mylog.log

..write a new log file to mylog.log, clearing out any previous results in the log if it exists already.

sudo fwts --results-output=stderr

..write output to stderr

sudo fwts -

..synonymous with --results-output=stdout or -r stdout, writes log to stdout. This can be piped or redirected, e.g.

sudo fwts - | tee mylog.log

Log formatting controls

By default, the log file displays several columns on information:

line-num test-name log-data

where line-num is a 5 digit line number, test-name is the name of the test being run, and log-data is the output from the test. One can re-configure the log output using the --log-format option. The --log-formation option allows one to specify which fields you require and the order and format of each logged line of output. The available --log-format field operators are as follows:

• %date - date
• %time - time
• %field - log-filter fields
• %owner - name of the test routine
• %level - test failure level
• %line - log line

By default, fwts will format each line based on the --log-format specifiers and append the test output after this formatted text. The best way to describe this is by examples:

sudo fwts --log-format="%date %time %line %owner: " 

An excerpt of the output from this format is:

22/01/11 12:56:06 00297 hpet_check     : HPET configuration test.
22/01/11 12:56:06 00298 hpet_check     : -------------------------------------------------------------------
22/01/11 12:56:06 00299 hpet_check     : Must be run as root or sudo to be able to read system information.
22/01/11 12:56:06 00300 hpet_check     : Aborted test, initialisation failed.
22/01/11 12:56:06 00301 hpet_check     : ===================================================================
22/01/11 12:56:06 00302 hpet_check     : 0 passed, 0 failed, 0 warnings, 3 aborted, 0 skipped, 0 info only.
22/01/11 12:56:06 00303 hpet_check     : ===================================================================

One can order the fields in anyway you require. One can even specify an empty format, which will then just output the text generated from the test an no preceding format field information, e.g:

sudo fwts --log-format=

An excerpt of the output from this format is:

HPET configuration test.
--------------------------------------------------------------------
Must be run as root or sudo to be able to read system information.
Aborted test, initialisation failed.
====================================================================
0 passed, 0 failed, 0 warnings, 3 aborted, 0 skipped, 0 info only.
====================================================================

One particular format field of interest is the %field option. This shows the way each log output is classified into different message types. One can filter messages in the log based on these field types, so for example, one just gets the results and summary of the tests to just focus on the test pass/fail data. Log filtering is activated using the --logfilter option. This allows one to specify the fields that you want or don't want. To see the available fields, use the --log-fields option, e.g. :

fwts --log-fields
Available fields: RES,ERR,WRN,DBG,INF,SUM,SEP,NLN,ADV,HED,TAG

One can specify a comma separated list of fields that one requires. Prefixing a field name with ~ disables that field.

the fields are as follows:

• RES - results, end of test pass/fail/warning/aborted/skipped/info test results
• ERR - report any errors when trying to run a test (for example, resources not available, having to run test with sudo, etc)
• WRN - report any warnings, e.g. a test may have picked up a possible problem, but it's not a failure and the test was able to complete.
• DBG - deprecated
• INF - information feedback from the test. This reports data that can be read by a developer to help diagnose issues if a test fails.
• SUM - summary data - reports collated summary of tests results at end of test execution
• SEP - separator fields between tests. One can use ~SEP to disable these.
• NLN - new line fields. One can use ~SEP to disable these.
• ADV - advice output. Some tests can diagnose problems and give some advice to help with workarounds or deeper explanations of why
  • an error has been detected.
• HED - headings.
• TAG - bug tags (see --tag options later).

Examples:

To just get the results and a summary report at the end of the tests, use:

sudo fwts --log-filter=RES,SUM

To get results and extra test information and help advice, use:

sudo fwts --log-filter=INF,RES,ADV

The default options give plenty of data, so the filtering can be useful to drill down on the fields you require. One can mix and match log filtering and log format controls to configure a log that matches ones taste.

Log Width

If the log is being written to a log file, it is assumed to write a log that is 120 characters wide. If writing to a terminal, then the log width is adjusted to the width of the terminal. To override the log with, use the -W or --log-width options. Widths under 80 characters are not allowed.

Examples:

sudo fwts -W 100

or

sudo fwts --log-width=90

Progress Indicators

When fwts is writing results to a log file it will also display a test by test progress summary to stderr. If the test takes a while to run the progress will be updated showing a percentage progress, which is a rough estimation of progress in that particular test. Some tests may run slower as the end completion, so bear in mind that the progress indication is not accurate.

If you want fwts to run silently, the use the -q or --quiet options.

The --stdout-summary option will dump a simple PASSED/FAILED message at the end of the completion of every test.

Progress can be piped into the dialog tool for a more appealing output. Use the -D or --show-progress-dialog option to do this:

sudo fwts -D | dialog --gauge "Firmware Test Suite" 10 80

Extra Tools

Fwts is a tool designed primarily for testing. However, it includes some helpful tools to dump firmware state which can be then forwarded to a developer to help debug and diagnose firmware issues.

The ACPI DSDT and SSDT tables contains ACPI Machine Language code which can sometimes be buggy. The --disassemble-aml option will find these tables and disassemble without the need of the usual acpidump, acpixtract and iasl tools. To run, use:

sudo fwts --disassemble-aml

This will produce files DSDT.dsl and SSDTx.dsl files (where x is the Nth SSDT table).

Another feature is the -d, --dump option. This will dump out several files:

• README.txt - containing an explanation of the files that have been dumped

• dmesg.log - output from the kernel message log (like running dmesg > dmesg.log)

• dmidecode.log - SMBIOS DMI tables, (output from dmidecode)
• lspci.log - PCI configuration data (output from lspci -vvnn)
• acpidump.log - ACPI tables in a format identical to that of acpidump
• cpuinfo.log - CPU configuration from /proc/cpuinfo

To gather this data, fwts needs to be run with root privilege, e.g.

sudo fwts --dump

Processing pre-dumped data

Some tests can take previously dumped out raw data from the firmware or kernel logs and process them on a different machine. This is useful if a user finds a problem and can then email the data to a developer who can then process this data with fwts to run some tests. The -k and --klog options allow one to specify a previously dumped log file and process them with the following tests:

• klog
  • Scan kernel log for general firmware errors
• acpiinfo
  • Scan kernel log for ACPI driver errors
• dmesg_common
  • Scan kernel log for common kernel log errors
• oops
  • Scan kernel log for oops messages

for example, using a copy of dmesg.log output from dmesg or fwts --dump, scan for bugs:

fwts klog acpiinfo dmesg_common oops --klog=dmesg.log

Note you don't need to run with sudo since fwts only needs to read log data.

Several tests can read dumped ACPI tables and scan these for possible bugs using output from sudo fwts --dump. The tests that can process this data are as follows:

• acpitables
  • Sanity check configuration in ACPI tables
• apicinstance
  • Check for just one instance of MADT
• checksum
  • Check ACPI table checksums
• method
  • Check ACPI DSDT and SSDT methods and objects
• osilinux
  • Check for _OSI("Linux") in DSDT
• syntaxcheck
  • Re-assemble DSDT and SSDT and find syntax errors and warnings.
• acpidump
  • Dump and annotate ACPI tables

These tests read in the ACPI data using the --dumpfile option. Suppose you have a log file called "acpidump.log" dumped using fwts --dump, you can run the above tests on it using:

fwts --dumpfile=acpidump.log acpitables apicinstance checksum method osilinux syntaxcheck acpidump

Miscellaneous Options

Fwts uses lspci and the default path to this can be overridden using --lspci, for example:

sudo fwts --lspci=/usr/bin/lspci

Fwts has a couple of databases used in scanning logs and by default these are located in /usr/share/fwts/ - however, this path can be overridden using the -j and --json-data-path options:

sudo fwts ---json-data-path=/usr/share/fwts

The version and build date of fwts can be dumped out using the -v and --version options:

fwts -v

Which Test to Run?