Reference

Revision 24 as of 2011-08-17 13:29:12

Clear message

Firmware Test Suite (fwts)

The tool fwts comprises of over thirty tests that are designed to exercise and test different aspects of a machine's firmware. Many of these tests need super user access to read BIOS data and ACPI tables, so the tool requires running with super user privileges (e.g. with sudo).

Running fwts with no options will run through all the batch tests that require no user interaction and by default append test results into a log file called "results.log". However, if require, one can specify specific tests to run rather than the default set.

There are 6 classes of tests in fwts:

  • Batch
    • These run automatically by default and require no user intervention. However, one can select these with the -b or --batch option.
  • Batch Experimental
    • This tests require no user intervention and contain code that may give false positive results - much akin to beta quality code. One can select these tests with the --batch-experimental option.
  • Interactive
    • These tests require user interaction, such as pressing keys, closing laptop lids, disconnecting/connecting power supplies etc. Selected with the --interactive option.
  • Interactive Experimental
    • These are beta quality interactive tests. Selected with the --interactive-experimental option.
  • Power Management
    • These run suspend (S3) and hibernate (S4) power management tests and may hang the machine, hence they are not in the default batch test category. Select these tests using -P or --power-states options.
  • Utilities
    • These are not strictly tests, but will gather system information useful for kernel developers to debug a system. So if a test fails, it is useful to run these and append the data to the results log to help diagnose and debug issues with the firmware. These tests can be selected with the -u or --utils options.

One can mix and match these tests to run the different classes of tests in one go, e.g.:

sudo fwts --batch --interactive --power-states

To run just one or a few tests rather than the default set of batch tests, one can specify the test name too. To see the names of the available tests use the --show-tests or --show-tests-full options. Currently, with version V0.22.00 there following tests are available:

Batch tests:

acpiinfo

General ACPI information check.

acpitables

ACPI table settings sanity checks.

apicedge

APIC Edge/Level Check.

apicinstance

Check for single instance of APIC/MADT table.

bios32

Check BIOS32 Service Directory.

bios_info

Gather BIOS DMI information.

checksum

Check ACPI table checksum.

crs

Check PCI host bridge configuration using _CRS.

dmesg_common

General dmesg common errors check.

dmi_decode

Test DMI/SMBIOS tables for errors.

ebda

Validate EBDA region is mapped and reserved in memory map table.

fadt

FADT SCI_EN enabled check.

fan

Simple Fan Tests.

hda_audio

Check HDA Audio Pin Configs.

hpet_check

HPET configuration test.

klog

Scan kernel log for errors and warnings.

maxfreq

Check max CPU frequencies against max scaling frequency.

maxreadreq

Checks firmware has set PCI Express MaxReadReq to a higher value on non-motherboard devices.

mcfg

MCFG PCI Express* memory mapped config space.

method

ACPI DSDT Method Semantic Tests.

microcode

Check if system is using latest microcode.

mpcheck

Check Multi Processor tables.

msr

CPU MSR consistency check.

mtrr

MTRR validation.

nx

Test if CPU NX is disabled by the BIOS.

oops

Scan kernel log for Oopses.

os2gap

OS/2 memory hole test.

osilinux

Disassemble DSDT to check for _OSI("Linux").

smbios

Check SMBIOS.

syntaxcheck

Re-assemble DSDT and find syntax errors and warnings.

thermal_trip

Test ACPI passive thermal trip points.

version

Gather kernel system information.

virt

Test CPU Virtualisation Configuration.

wakealarm

Test ACPI Wakealarm.

wmi

Extract and analyse Windows Management Instrumentation (WMI).

Interactive tests:

ac_adapter

Interactive ac_adapter power test.

battery

Battery Tests.

brightness

Interactive LCD brightness test.

hotkey

Hotkey scan code tests.

lid

Interactive lid button test.

power_button

Interactive power_button button test.

Batch Experimental tests:

cpufreq

CPU frequency scaling tests (takes ~1-2 mins).

cstates

Check processor C state support.

dmar

Check sane DMA Remapping (VT-d).

Power States tests:

s3

S3 suspend/resume test.

s3power

S3 power loss during suspend test.

s4

S4 hibernate/resume test.

Utilities:

acpidump

Check ACPI table acpidump.

cmosdump

Dump CMOS Memory.

ebdadump

Dump Extended BIOS Data Area.

memmapdump

Dump system memory map.

mpdump

Dump Multi Processor tables.

romdump

Dump ROM data.

uefidump

Dump UEFI variables.

One can specify one or more of these tests, for example:

sudo fwts klog checksum wakealarm s3

..and even mix and match with classes of tests, e.g.

sudo fwts klog --power-states --batch-experimental

One can run exclude tests from being run with the -S and --skip-test options. List the names of the tests in a comma separated list and fwts will skip over and ignore tests tests. For example

sudo fwts --batch --skip-test=nx,oops,smbios

or

sudo fwts --interactive -S lid,power_button

Finally, the -a and --all options allow one to run *all* the tests.

Getting Help

fwts has two methods of getting help. Use -?, -h or --help options to display a short summary of available options. Alternatively, a manual is availble using:

man fwts

S3 and S4 Test Specific Options

While most tests have default modes of operation that cater for most use cases, some tests can be driven in different modes to perform intensive soak testing. The S3 and S4 tests have a range of options to allow one to do multiple repeats and to configure the delay times when awake and the sleep times when suspended or hibernated. There are also options to pass quirks to the pm-suspend and pm-hibernate scripts that initiate the suspend/hibernates. The options are as follows:

  • --s3-quirks
    • Comma separated list of quirk arguments to pass to pm-suspend.
  • --s4-quirks
    • Comma separated list of quirk arguments to pass to pm-hibernate.

for example:

sudo fwts s3 --s3-quirks=--quirk-s3-bios,--quirk-dpms-suspend

see man pm-utils for a list of available quirks.

  • --s3-multiple=N
    • Run S3 tests multiple times, e.g. --s3-multiple=250
  • --s4-multiple=N
    • Run S3 tests multiple times, e.g. --s4-multiple=30

By default the multiple tests is set to 1, i.e. just one cycle. These options are designed for repeated S3/S4 soak testing.

  • --s3-sleep-delay=N
    • Sleep N seconds between start of suspend and wakeup, e.g. --s3-sleep-delay=30
  • --s4-sleep-delay=N
    • Sleep N seconds between start of hibernate and wakeup, e.g. --s4-sleep-delay=60

By default, fwts allows plenty of time for a suspend and a hibernate to occur before triggering a wakeup alarm to come out of suspend/hibernation. However, one can tune the sleep delay to match the suspend or hibernate time on one's hardware. Note that this time is measured from the start of the suspend/hibernate to the time that the wake alarm triggers. In retrospect, these options should have been called "wakeup-delay".

  • --s3-device-check
    • Check differences between device configurations over a suspend cycle.
  • --s4-device-check
    • Check differences between device configurations over a hibernate cycle.

These options enable a simple device status check. Device status/configuration for devices such as Bluetooth, Wireless, Ethernet, etc are gathered before the suspend/hibernate and then checked against the status/configuration on wakeup to see if any devices fail to re-configure. Because some devices take a while to get back to their original state (such as wifi that need to re-association), this tests waits 15 seconds after wakeup before doing the status/configuration check. This means these options cannot be used with the --s3-min-delay, --s3-max-delay and --s3-delay-delta options.

Note we add a default of 15 seconds to allow wifi to re-associate. Cannot be used with --s3-min-delay, --s3-max-delay and --s3-delay-delta.

The final options controls a delay time between waking up and going into the next suspend/hibernate. One can specify a min/max range of times to delay and an increment that is added to each cycle up to the max time. Then the delay wraps back to the min time and restarts again. The main use for this is to add some variable delay which may trip bugs in drivers that have timing bugs, for example, Wifi drivers may oops when being suspend while they are re-associating. Hence selecting appropriate min/max times can sometimes trigger race conditions in drivers when going into suspend/hibernate. The options are as follows:

  • --s3-min-delay=N
    • Minimum time in seconds between suspend iterations, e.g. --s3-min-delay=10
  • --s4-min-delay=N
    • Minimum time in seconds between hibernate iterations, e.g. --s4-min-delay=10
  • --s3-max-delay=N
    • Maximum time in seconds between suspend iterations, e.g. --s3-max-delay=20
  • --s4-max-delay=N
    • Maximum time in seconds between hibernate iterations, e.g. --s4-max-delay=20
  • --s3-delay-delta=N
    • Time to be added to delay between suspend iterations. Can be a fraction of a second.
  • --s4-delay-delta=N
    • Time to be added to delay between hibernate iterations. Can be a fraction of a second.

These options cannot be run with the --s3-device-check/--s4-device-check options.

Example:

sudo fwts s4 --s3-min-delay=1 --s4-max-delay=20 --s4-delay-delta=0.25

gives a delay ranging from 1..20 seconds, and adds 1/4 of a second per iteration to the delay.

Logging

fwts has a fairly complex logging mechanism designed to capture and summarise test pass/fail status messages. Arguably it may gather too much data, but the underling principles behind the design are as follows:

  • To log test status, e.g. Pass, Fail, Skipped (not applicable on the hardware), Aborted (due to some error), Info (such as data from utilities)
  • To capture error conditions to allow a developer to diagnose failures or warnings
  • To provide contextual advice feedback. These lines are prefixed by "ADVICE" tags.

By default, fwts will append it's output to the log file "results.log". The -f and --force-clean options will start the log from fresh, clearing an previous log messages. One can redirect the log data to a different file using the -r and --results-output. If the output file name is stderr or stdout then the output will be directed to these file descriptors. Finally, the - option will redirect the output to stdout.

Examples:

sudo fwts -f -r mylog.log

..write a new log file to mylog.log, clearing out any previous results in the log if it exists already.

sudo fwts --results-output=stderr

..write output to stderr

sudo fwts -

..synonymous with --results-output=stdout or -r stdout, writes log to stdout. This can be piped or redirected, e.g.

sudo fwts - | tee mylog.log

Log formatting controls

By default, the log file displays several columns on information:

line-num test-name log-data

where line-num is a 5 digit line number, test-name is the name of the test being run, and log-data is the output from the test. One can re-configure the log output using the --log-format option. The --log-formation option allows one to specify which fields you require and the order and format of each logged line of output. The available --log-format field operators are as follows:

  • %date - date
  • %time - time
  • %field - log-filter fields
  • %owner - name of the test routine
  • %level - test failure level
  • %line - log line

By default, fwts will format each line based on the --log-format specifiers and append the test output after this formatted text. The best way to describe this is by examples:

sudo fwts --log-format="%date %time %line %owner: " 

An excerpt of the output from this format is:

22/01/11 12:56:06 00297 hpet_check     : HPET configuration test.
22/01/11 12:56:06 00298 hpet_check     : -------------------------------------------------------------------
22/01/11 12:56:06 00299 hpet_check     : Must be run as root or sudo to be able to read system information.
22/01/11 12:56:06 00300 hpet_check     : Aborted test, initialisation failed.
22/01/11 12:56:06 00301 hpet_check     : ===================================================================
22/01/11 12:56:06 00302 hpet_check     : 0 passed, 0 failed, 0 warnings, 3 aborted, 0 skipped, 0 info only.
22/01/11 12:56:06 00303 hpet_check     : ===================================================================

One can order the fields in anyway you require. One can even specify an empty format, which will then just output the text generated from the test an no preceding format field information, e.g:

sudo fwts --log-format=

An excerpt of the output from this format is:

HPET configuration test.
--------------------------------------------------------------------
Must be run as root or sudo to be able to read system information.
Aborted test, initialisation failed.
====================================================================
0 passed, 0 failed, 0 warnings, 3 aborted, 0 skipped, 0 info only.
====================================================================

One particular format field of interest is the %field option. This shows the way each log output is classified into different message types. One can filter messages in the log based on these field types, so for example, one just gets the results and summary of the tests to just focus on the test pass/fail data. Log filtering is activated using the --logfilter option. This allows one to specify the fields that you want or don't want. To see the available fields, use the --log-fields option, e.g. :

fwts --log-fields
Available fields: RES,ERR,WRN,DBG,INF,SUM,SEP,NLN,ADV,HED,TAG

One can specify a comma separated list of fields that one requires. Prefixing a field name with ~ disables that field.

the fields are as follows:

  • RES - results, end of test pass/fail/warning/aborted/skipped/info test results
  • ERR - report any errors when trying to run a test (for example, resources not available, having to run test with sudo, etc)
  • WRN - report any warnings, e.g. a test may have picked up a possible problem, but it's not a failure and the test was able to complete.
  • DBG - deprecated
  • INF - information feedback from the test. This reports data that can be read by a developer to help diagnose issues if a test fails.
  • SUM - summary data - reports collated summary of tests results at end of test execution
  • SEP - separator fields between tests. One can use ~SEP to disable these.
  • NLN - new line fields. One can use ~SEP to disable these.
  • ADV - advice output. Some tests can diagnose problems and give some advice to help with workarounds or deeper explanations of why
    • an error has been detected.
  • HED - headings.
  • TAG - bug tags (see --tag options later).

Examples:

To just get the results and a summary report at the end of the tests, use:

sudo fwts --log-filter=RES,SUM

To get results and extra test information and help advice, use:

sudo fwts --log-filter=INF,RES,ADV

The default options give plenty of data, so the filtering can be useful to drill down on the fields you require. One can mix and match log filtering and log format controls to configure a log that matches ones taste.

Log Width

If the log is being written to a log file, it is assumed to write a log that is 120 characters wide. If writing to a terminal, then the log width is adjusted to the width of the terminal. To override the log with, use the -W or --log-width options. Widths under 80 characters are not allowed.

Examples:

sudo fwts -W 100

or

sudo fwts --log-width=90

Progress Indicators

When fwts is writing results to a log file it will also display a test by test progress summary to stderr. If the test takes a while to run the progress will be updated showing a percentage progress, which is a rough estimation of progress in that particular test. Some tests may run slower as the end completion, so bear in mind that the progress indication is not accurate.

If you want fwts to run silently, the use the -q or --quiet options.

The --stdout-summary option will dump a simple PASSED/FAILED message at the end of the completion of every test.

Progress can be piped into the dialog tool for a more appealing output. Use the -D or --show-progress-dialog option to do this:

sudo fwts -D | dialog --gauge "Firmware Test Suite" 10 80

Extra Tools

Fwts is a tool designed primarily for testing. However, it includes some helpful tools to dump firmware state which can be then forwarded to a developer to help debug and diagnose firmware issues.

The ACPI DSDT and SSDT tables contains ACPI Machine Language code which can sometimes be buggy. The --disassemble-aml option will find these tables and disassemble without the need of the usual acpidump, acpixtract and iasl tools. To run, use:

sudo fwts --disassemble-aml

This will produce files DSDT.dsl and SSDTx.dsl files (where x is the Nth SSDT table).

Another feature is the -d, --dump option. This will dump out several files:

  • README.txt - containing an explanation of the files that have been dumped
  • dmesg.log - output from the kernel message log (like running dmesg > dmesg.log)

  • dmidecode.log - SMBIOS DMI tables, (output from dmidecode)
  • lspci.log - PCI configuration data (output from lspci -vvnn)
  • acpidump.log - ACPI tables in a format identical to that of acpidump

To gather this data, fwts needs to be run with root privilege, e.g.

sudo fwts --dump

LaunchPad Tags

Being able to identify and classify firmware bugs is another feature that fwts provides. The --lp-tags-log option will output tags that can be added by hand to LaunchPad bugs. The tags appear at the end of each failed test prefixed by the "Tags:" marker. At the end of the all the tests all the tags found in all the tests are also listed in the summary report.

The --lp-tags option will ONLY output the tags to the report log.

Example: run all the batch tests and gather LaunchPad specific tags for the failed tests on this hardware and output these tags to stdout:

sudo fwts --lp-tags -

Processing pre-dumped data

Some tests can take previously dumped out raw data from the firmware or kernel logs and process them on a different machine. This is useful if a user finds a problem and can then email the data to a developer who can then process this data with fwts to run some tests. The -k and --klog options allow one to specify a previously dumped log file and process them with the following tests:

  • klog
    • Scan kernel log for general firmware errors
  • acpiinfo
    • Scan kernel log for ACPI driver errors
  • dmesg_common
    • Scan kernel log for common kernel log errors
  • oops
    • Scan kernel log for oops messages

for example, using a copy of dmesg.log output from dmesg or fwts --dump, scan for bugs:

fwts klog acpiinfo dmesg_common oops --klog=dmesg.log

Note you don't need to run with sudo since fwts only needs to read log data.

Several tests can read dumped ACPI tables and scan these for possible bugs using output from sudo fwts --dump. The tests that can process this data are as follows:

  • acpitables
    • Sanity check configuration in ACPI tables
  • apicinstance
    • Check for just one instance of MADT
  • checksum
    • Check ACPI table checksums
  • method
    • Check ACPI DSDT and SSDT methods and objects
  • osilinux
    • Check for _OSI("Linux") in DSDT
  • syntaxcheck
    • Re-assemble DSDT and SSDT and find syntax errors and warnings.
  • acpidump
    • Dump and annotate ACPI tables

These tests read in the ACPI data using the --dumpfile option. Suppose you have a log file called "acpidump.log" dumped using fwts --dump, you can run the above tests on it using:

fwts --dumpfile=acpidump.log acpitables apicinstance checksum method osilinux syntaxcheck acpidump

Miscellaneous Options

The dmidecode tool is used by fwts to gather SMBIOS data. One can override the default path for this tool using --dmidecode, for example

sudo fwts --dmidecode=/usr/sbin/dmidecode

Similary, fwts uses lpci too, and the default path to this can be overridden using --lspci, for example:

sudo fwts --lspci=/usr/bin/lspci

Fwts has a couple of databases used in scanning logs and by default these are located in /usr/share/fwts/ - however, this path can be overridden using the -j and --json-data-path options:

sudo fwts ---json-data-path=/usr/share/fwts

The version and build date of fwts can be dumped out using the -v and --version options:

fwts -v

Which Test to Run?

Scenario

Test to run

Power Management

s3 s4

Gather System Information

acpidump cmosdump memmapdump romdump uefidump

Check kernel log errors

dmesg_common klog oops acpiinfo

Check BIOS tables

smbios bios32 bios_info dmi_decode

Check ACPI tables

acpitables apicinstance checksum fadt method osilinux syntaxcheck wmi

CPU + config in firmware

nx virt maxfreq microcode cpufreq cstates

Thermal + Fan

thermal_trip fan

Memory configuration

ebda mcfg mtrr os2gap dmar crs maxreadreq

Buttons, keys, events

brightness hotkey lid power_button

Power

ac_adapter battery

ACPI alarm

wakealarm

Timers + Interrupts

apicedge hpet_check

Audio Pin configuration

hda_audio

Development and Stable PPAs

We are continually finding new ways that firmware can be broken, and hence fwts is continually being improved to detect firmware issues. Every release cycle a stable version of fwts is uploading into Universe and occasional updates appear as bug fix releases. However, two PPAs are provided containing more up-to-date version of fwts.

The first PPA contains the development version. This has code that can be considered as "cutting edge" or "crack of the day" and generally contains builds that are very close to the source in the tip of the fwts git repository. Currently we support Maverick and Natty builds of this version. To install, use:

sudo add-apt-repository ppa:firmware-testing-team/ppa-fwts-devel
sudo apt-get update
sudo apt-get install fwts

The second PPA contains more stable updates, generally code that has been tested for quite a while and is thus less likely to contain new features that cause breakage. To install from this PPA use:

sudo add-apt-repository ppa:firmware-testing-team/ppa-fwts-stable
sudo apt-get update
sudo apt-get install fwts

Source

The fwts source is in a git repository: git://kernel.ubuntu.com/cking/fwts/.git

There are several branches: maverick and natty contain the released code in Universe for Ubuntu Maverick 10.10 and Natty 11.04 respectively. New code is now being added into the master branch, so this branch contains the current new code. Small bug fixes will be periodically applied to the natty branches.