Reference
|
Firmware Test Suite (fwts)
The tool fwts comprises of over fifty tests that are designed to exercise and test different aspects of a machine's firmware. Many of these tests need super user access to read BIOS data and ACPI tables, so the tool requires running with super user privileges (e.g. with sudo).
Running fwts with no options will run through all the batch tests that require no user interaction and by default append test results into a log file called "results.log". However, if require, one can specify specific tests to run rather than the default set.
There are 6 classes of tests in fwts:
- Batch
- These run automatically by default and require no user intervention. However, one can select these with the -b or --batch option.
- Batch Experimental
- This tests require no user intervention and contain code that may give false positive results - much akin to beta quality code. One can select these tests with the --batch-experimental option.
- Interactive
- These tests require user interaction, such as pressing keys, closing laptop lids, disconnecting/connecting power supplies etc. Selected with the --interactive option.
- Interactive Experimental
- These are beta quality interactive tests. Selected with the --interactive-experimental option.
- Power Management
- These run suspend (S3) and hibernate (S4) power management tests and may hang the machine, hence they are not in the default batch test category. Select these tests using -P or --power-states options.
- Utilities
- These are not strictly tests, but will gather system information useful for kernel developers to debug a system. So if a test fails, it is useful to run these and append the data to the results log to help diagnose and debug issues with the firmware. These tests can be selected with the -u or --utils options.
One can mix and match these tests to run the different classes of tests in one go, e.g.:
sudo fwts --batch --interactive --power-states
To run just one or a few tests rather than the default set of batch tests, one can specify the test name too. To see the names of the available tests use the --show-tests or --show-tests-full options. Currently, with version V0.23.17 there following tests are available:
Batch tests
General ACPI information check. |
|
ACPI table settings sanity checks. |
|
APIC Edge/Level Check. |
|
Check for single instance of APIC/MADT table. |
|
Check PCIe ASPM. |
|
Check BIOS32 Service Directory. |
|
Gather BIOS DMI information. |
|
Check ACPI table checksum. |
|
CPU frequency scaling tests (takes ~1-2 mins). |
|
Check PCI host bridge configuration using _CRS. |
|
Check for UEFI Compatability Support Module. |
|
Check processor C state support. |
|
Check sane DMA Remapping (VT-d). |
|
General dmesg common errors check. |
|
Test DMI/SMBIOS tables for errors. (merged to dmicheck in 13.08) |
|
Test DMI/SMBIOS tables for errors. (introduced in 13.08) |
|
Validate EBDA region is mapped and reserved in memory map table. |
|
FADT SCI_EN enabled check. |
|
Simple Fan Tests. |
|
Check HDA Audio Pin Configs. |
|
HPET configuration test. |
|
Scan kernel log for errors and warnings. |
|
Check max CPU frequencies against max scaling frequency. |
|
Checks firmware has set PCI Express MaxReadReq to a higher value on non-motherboard devices. |
|
MCFG PCI Express* memory mapped config space. |
|
ACPI DSDT Method Semantic Tests. |
|
Check if system is using latest microcode. |
|
Check Multi Processor tables. |
|
CPU MSR consistency check. |
|
MTRR validation. |
|
Test if CPU NX is disabled by the BIOS. |
|
Scan kernel log for Oopses. |
|
OS/2 memory hole test. |
|
Disassemble DSDT to check for _OSI("Linux"). |
|
This test checks the sanity of the Processor Clocking Control as found on some HP ProLiant machines. |
|
Check legacy BIOS PCI IRQ Routing Table. |
|
Check PnP BIOS Support Installation structure. |
|
UEFI secure boot test. |
|
Check SMBIOS. (merged to dmicheck in 13.08) |
|
Re-assemble DSDT and find syntax errors and warnings. |
|
Gather kernel system information. |
|
Test CPU Virtualisation Configuration. |
|
Test ACPI Wakealarm. |
|
Extract and analyse Windows Management Instrumentation (WMI). |
UEFI tests
Miscellaneous UEFI runtime tests |
|
UEFI runtime service time interface tests |
|
UEFI runtime service variable interface tests |
Interactive tests
Interactive ac_adapter power test. |
|
Battery Tests. |
|
Interactive LCD brightness test. |
|
Hotkey scan code tests. |
|
Interactive lid button test. |
|
Interactive power_button button test. |
Power States tests
Utilities
Check ACPI table acpidump. |
|
Dump CMOS Memory. |
|
Dump ACPI annotated _CRS packages. |
|
Dump Extended BIOS Data Area. |
|
Dump FADT defined General Purpose Event configuration. |
|
Dump system memory map. |
|
Dump Multi Processor tables. |
|
Dump ACPI annotated _PLD packages. |
|
Dump ACPI annotated _PRS packages. |
|
Dump ROM data. |
|
Dump UEFI variables. |
|
Get the NVRAM and varaible information. |
One can specify one or more of these tests, for example:
sudo fwts klog checksum wakealarm s3
..and even mix and match with classes of tests, e.g.
sudo fwts klog --power-states --batch-experimental
One can run exclude tests from being run with the -S and --skip-test options. List the names of the tests in a comma separated list and fwts will skip over and ignore tests tests. For example
sudo fwts --batch --skip-test=nx,oops,smbios
or
sudo fwts --interactive -S lid,power_button
Finally, the -a and --all options allow one to run *all* the tests.
Getting Help
fwts has two methods of getting help. Use -?, -h or --help options to display a short summary of available options. Alternatively, a manual is availble using:
man fwts
S3 and S4 Test Specific Options
While most tests have default modes of operation that cater for most use cases, some tests can be driven in different modes to perform intensive soak testing. The S3 and S4 tests have a range of options to allow one to do multiple repeats and to configure the delay times when awake and the sleep times when suspended or hibernated. There are also options to pass quirks to the pm-suspend and pm-hibernate scripts that initiate the suspend/hibernates. The options are as follows:
- --s3-quirks
- Comma separated list of quirk arguments to pass to pm-suspend.
- --s4-quirks
- Comma separated list of quirk arguments to pass to pm-hibernate.
for example:
sudo fwts s3 --s3-quirks=--quirk-s3-bios,--quirk-dpms-suspend
see man pm-utils for a list of available quirks.
- --s3-multiple=N
- Run S3 tests multiple times, e.g. --s3-multiple=250
- --s4-multiple=N
- Run S3 tests multiple times, e.g. --s4-multiple=30
By default the multiple tests is set to 1, i.e. just one cycle. These options are designed for repeated S3/S4 soak testing.
- --s3-sleep-delay=N
- Sleep N seconds between start of suspend and wakeup, e.g. --s3-sleep-delay=30
- --s4-sleep-delay=N
- Sleep N seconds between start of hibernate and wakeup, e.g. --s4-sleep-delay=60
By default, fwts allows plenty of time for a suspend and a hibernate to occur before triggering a wakeup alarm to come out of suspend/hibernation. However, one can tune the sleep delay to match the suspend or hibernate time on one's hardware. Note that this time is measured from the start of the suspend/hibernate to the time that the wake alarm triggers. In retrospect, these options should have been called "wakeup-delay".
- --s3-device-check
- Check differences between device configurations over a suspend cycle.
- --s4-device-check
- Check differences between device configurations over a hibernate cycle.
These options enable a simple device status check. Device status/configuration for devices such as Bluetooth, Wireless, Ethernet, etc are gathered before the suspend/hibernate and then checked against the status/configuration on wakeup to see if any devices fail to re-configure. Because some devices take a while to get back to their original state (such as wifi that need to re-association), this tests waits 15 seconds after wakeup before doing the status/configuration check. This means these options cannot be used with the --s3-min-delay, --s3-max-delay and --s3-delay-delta options.
Note we add a default of 15 seconds to allow wifi to re-associate. Cannot be used with --s3-min-delay, --s3-max-delay and --s3-delay-delta.
The final options controls a delay time between waking up and going into the next suspend/hibernate. One can specify a min/max range of times to delay and an increment that is added to each cycle up to the max time. Then the delay wraps back to the min time and restarts again. The main use for this is to add some variable delay which may trip bugs in drivers that have timing bugs, for example, Wifi drivers may oops when being suspend while they are re-associating. Hence selecting appropriate min/max times can sometimes trigger race conditions in drivers when going into suspend/hibernate. The options are as follows:
- --s3-min-delay=N
- Minimum time in seconds between suspend iterations, e.g. --s3-min-delay=10
- --s4-min-delay=N
- Minimum time in seconds between hibernate iterations, e.g. --s4-min-delay=10
- --s3-max-delay=N
- Maximum time in seconds between suspend iterations, e.g. --s3-max-delay=20
- --s4-max-delay=N
- Maximum time in seconds between hibernate iterations, e.g. --s4-max-delay=20
- --s3-delay-delta=N
- Time to be added to delay between suspend iterations. Can be a fraction of a second.
- --s4-delay-delta=N
- Time to be added to delay between hibernate iterations. Can be a fraction of a second.
These options cannot be run with the --s3-device-check/--s4-device-check options.
Example:
sudo fwts s4 --s3-min-delay=1 --s4-max-delay=20 --s4-delay-delta=0.25
gives a delay ranging from 1..20 seconds, and adds 1/4 of a second per iteration to the delay.
Logging
fwts has a fairly complex logging mechanism designed to capture and summarise test pass/fail status messages. Arguably it may gather too much data, but the underling principles behind the design are as follows:
- To log test status, e.g. Pass, Fail, Skipped (not applicable on the hardware), Aborted (due to some error), Info (such as data from utilities)
- To capture error conditions to allow a developer to diagnose failures or warnings
- To provide contextual advice feedback. These lines are prefixed by "ADVICE" tags.
By default, fwts will append it's output to the log file "results.log". The -f and --force-clean options will start the log from fresh, clearing an previous log messages. One can redirect the log data to a different file using the -r and --results-output. If the output file name is stderr or stdout then the output will be directed to these file descriptors. Finally, the - option will redirect the output to stdout.
Examples:
sudo fwts -f -r mylog.log
..write a new log file to mylog.log, clearing out any previous results in the log if it exists already.
sudo fwts --results-output=stderr
..write output to stderr
sudo fwts -
..synonymous with --results-output=stdout or -r stdout, writes log to stdout. This can be piped or redirected, e.g.
sudo fwts - | tee mylog.log
Log formatting controls
By default, the log file displays several columns on information:
line-num test-name log-data
where line-num is a 5 digit line number, test-name is the name of the test being run, and log-data is the output from the test. One can re-configure the log output using the --log-format option. The --log-formation option allows one to specify which fields you require and the order and format of each logged line of output. The available --log-format field operators are as follows:
- %date - date
- %time - time
- %field - log-filter fields
- %owner - name of the test routine
- %level - test failure level
- %line - log line
By default, fwts will format each line based on the --log-format specifiers and append the test output after this formatted text. The best way to describe this is by examples:
sudo fwts --log-format="%date %time %line %owner: "
An excerpt of the output from this format is:
22/01/11 12:56:06 00297 hpet_check : HPET configuration test. 22/01/11 12:56:06 00298 hpet_check : ------------------------------------------------------------------- 22/01/11 12:56:06 00299 hpet_check : Must be run as root or sudo to be able to read system information. 22/01/11 12:56:06 00300 hpet_check : Aborted test, initialisation failed. 22/01/11 12:56:06 00301 hpet_check : =================================================================== 22/01/11 12:56:06 00302 hpet_check : 0 passed, 0 failed, 0 warnings, 3 aborted, 0 skipped, 0 info only. 22/01/11 12:56:06 00303 hpet_check : ===================================================================
One can order the fields in anyway you require. One can even specify an empty format, which will then just output the text generated from the test an no preceding format field information, e.g:
sudo fwts --log-format=
An excerpt of the output from this format is:
HPET configuration test. -------------------------------------------------------------------- Must be run as root or sudo to be able to read system information. Aborted test, initialisation failed. ==================================================================== 0 passed, 0 failed, 0 warnings, 3 aborted, 0 skipped, 0 info only. ====================================================================
One particular format field of interest is the %field option. This shows the way each log output is classified into different message types. One can filter messages in the log based on these field types, so for example, one just gets the results and summary of the tests to just focus on the test pass/fail data. Log filtering is activated using the --logfilter option. This allows one to specify the fields that you want or don't want. To see the available fields, use the --log-fields option, e.g. :
fwts --log-fields Available fields: RES,ERR,WRN,DBG,INF,SUM,SEP,NLN,ADV,HED,TAG
One can specify a comma separated list of fields that one requires. Prefixing a field name with ~ disables that field.
the fields are as follows:
- RES - results, end of test pass/fail/warning/aborted/skipped/info test results
- ERR - report any errors when trying to run a test (for example, resources not available, having to run test with sudo, etc)
- WRN - report any warnings, e.g. a test may have picked up a possible problem, but it's not a failure and the test was able to complete.
- DBG - deprecated
- INF - information feedback from the test. This reports data that can be read by a developer to help diagnose issues if a test fails.
- SUM - summary data - reports collated summary of tests results at end of test execution
- SEP - separator fields between tests. One can use ~SEP to disable these.
- NLN - new line fields. One can use ~SEP to disable these.
- ADV - advice output. Some tests can diagnose problems and give some advice to help with workarounds or deeper explanations of why
- an error has been detected.
- HED - headings.
- TAG - bug tags (see --tag options later).
Examples:
To just get the results and a summary report at the end of the tests, use:
sudo fwts --log-filter=RES,SUM
To get results and extra test information and help advice, use:
sudo fwts --log-filter=INF,RES,ADV
The default options give plenty of data, so the filtering can be useful to drill down on the fields you require. One can mix and match log filtering and log format controls to configure a log that matches ones taste.
Log Width
If the log is being written to a log file, it is assumed to write a log that is 120 characters wide. If writing to a terminal, then the log width is adjusted to the width of the terminal. To override the log with, use the -W or --log-width options. Widths under 80 characters are not allowed.
Examples:
sudo fwts -W 100
or
sudo fwts --log-width=90
Progress Indicators
When fwts is writing results to a log file it will also display a test by test progress summary to stderr. If the test takes a while to run the progress will be updated showing a percentage progress, which is a rough estimation of progress in that particular test. Some tests may run slower as the end completion, so bear in mind that the progress indication is not accurate.
If you want fwts to run silently, the use the -q or --quiet options.
The --stdout-summary option will dump a simple PASSED/FAILED message at the end of the completion of every test.
Progress can be piped into the dialog tool for a more appealing output. Use the -D or --show-progress-dialog option to do this:
sudo fwts -D | dialog --gauge "Firmware Test Suite" 10 80
Extra Tools
Fwts is a tool designed primarily for testing. However, it includes some helpful tools to dump firmware state which can be then forwarded to a developer to help debug and diagnose firmware issues.
The ACPI DSDT and SSDT tables contains ACPI Machine Language code which can sometimes be buggy. The --disassemble-aml option will find these tables and disassemble without the need of the usual acpidump, acpixtract and iasl tools. To run, use:
sudo fwts --disassemble-aml
This will produce files DSDT.dsl and SSDTx.dsl files (where x is the Nth SSDT table).
Another feature is the -d, --dump option. This will dump out several files:
- README.txt - containing an explanation of the files that have been dumped
dmesg.log - output from the kernel message log (like running dmesg > dmesg.log)
- dmidecode.log - SMBIOS DMI tables, (output from dmidecode)
- lspci.log - PCI configuration data (output from lspci -vvnn)
- acpidump.log - ACPI tables in a format identical to that of acpidump
To gather this data, fwts needs to be run with root privilege, e.g.
sudo fwts --dump
Processing pre-dumped data
Some tests can take previously dumped out raw data from the firmware or kernel logs and process them on a different machine. This is useful if a user finds a problem and can then email the data to a developer who can then process this data with fwts to run some tests. The -k and --klog options allow one to specify a previously dumped log file and process them with the following tests:
- klog
- Scan kernel log for general firmware errors
- acpiinfo
- Scan kernel log for ACPI driver errors
- dmesg_common
- Scan kernel log for common kernel log errors
- oops
- Scan kernel log for oops messages
for example, using a copy of dmesg.log output from dmesg or fwts --dump, scan for bugs:
fwts klog acpiinfo dmesg_common oops --klog=dmesg.log
Note you don't need to run with sudo since fwts only needs to read log data.
Several tests can read dumped ACPI tables and scan these for possible bugs using output from sudo fwts --dump. The tests that can process this data are as follows:
- acpitables
- Sanity check configuration in ACPI tables
- apicinstance
- Check for just one instance of MADT
- checksum
- Check ACPI table checksums
- method
- Check ACPI DSDT and SSDT methods and objects
- osilinux
- Check for _OSI("Linux") in DSDT
- syntaxcheck
- Re-assemble DSDT and SSDT and find syntax errors and warnings.
- acpidump
- Dump and annotate ACPI tables
These tests read in the ACPI data using the --dumpfile option. Suppose you have a log file called "acpidump.log" dumped using fwts --dump, you can run the above tests on it using:
fwts --dumpfile=acpidump.log acpitables apicinstance checksum method osilinux syntaxcheck acpidump
Miscellaneous Options
Fwts uses lspci and the default path to this can be overridden using --lspci, for example:
sudo fwts --lspci=/usr/bin/lspci
Fwts has a couple of databases used in scanning logs and by default these are located in /usr/share/fwts/ - however, this path can be overridden using the -j and --json-data-path options:
sudo fwts ---json-data-path=/usr/share/fwts
The version and build date of fwts can be dumped out using the -v and --version options:
fwts -v
Which Test to Run?
Scenario |
Test to run |
Power Management |
s3 s4 |
Gather System Information |
acpidump cmosdump memmapdump romdump uefidump |
Check kernel log errors |
dmesg_common klog oops acpiinfo |
Check BIOS tables |
smbios bios32 bios_info dmi_decode |
Check ACPI tables |
acpitables apicinstance checksum fadt method osilinux syntaxcheck wmi |
CPU + config in firmware |
nx virt maxfreq microcode cpufreq cstates |
Thermal + Fan |
thermal_trip fan |
Memory configuration |
ebda mcfg mtrr os2gap dmar crs maxreadreq |
Buttons, keys, events |
brightness hotkey lid power_button |
Power |
ac_adapter battery |
ACPI alarm |
wakealarm |
Timers + Interrupts |
apicedge hpet_check |
Audio Pin configuration |
hda_audio |
UEFI testing |
uefirtmisc uefirttime uefirtvariable |
Stable PPA
We are continually finding new ways that firmware can be broken, and hence fwts is continually being improved to detect firmware issues. Every release cycle a stable version of fwts is uploading into Universe and occasional updates appear as bug fix releases. However, a PPA is provided containing more up-to-date version of fwts.
The stable PPA contains more stable updates, generally code that has been tested for quite a while and is thus less likely to contain new features that cause breakage. To install from this PPA use:
sudo add-apt-repository ppa:firmware-testing-team/ppa-fwts-stable sudo apt-get update sudo apt-get install fwts
Source
The fwts source is in a git repository: git://kernel.ubuntu.com/hwe/fwts
Mailing Lists
fwts-announce : Formal announcements for the FWTS and the FWTS-Live projects.
fwts-devel : Patches, discussions about the FWTS and the FWTS-Live projects.
More Information
Please refer to the FWTS portal page for more information.