DebuggingKernelSuspend
4725
Comment:
|
← Revision 56 as of 2019-01-28 21:01:56 ⇥
8338
As per LP#1812561 add usb debug note for suspend debug.
|
Deletions are marked like this. | Additions are marked like this. |
Line 6: | Line 6: |
This page describes how to debug Suspend to RAM/Resume problems on your computer. Do not confuse this with Suspend to disk (also known as hibernate). [[UnderstandingSuspend]] may have useful background information on where problems can occur. | This page describes how to debug suspend to RAM, and resume from suspend to RAM problems with your computer. |
Line 8: | Line 8: |
Suspend and Resume use facilities within your BIOS called ACPI, or Advanced Configuration and Power Interface. Linux provides an ACPI subsystem that manages the suspend and resume process. Usually problems occur when resuming, and normally the culprit is a device driver that does not recover from a powered down state. If your computer successfully performs a suspend, then it is quite likely any resume problems are due to another device driver and not the ACPI subsystem. | For issues regarding suspend to disk (i.e. hibernate), please see [[https://wiki.ubuntu.com/DebuggingKernelHibernate|here]]. |
Line 10: | Line 10: |
The debugging procedure described below requires a linux kernel with the capability of "/sys/power/pm_trace". You can ensure that pm_trace'ing is possible by looking in the directory /sys/power/ (from a terminal type {{{ls /sys/power/}}}) | Suspend and resume use facilities within your BIOS called ACPI, or Advanced Configuration and Power Interface. Linux provides an ACPI subsystem that manages the suspend and resume process. Usually problems occur when resuming, and normally the culprit is a device driver that does not recover from a powered down state. If your computer successfully performs a suspend, then it is quite likely any resume problems are due to another device driver and not the ACPI subsystem. The debugging procedure described below assumes you have already installed the latest BIOS from your vendor, and requires a linux kernel with the capability of "/sys/power/pm_trace". You can ensure that pm_trace'ing is possible by looking in the directory /sys/power/ (from a terminal type {{{ls /sys/power/}}}) |
Line 13: | Line 15: |
= "resume-trace" debugging procedure for finding buggy drivers = | = Familiarize yourself with pm_tracing a resume to find buggy drivers = |
Line 15: | Line 17: |
Resume problems are difficult to debug. The approach used here needs to make notes on progress during resume and be able to recover them after a manual reboot. But there is no non-volatile storage available at the time resume is bringing up your computer. The only hardware on a PC motherboard that retains information across power cycles is the real time clock (RTC), so that is what is used. For those that want to know the details, read Documentation/power/s2ram.txt in your kernel sources. The implementation of suspend/resume debug trace is in drivers/base/power/trace.c. | Resume problems are difficult to debug. The approach used here needs to make notes on progress during resume and be able to recover them after a manual reboot. But there is no non-volatile storage available at the time resume is bringing up your computer. The only hardware on a PC motherboard that retains information across power cycles is the real time clock (RTC), so that is what is used. For those that want to know the details, please read Documentation/power/s2ram.txt in your kernel sources, or see [[https://www.kernel.org/doc/Documentation/power/s2ram.txt|here]]. The implementation of suspend/resume debug trace is in drivers/base/power/trace.c. |
Line 17: | Line 19: |
Caveat Emptor: Using the following debug suggestions will radically change the values in your RTC chip, so much so that your file system will think it has been eons since the last fsck. You can avoid a long fsck delay by using 'tune2fs'. For example, 'tune2fs -i 0 /dev/sda1' disables fsck on boot. But first you'll want to use 'tune2fs -l <partition>' to find your current settings - look at the "Check interval" setting. | Caveat Emptor: Using the following debug suggestions will radically change the values in your RTC chip, so much so that your file system will think it has been eons since the last fsck. You can avoid a long fsck delay by using 'tune2fs'. For example, 'tune2fs -i 0 /dev/sda1' disables fsck on boot. But first you'll want to use 'tune2fs -l <partition>' to find your current settings - look at the "Check interval" setting. |
Line 21: | Line 23: |
Remember that when you are all done, you'll want to set your RTC clock back. Note that NTP by itself is not sufficient to recover the correct time of day, since by default NTP is configured to do nothing if the time is off by more than 1000 seconds. You must first use the 'ntpdate' or 'date' command to get the clock close. | Remember that when you are all done, you'll want to set your RTC clock back. Note that NTP by itself is not sufficient to recover the correct time of day, since by default NTP is configured to do nothing if the time is off by more than 1000 seconds. You must first use the 'ntpdate' or 'date' command to get the clock close. |
Line 23: | Line 25: |
OK. With that understood.... In order to simulate your suspend/resume process, enter the following commands: |
In order to simulate your suspend/resume process, enter the following commands: |
Line 26: | Line 27: |
sudo sh -c "sync; echo 1 > /sys/power/pm_trace; pm-suspend" | sudo sh -c "sync && echo 1 > /sys/power/pm_trace && pm-suspend" |
Line 32: | Line 33: |
Line 38: | Line 38: |
Line 45: | Line 44: |
Line 50: | Line 48: |
The only way to prove this is to remove the module prior to initiating suspend. Repeat as needed... | The only way to prove this is to remove the module prior to initiating suspend. Repeat as needed. |
Line 52: | Line 50: |
If you get a device number rather than name, lspci and /sys/devices/pci* are your friends. | If you get a device number rather than a name, lspci and /sys/devices/pci* are your friends. |
Line 54: | Line 52: |
= Further hints = [[http://ubuntuforums.org/showthread.php?p=3066404|Detailed analysis of ACPI kernel code for debugging a suspend problem]] |
= Debugging information to provide in your bug report = |
Line 57: | Line 54: |
------ '''Also see:''' |
In order to identify the root cause of the issue, all of the following information needs to be provided, uncompressed and untarred, in order for a developer to work on it. 1. Which part of the process does the issue occur with, the suspend to ram, or resuming from? 1. Please advise how you suspended specifically. For example: * Executing at a terminal pm-suspend * Shutting the lid of your laptop, which is set to suspend on close. * Clicking the word Suspend in the GUI. * The computer suspends automatically on inactivity. 1. Please advise how you resumed specifically. For example: * Pressed the power button * Lifted the lid * Clicked a key on the keyboard 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal: {{{ cat /proc/acpi/wakeup > wakeup }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal:{{{ sudo su echo freezer > /sys/power/pm_test exit }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal:{{{ sudo su echo devices > /sys/power/pm_test exit }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal:{{{ sudo su echo platform > /sys/power/pm_test exit }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal:{{{ sudo su echo processors > /sys/power/pm_test exit }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal:{{{ sudo su echo core > /sys/power/pm_test exit }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. While booted into the latest [[https://wiki.ubuntu.com/Kernel/MainlineBuilds|non-daily mainline kernel]], execute at a terminal:{{{ sudo su echo none > /sys/power/pm_test exit }}} After executing this, perform a pm_trace and attach the results to your report. <<BR>> 1. Please provide the output of the following terminal command: {{{ cat /sys/kernel/debug/suspend_stats }}} 1. If you have a graphics related issue after resume (corruption, display blank, etc.) please [[https://help.ubuntu.com/community/DebuggingSystemCrash#SSH|SSH]] into your machine and capture both /var/log/Xorg.0.log and /var/log/Xorg.0.log.old. Please attach each separately. 1. If your issue may be USB related, for example, disabling XHC1 prior to sleeping is a workaround via a terminal: {{{ # echo XHC1 > /proc/acpi/wakeup }}} then one will want to use the following kernel boot parameter, and attach dmesg in full: {{{ usbcore.dyndbg=+p }}} |
Line 60: | Line 107: |
* [[http://lxr.linux.no/linux/Documentation/power/s2ram.txt|How to get s2ram working]] * [[http://lxr.linux.no/linux/Documentation/power/basic-pm-debugging.txt|Testing suspend to disk (hibernate)]] |
= Further reading = * [[https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-power]] - Documentation on pm_trace and additional methods for debugging * [[http://ubuntuforums.org/showthread.php?p=3066404|Detailed analysis of ACPI kernel code for debugging a suspend problem]] = Links = * [[https://wiki.ubuntu.com/UnderstandingSuspend]] * [[https://01.org/blogs/rzhang/2015/best-practice-debug-linux-suspend/hibernate-issues]] Best practices to debug suspend issues * [[DebuggingKernelHibernate]] |
This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages. |
Introduction
This page describes how to debug suspend to RAM, and resume from suspend to RAM problems with your computer.
For issues regarding suspend to disk (i.e. hibernate), please see here.
Suspend and resume use facilities within your BIOS called ACPI, or Advanced Configuration and Power Interface. Linux provides an ACPI subsystem that manages the suspend and resume process. Usually problems occur when resuming, and normally the culprit is a device driver that does not recover from a powered down state. If your computer successfully performs a suspend, then it is quite likely any resume problems are due to another device driver and not the ACPI subsystem.
The debugging procedure described below assumes you have already installed the latest BIOS from your vendor, and requires a linux kernel with the capability of "/sys/power/pm_trace". You can ensure that pm_trace'ing is possible by looking in the directory /sys/power/ (from a terminal type ls /sys/power/) to see if a file called pm_trace exists.
Familiarize yourself with pm_tracing a resume to find buggy drivers
Resume problems are difficult to debug. The approach used here needs to make notes on progress during resume and be able to recover them after a manual reboot. But there is no non-volatile storage available at the time resume is bringing up your computer. The only hardware on a PC motherboard that retains information across power cycles is the real time clock (RTC), so that is what is used. For those that want to know the details, please read Documentation/power/s2ram.txt in your kernel sources, or see here. The implementation of suspend/resume debug trace is in drivers/base/power/trace.c.
Caveat Emptor: Using the following debug suggestions will radically change the values in your RTC chip, so much so that your file system will think it has been eons since the last fsck. You can avoid a long fsck delay by using 'tune2fs'. For example, 'tune2fs -i 0 /dev/sda1' disables fsck on boot. But first you'll want to use 'tune2fs -l <partition>' to find your current settings - look at the "Check interval" setting.
From a fresh install, the default time interval is 15552000, which is 6 months.
Remember that when you are all done, you'll want to set your RTC clock back. Note that NTP by itself is not sufficient to recover the correct time of day, since by default NTP is configured to do nothing if the time is off by more than 1000 seconds. You must first use the 'ntpdate' or 'date' command to get the clock close.
In order to simulate your suspend/resume process, enter the following commands:
sudo sh -c "sync && echo 1 > /sys/power/pm_trace && pm-suspend"
At this point your computer should enter the suspend state within a few seconds. Usually the power LED will slowly flash when in the suspended state. When that has happened, initiate the resume process by pressing the power button. Observe closely if the disk light comes on briefly. This indicates that resume has begun. If resume fails to complete, then press the power button until the computer turns off. Power on your computer making sure that it loads the same kernel that exhibited the resume problem. You have about 3 minutes to start this boot process before the information saved in the RTC gets corrupted.
Start a console and enter:
dmesg > dmesg.txt
You can edit this file and find lines similar to these:
[ 11.323206] Magic number: 0:798:264 [ 11.323257] hash matches drivers/base/power/resume.c:46
There may well be another 'hash matches' line beyond that. If so, then you are in luck because the last one is the likely culprit. For example:
hash matches device i2c-9191
The only way to prove this is to remove the module prior to initiating suspend. Repeat as needed.
If you get a device number rather than a name, lspci and /sys/devices/pci* are your friends.
Debugging information to provide in your bug report
In order to identify the root cause of the issue, all of the following information needs to be provided, uncompressed and untarred, in order for a developer to work on it.
- Which part of the process does the issue occur with, the suspend to ram, or resuming from?
- Please advise how you suspended specifically. For example:
- Executing at a terminal pm-suspend
- Shutting the lid of your laptop, which is set to suspend on close.
- Clicking the word Suspend in the GUI.
- The computer suspends automatically on inactivity.
- Please advise how you resumed specifically. For example:
- Pressed the power button
- Lifted the lid
- Clicked a key on the keyboard
While booted into the latest non-daily mainline kernel, execute at a terminal:
cat /proc/acpi/wakeup > wakeup
After executing this, perform a pm_trace and attach the results to your report.
While booted into the latest non-daily mainline kernel, execute at a terminal:
sudo su echo freezer > /sys/power/pm_test exit
After executing this, perform a pm_trace and attach the results to your report.
While booted into the latest non-daily mainline kernel, execute at a terminal:
sudo su echo devices > /sys/power/pm_test exit
After executing this, perform a pm_trace and attach the results to your report.
While booted into the latest non-daily mainline kernel, execute at a terminal:
sudo su echo platform > /sys/power/pm_test exit
After executing this, perform a pm_trace and attach the results to your report.
While booted into the latest non-daily mainline kernel, execute at a terminal:
sudo su echo processors > /sys/power/pm_test exit
After executing this, perform a pm_trace and attach the results to your report.
While booted into the latest non-daily mainline kernel, execute at a terminal:
sudo su echo core > /sys/power/pm_test exit
After executing this, perform a pm_trace and attach the results to your report.
While booted into the latest non-daily mainline kernel, execute at a terminal:
sudo su echo none > /sys/power/pm_test exit
After executing this, perform a pm_trace and attach the results to your report.
Please provide the output of the following terminal command:
cat /sys/kernel/debug/suspend_stats
If you have a graphics related issue after resume (corruption, display blank, etc.) please SSH into your machine and capture both /var/log/Xorg.0.log and /var/log/Xorg.0.log.old. Please attach each separately.
If your issue may be USB related, for example, disabling XHC1 prior to sleeping is a workaround via a terminal:
# echo XHC1 > /proc/acpi/wakeup
then one will want to use the following kernel boot parameter, and attach dmesg in full:
usbcore.dyndbg=+p
Further reading
https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-power - Documentation on pm_trace and additional methods for debugging
Detailed analysis of ACPI kernel code for debugging a suspend problem
Links
https://01.org/blogs/rzhang/2015/best-practice-debug-linux-suspend/hibernate-issues Best practices to debug suspend issues
DebuggingKernelSuspend (last edited 2019-01-28 21:01:56 by penalvch)