Testing and Working

Getting Started

TODO: This section needs more detail and more work. Everything on this page also needs review by someone on the Ubuntu Platform Team.

Welcome to the wonderful world of Ubuntu testing. This guide will help you avoid common pitfalls of running an Ubuntu developmental release. Some guidelines:

• Check the changelogs on the changes list.

• The Archive Status Page to see which packages are broken and if they're important to you.

• Check the development forums to see if people are having serious problems.

• sudo apt-get update
• sudo apt-get upgrade
  • Held Back packages:
    • Examples go here
    • Never do a partial upgrade, things are held back for a reason.
    • Wait for the archive to update
    • apt-get dist-upgrade does what you tell it to do, if the archive is in flux then it will remove things to keep your computer consistent. Usually you do not want this, so don't do it. If a package is held up for a long time (a few days), then try to apt-get install it, it might be held back because it depends on new packages. If you have a bunch held back just go through them individually and see if anything wants to be removed.
      • Sometimes it's obvious that it wants to remove something and replace it with something newer (libfoo1 is going to be REMOVED, replaced by libfoo2)
    • Update manager
      • Don't just click partial upgrade, odds are eventually it will break.

      • Read this guide for more details on managing partial upgrades.

Getting Help

General

"Living on the Edge" can sometimes involve debugging problems, asking for help and filing bug reports if you encounter problems.

• IRC: #ubuntu+1 on irc.freenode.net for support and to find out if a certain problem is already known

  • please use #ubuntu-devel only if helping with development

  • please use #ubuntu-bugs if helping report/triage bugs

• If you use the Ubuntu Forums, ask for help in the development forum.

• You can also ask a question in the Launchpad Answers tracker.

• If you encounter serious problems, that looks like maintainers should know about, consider reporting a bug

  • add as much information to the bug reports as possible: DebuggingProcedures

• If you're on the ubuntu-devel-discuss mailing list, you can ask for input on an issue or a possible fix for it there.

Specific

Mirror Problems

• checksum failures (development branch changes hourly; sometimes mirrors get skewed)
• unauthenticated package warnings
  • don't install unauthenticated packages; this warning is for your protection (it could be a real attack))
  • next mirror push may resolve the problems, so wait an hour and try again
  • try temporarily using archive.ubuntu.com
    • remember to switch back! it's slow and expensive

Uninstallable packages

• incorrect dependencies
• file-level conflicts ("trying to overwrite '/bar/baz', which is also in package “foo")
• broken maintainer scripts
• note that apt will fail if the system is too inconsistent, and you may have to fall back to using dpkg directly

Unpack failures

• typically haven't done much to your system
• if due to file-level conflict, may be fixable using:
  • sudo dpkg --force-overwrite -i /var/cache/apt/archives/foo.deb
  • almost always indicates a missing Replaces field
• bugs in pre-installation scripts generally unfixable without repacking .deb
• sometimes pre-removal script of old package may be broken; may require editing by hand
  • see /var/lib/dpkg/info/foo.prerm

Configure Options

• package's files are unpacked, but post-installation script failed
• sometimes a retry is enough
  • sudo dpkg --configure -a
• sometimes post-installation script is broken; may require editing by hand
  • see /var/lib/dpkg/info/foo.postinst
• update-alternatives and dpkg-divert are often used and sometimes used wrongly; see the documentation if need be

Package system debugging

• dpkg itself only gives you an exit code, which isn't very useful
• read back through the output for the earliest error that caused dpkg to fail, and fix that
• there is a log in /var/log/apt/term.log that contains the dpkg output when the scripts were run
• when reporting a bug, quote the full output, not just the end
• most maintainer scripts are shell; to get a full execution trace, edit them in /var/lib/dpkg/info/, put 'set -x' on the second line, and retry
• if debconf goes wrong (often indicated by exit codes that are multiples of 10), get a debconf trace:
  • export DEBCONF_DEBUG=developer

  • exit codes 10 and 20 are usually bad arguments to db_something, while 30 is often just a missing “|| true” after db_input or db_go

  • add verbose flags to commands

• strace (sledgehammer tracing tool, output can be hard to read)

  • sudo strace -f dpkg-command... 2>/tmp/foo.out

• debug apt-get dependency problems; output can be hard to read
  • sudo apt-get -o Debug::pkgProblemResolver=true ...
• occasionally dpkg itself goes wrong; see help
  • dpkg –debug=help
  • dpkg debug options do not help with maintainer script problems!

Incorrect removals

• always check removal list carefully when performing a dist-upgrade
• update-manager has a list of known-good removals, but this may not be up to date
• "Following essential packages will be removed"
  • never say yes unless you are well beyond needing this talk
• packages that have been removed from the archive are usually fair game to remove
• check package states with apt-cache policy PACKAGE

Administrative error

• forcibly installed package that causes problems
  • sudo dpkg --remove foo; sudo apt-get -f install
• packaging system files were modified by hand
  • reinstall packages and next time use dpkg-divert or dpkg-statoverride as necessary
• packages from third-party archives
  • we don't deliberately break them, but sometimes it's unavoidable or unnoticed
  • consider removing them to make the upgrade finish, and reinstall later
  • report bugs to third party in question

Crashes during upgrades

• try to resist the urge to pull the power cord during upgrades, but if you must...
  • sudo dpkg --configure -a
  • sudo apt-get -f install
• "Package is in a very bad inconsistent state"
  • dpkg --unpack /var/cache/apt/archives/foo.deb
  • dpkg --configure -a
• /var/lib/dpkg/status is critically important
  • copies in /var/lib/dpkg/status-old and /var/backups/
• /var/lib/dpkg/available corrupt
  • sudo dselect update
• files filled with zero bytes

  • XFS does this if you're unlucky; get a UPS

• corrupt files belonging to packages
  • dpkg -S /path/to/file
  • sudo apt-get install --reinstall PACKAGE
• /var/cache/apt files can be removed if corrupt
  • debconf database files can sort of be removed if corrupt, but you may have to reinstall packages and re-answer questions afterwards
• sudo dpkg-reconfigure -plow PACKAGE

• Links: DebuggingUpdateManager

Hardware-specific breakage

• most important area to report bugs
  • Not everyone has your hardware, so it can be a challenge to test all combinations.

Kernel breakage

• new kernel (or initramfs) breaks on boot
  • always keep at least one old known-good kernel
  • kernel ABI changes frequently in development branches so you will generally be able to boot an old one
    • linux-image-2.6.22-8-generic 2.6.22-8.18
• kernel seems to work, but oopses later
  • be cautious (is the filesystem oopsing?)
  • report a bug and consider reverting to an older kernel
  • include full oops log in /var/log/kern.log and/or dmesg

• Links: KernelTeamBugPolicies, DebuggingKernelOops, DebuggingKernelSuspend

X.org breakage

• log in at console (Ctrl-Alt-F1)
• can generally fall back to vesa or vga driver

  • edit /etc/X11/xorg.conf, e.g.:

    Section “Device”
        Driver “vesa”
    EndSection

  • restart X.org
    • since Karmic:

      • sudo restart gdm

      • or more generally:

        sudo restart $(cat /etc/X11/default-display-manager | awk -F/ '{print $NF}')

    • before Karmic:

      • sudo /etc/init.d/gdm restart

      • or more generally:

        sudo /etc/init.d/$(cat /etc/X11/default-display-manager | awk -F/ '{print $NF}') restart

• should be less of a problem in future (bullet-proof-x), with automatic vesa/vga fallback

• Links: X/Debugging

Network breakage

• network-manager takes down the network interface you were using
  • interaction with /etc/network/interfaces is hard
  • may require manual prodding (ifconfig, ifup/ifdown)
  • if want you, you can stop network-manager
    • sudo /etc/dbus-1/event.d/26NetworkManagerDispatcher stop
    • sudo /etc/dbus-1/event.d/25NetworkManager stop

• Links: DebuggingNetworkManager

Unpredictable bugs

• almost anything can happen in theory
• keep a login session open in case authentication breaks
  • check /var/log/auth.log for authentication problems
• may need to reboot in recovery mode to fix
  • in a pinch, (e)dit the grub menu item and use this for the kernel command-line arguments instead of “ro”:
    • rw init=/bin/sh
• report any library crashes or compilation breakage

• Links: DebuggingProcedures, Bugs/FindRightPackage, DebuggingProgramCrash, DebuggingSystemCrash

Remote upgrades

• <broken record>not a good idea with development branches</broken record>

• keep an ssh session open
  • make sure you can still log in before closing it
  • may still lose the battle if network interfaces go away

• if you need to reboot remotely, invest in a remote console server

Comments

• There seems to be widely varying views on how exactly one should upgrade during the dev releases. I think it would be beneficial to have a common "do it this way" method we can tell people to use as opposed to "someone at the lug told me to use aptitude, some other guy told me synaptic, and this other person told me to use apt-get." -- jorge 2009-09-15 21:48:21

I concur as well. When examinig the forums I found that people who were testing the development releases were new and in-experienced with testing. It should be noted that as time goes on more and more people will be turning towards Ubuntu and so in-experienced users will be answering more questions than before. An information database of some sort should be compiled such that a tester or inexperineced person can use it to report a problem.

For example: One tester for Karmic said "I installed Karmic update and now I have no sound. What should I try to do?"

The information database would very much help in this respect by telling the person to type: lspci|grep -i audio which would display hardware info. In addition it would also check to see if mute is enabled as some updates/releases cause programs to set to mute when not expected.

I.E. at the end of the day it would be nice to have a database of commands which a tester should know in order to figure out or fix his problem. In-experienced people find it difficult to come to terms with commands they dont know or have never heard of before. nolanking <<Saturday, October 17 2009: 01:12)>>

CategoryUbuntuDevelopment