## page was renamed from UserDaysTeam/maverick/UserDays/01292011/How to fix a broken machine {{{#!IRC [14:01] up next is popey... [14:01] popey needs a couple minutes, so we will start shortly === starcraft is now known as starcraftman [14:06] Hello [14:07] == Introduction == [14:07] Hello, I am Alan Pope. I'm a member of my LoCo Team in the UK where we make a little podcast . I'm also on the Ubuntu Community Council, LoCo Council and EMEA Membership Board. [14:07] I try to give people support in #ubuntu and #ubuntu-uk and on http://askubuntu.com/ when I get time. [14:08] This session is called "How to fix a broken machine". [14:08] The target audience for this session is 'People who may at some point experience a broken Ubuntu machine' - which lets face it is potentially everyone :) [14:08] The goal of the session is to empower people to fix their own computer when something goes wrong. [14:08] What do I mean by "goes wrong" and "broken"? [14:09] The types of issues I am thinking of are the 'potential show-stoppers' which could be described using phrases like:- [14:09] "When I turn my computer on I get..." [14:09] "... no desktop or logon screen!" [14:09] "... a black screen!" [14:09] "... some crazy text I don't understand!" [14:09] Has this ever happened to you? [14:09] (It has to me, on more occasions than I can remember!) [14:10] == What I will cover == [14:10] * An overview of the boot process [14:10] * What goes wrong [14:10] * Toolbox contents [14:10] * Diagnosing issues [14:10] * Solving some problems that occur [14:10] I'm happy to take questions any time :D [14:10] == What I can't cover == [14:10] * Installations of Ubuntu using WUBI (Windows on Ubuntu Installer) [14:10] - Whilst many bits are similar between a 'bare metal' install of Ubuntu and one done inside Windows, I've not had enough experience of it to speak about it confidently [14:11] == WARNING! == [14:11] Despite this being a 'user day' I will be using the command line! [14:11] The reason for this is that when there are show-stopping bugs which prevent you getting a GUI up and running, the command line is the best way to fix the issue. [14:11] Note: I do not consider 'reinstall Ubuntu' as a fix, it's a work-around. [14:11] Ok, so lets start.. [14:11] = An overview of the boot process = [14:11] (simplified) [14:11] (massively) [14:12] So what actually happens on a standard installation of Ubuntu on a pretty normal Intel/AMD based PC? [14:12] (the reason I say Intel/AMD based PC is because things are a little different on ARM, Phones, Macs etc) [14:12] Right after you power on, a bunch of checks are done, hardware is looked for and eventually the BIOS (chip on the motherboard) does some hardware checks called POST (Power On Self Test) [14:12] If those pass (and your BIOS is configured correctly) then the BIOS will then go looking for a device to boot from. [14:13] describes that process in a little more detail. [14:13] [14:13] Under normal circumstances you can boot from:- [14:13] - An internal hard disk (be that of a spinning variety or a new fangled SSD (Solid State Drive) [14:13] - An external device attached via USB [14:13] - Some kind of optical device (like a CD or DVD) [14:13] - A network server - often called PXE (Pre-boot eXecution Environment) [14:14] - Some other magic from the future that hasn't been invented yet. [14:14] [14:14] Lets assume that it's booting from a local hard disk which happens to contain Ubuntu. [14:14] [14:14] The BIOS will look for an MBR (Master Boot Record) which is located at the start of the disk. We call this 'Stage 1' [14:14] The MBR is pretty small and contains enough program code to find, load and execute the next part of the boot process. [14:14] The program used on Ubuntu is called GRUB (Grand Unified Bootloader), and we call this 'Stage 2'. [14:15] Optionally at this point we may get a menu displayed by GRUB. If no menu appears then it can be triggered by pressing and holding down the [SHIFT] key as GRUB loads. [14:15] (the menu is configured based on files in /boot/grub/ - specifically /boot/grub/grub.cfg) [14:15] (older versions of GRUB in previous releases of Ubuntu used a different file) [14:15] The menu contains a list of operating systems to boot from. If one is selected from the menu then it will continue the boot process.. [14:15] The next step is to load the operating system selected in GRUB, this is usually a two-stage operation. [14:16] (well, technically it's a many many stage operation, but I'm simplifying remember) :) [14:16] First a small Linux Kernel is loaded which is called an 'Initial RAMdisk' (although you may see it called 'initrd' or 'initramfs'. [14:16] Secondly the actual Linux Kernel which will be used by your desktop/laptop is loaded. The Kernel has drivers for many things, and will go through its initialisation process loading whatever drivers are necessary. [14:16] Some are built into the Kernel image when it's compiled by the developers. [14:16] Some are 'modules' which the Kernel can load in and run after it's started running. [14:17] (the kernel usually figures out what modules to load, but sometimes it needs a little help) [14:17] Once the Kernel is up and running it hands over to a program called 'init' (short for Initialisation). On Ubuntu 'init' is provided by a package called 'Upstart'. [14:17] init/upstart then takes care of loading all the subsequent programs required to get the desktop up and running. [14:17] (which you might see listed in places like /etc/rc2.d/ ) [14:18] At some point during that last process GDM (GNOME Display Manager) will load. This will present the logon screen (or not if you chose to skip the logon screen). === xindz`off is now known as xindz [14:18] (KDE and XFCE use other display managers like KDM) [14:18] Once you have logged in your desktop starts. [14:18] [14:18] So thats BIOS -> MBR -> Stage 1 -> Stage 2 (GRUB) -> Initial RAMdisk -> Kernel -> init (Upstart) -> GDM -> Desktop [14:18] (roughly speaking) [14:19] As I said, massively simplified. If you'd like to know more about the boot process I'd recommend reading this page (which is quite technical, but interesting):- [14:19] http://www.ibm.com/developerworks/linux/library/l-linuxboot/ [14:19] [14:19] You can find out more about GRUB at:- [14:19] http://en.wikipedia.org/wiki/GNU_GRUB [14:19] [14:19] You can find out more about Upstart at:- [14:19] http://upstart.ubuntu.com/ [14:19] [14:19] Any questions about that boot process? [14:20] [14:20] = What goes wrong? = [14:20] So now you know that the boot process consists of a bunch of parts linked together, lets see where it can all fall apart! [14:20] [14:20] == Hardware Failure == [14:20] Commonly forgotten about when diagnosing issues with Ubuntu is that it's possible for hardware to fail. Some ways in which hardware can fail and what the result could be:- [14:20] * Failing RAM [14:20] - Unfortunately this can cause all kinds of issues which are often hard to pinpoint [14:21] - Machine may not boot at all, it may boot to the BIOS and no further, or it may get part way through booting and then lock-up, shutdown or reboot. [14:21] [14:21] * Failing Disk [14:21] - When a disk is failing it can also look like files are corrupted, so can look like a piece of software has gone wrong [14:21] - Machine may not boot at all beyond the BIOS, or it may get part way through and (as with RAM issues) fail in sporadic or spectacular ways [14:21] [14:21] * Failing CPU [14:21] - In my experience least likely, but can happen. More likely is failing to cool the CPU properly, so this could mean a failed heatsink/fan or broken down thermal paste between the CPU and heatsink. [14:22] - Usually this results in a failure to boot at all, beeps during the POST or random crashes [14:22] [14:22] * Failing GPU (Video Card) [14:22] - If completely failed then this can cause a machine to not boot, but beep in a cryptic but documented way when the video card is tested during boot up. [14:22] - This often manifests itself as a blank screen or more commonly as colourful screen corruption, which may last all the way from the BIOS screen through to GDM. [14:22] [14:22] Ok, so that's a few ways in which hardware can fail, lets move on to see other ways we can fail. [14:23] [14:23] = BIOS = [14:23] * Can't get past the BIOS boot screen [14:23] * BIOS boots off wrong device [14:23] [14:23] == GRUB == [14:23] * Can't boot past stage 1 or 2, but get a blank screen, or perhaps the GRUB menu, but can't get past it. [14:23] [14:24] == initramdisk == [14:24] * Get dropped to an (initramfs) prompt [14:24] [14:24] All of which makes a user into a sad puppy :( [14:24] [14:24] == Kernel == [14:24] * Kernel panic on boot [14:24] * Kernel cannot find root file system [14:24] * Graphics driver no loading [14:25] [14:25] ..and so on. So there's a large and diverse number of ways in which the system can fail to boot. Lets move on to look at some tools we can use to diagnose these issues. [14:25] [14:25] == Toolbox Contents == [14:26] Here's some things that are useful to have when diagnosing issues! [14:26] * Ubuntu Live CD [14:26] - Essential because it contains the following great diagnostic utilities [14:26] - Memtest, GRUB, GParted, a web browser in which you can go and find answers to all your questions :D -> see http://askubuntu.com/ [14:26] [14:26] * Ubuntu Live USB Key [14:26] - Create using USB creator, UNetbootin, or by installing onto USB from CDROM (or other USB stick) [14:26] - Useful for when you have netbook or other machine which has no optical drive [14:27] - You can install from CD onto USB stick which means you can boot from it and add additional tools / utilities to the stick. [14:27] I personally carry round a 32GB usb stick which has Ubuntu installed on it, and from there can diagnose issues with a hard disk based install. Very handy! [14:27] * Ethernet Cable [14:27] - For when wireless messes up :( [14:27] * USB Hard disk [14:27] - Useful for backing up the system [14:28] * A backup of your existing system! <- most essential! [14:28] * A backup of any passwords / encryption keys which may have been used to install the system [14:28] [14:29] == Diagnosing Issues == [14:29] So given we now know the steps of the boot process we can identify potentially where issues can occur. First off I'd start by asking some questions:- [14:29] * What has changed since the system worked? [14:29] often the answer from the user is "nothing" and "I never touched it" [14:30] (until you shine a light in their face and threaten them) [14:30] So this could include:- [14:30] - Hardware added / taken away / moved ? [14:31] e.g. adding a new USB device on my desktop caused it not to boot! [14:31] (the USB device was an Android phone) :( [14:31] - Software updates, drivers added, new packages installed, packages removed? [14:31] e.g. installing the latest and greatest applications you saw on omgubuntu! :D [14:32] [14:32] * What has been done differently today than on other days? [14:32] Maybe today the user didn't plug in their external screen, but usually they do.. [14:33] Any little change can potenially cause an issue. [14:33] [14:33] The final question:- [14:33] * Do I have a backup of all my data and passwords/keys? [14:33] Because it's entirely possible that the system is irreversibly broken, and if that's the case then new hardware or a reinstall may be the _only_ option. [14:33] But I'd like to hope we can fix most things without a reinstall. [14:33] [14:34] == Solving Problems That Occur == [14:34] [14:34] Ok, so it's impossible in the next 30 minutes for me to go through every single possible scenario of what might go wrong :D [14:34] But I can pick out a couple that are quite common [14:35] And I can explain how to get your environment setup so you can diagnose these issues [14:35] [14:35] The main thing that's useful to do when you have a completely unbootable system, is to boot to another install and inspect the contents of the broken system [14:35] That can be done in at least two ways:- [14:35] [14:36] 1. Pull the disk out of the broken machine and put it in another machine as a slave disk [14:36] (this is actually impossible in some systems like the Asus Eee 900 where the SSD 'disk' is soldered onto the motherboard) [14:37] (this is also unpleasant to do on many laptops because you have a bazillion screws to undo and little ribbon cables that tear easily) [14:37] So that's why I often advocate option 2.. [14:37] 2. Boot from an Ubuntu Live CD / USB stick. [14:37] (hence me saying you should always carry one) :D [14:37] [14:38] Now, booting from a Live CD/USB stick is great, and can let you inspect the contents of the hard disk of the broken machine [14:38] But you often need to do more than look. [14:38] [14:38] In the same way that a Doctor might start by giving a patient a non-invasive X-Ray or MRI scan [14:39] Sometimes you need to slice people open in order to figure out what's wrong [14:39] Note: I do not advocate slicing people open randomly :) [14:39] [14:39] So the diagnostic tool I'm going to talk about is using a Live CD/USB with a program called "chroot" [14:40] The main use for chroot here is to attach to / login to, an existing install of Ubuntu and then look, and optionally make changes where needed. [14:40] [14:40] This is a slightly complex process which involves a lot of commands on the command line:- [14:40] Here's what I do:- [14:41] 1. Boot from a Live CD/USB [14:41] 2. Use "sudo fdisk -l" to identify which device my local hard disk is [14:42] http://paste.ubuntu.com/559909/ <- that is the result of me running that command on my PC [14:42] As you can see I have 3 disks on this computer. [14:42] /dev/sda is a 500GB disk [14:42] /dev/sdb is a 250GB disk [14:42] /dev/sdg is a 32GB USB stick [14:43] Now looking closer at /dev/sda we can see multiple partitions:- [14:43] /dev/sda1 * 1 24832 199463008+ 7 HPFS/NTFS [14:43] ^^ that one contains Windows XP [14:43] So I am not interested in that for this session :) [14:43] /dev/sda2 32251 60802 229338113 5 Extended [14:43] ^^ this is an extended partition, it contains the next three partitions. [14:44] (note: there is a limit of 4 primary partitions on disks, so we tend to have lots of extra partitions under an Extended one, so we can have more than 4) [14:44] /dev/sda5 32251 59639 220001280 83 Linux [14:45] thats a biggie! [14:45] That's my root partition, which contains everything on my pc. [14:45] Some people separate / and /home as separate partitions, I haven't on this PC. [14:45] /dev/sda6 59640 60802 9335808 82 Linux swap / Solaris [14:45] that's my swap space. [14:46] Ok, so assuming that /dev/sda5 is my root partition and it's in some way broken. [14:46] Here's what we move on to in step 3. [14:46] 3. Open a terminal [14:46] Now this may be scary for some, but it's really the best way to diagnose these issues [14:46] 4. Issue the following commands:- [14:46] (assuming /dev/sda5 is indeed the broken root partition) [14:47] Now, remember at this point we're booted into a Live USB/CD environment. [14:47] We need to make some directories under which we will "mount" the broken installation. [14:47] mkdir ~/target [14:47] mkdir ~/target/dev [14:47] mkdir ~/target/sys [14:47] mkdir ~/target/proc [14:47] ~ means "my home directory", so all of those will be in the home directory of the Live environment user [14:47] (the above are going to be used to 'mount up' the broken install into) [14:47] Now we mount up:- [14:48] sudo mount /dev/sda5 ~/target [14:48] (the above mounts up the hard disk into onto the target directory, you can see your files in there) [14:48] So you could open nautilus on the Live CD/USB and navigate to the home directory, and under 'target/' you'll see your files (hopefully) [14:48] (the next three lines mount up some special folders that may be needed by the diagnostic tools) [14:48] sudo mount -o bind /dev ~/target/dev [14:48] sudo mount -o bind /proc ~/target/proc [14:48] sudo mount -o bind /sys ~/target/sys [14:49] (At this point we have the local install mounted up in a folder) [14:49] Now we "chroot" into it.. [14:49] 5. 'chroot' into the install with the following command:- [14:49] chroot ~/target [14:50] actually, sorry. [14:50] sudo chroot ~/target [14:50] (at this point we'll get a root prompt inside the 'broken' system and can start poking around looking for problems). [14:50] [14:50] This is the single most useful diagnostics system (in my opinion) for troubleshooting a broken system. [14:51] Now we're running low on time, so I'll go through an example of a specific issue that can be fixed once we have the sytem in this state. [14:51] There are 10 minutes remaining in the current session. [14:51] [14:51] === Fixing broken GRUB === [14:52] If GRUB has become broken we can at this point check the configuration of GRUB by looking in ~/target/boot/grub/grub.cfg [14:52] Here is my grub.cfg http://paste.ubuntu.com/559911/ [14:52] Note at the top:- "DO NOT EDIT THIS FILE" [14:53] However it is 'interesting' to look at === MichealH is now known as MH0 [14:54] So fixing all grub issues is beyond the time we have here, but, now we have the system mounted up and chrooted we can perform additional steps which can help the diagnosis or indeed resolve the issue) [14:54] so for example at this point I would head over to the following pages:- [14:54] https://help.ubuntu.com/community/GrubHowto [14:54] (or if windows had rubbed out my grub install) :- [14:54] https://help.ubuntu.com/community/RecoveringUbuntuAfterInstallingWindows [14:55] [14:55] The massive stumbling block many people have is a chicken-and-egg problem that goes like this:- [14:55] 1. My system wont boot, grub is broken [14:56] 2. To fix grub, boot your system and run these commands.. [14:56] 3. Goto 1. [14:56] [14:56] There are 5 minutes remaining in the current session. [14:56] By using the chroot method above, we can circumvent the "my system wont boot" and get logged into the system, bypassing the broken grub / kernel / video issue that's preventing bootup in the first place [14:56] [14:57] I'll very quickly mention another issue that this can help fix.. [14:57] [14:57] === Fixing a half-finished upgrade/update === [14:57] If the system powered down or crashed during system update, sometimes this can render the system unusable until the updates are 'finished'. Some commands which can help here:- [14:57] (assuming you're already chrooted in) [14:57] dpkg --configure -a [14:57] (will continue processing outstanding updates) [14:57] and.. [14:57] apt-get dist-upgrade [14:58] (which will be useful if some updates need to be installed) [14:58] Note: Both of those commands, I did not use sudo. [14:58] The reason for that is because when you "chroot" you "become root" on the system. this is of course dangerous, so be careful. [14:58] [14:59] Hmm, I'm getting to the end of my slot, and we don't really have any time left for questions, sorry about that. [14:59] I'm hoping to make a video about this which I'll put up online at http://ucasts.tv/ [14:59] feel free to ping me on irc if you have questions later :) [15:00] Apparently the next session is blank for some reason. Free period! [15:01] thanks to everyone for watching me wibble on, and thanks to the classroom team! [15:01] Logs for this session will be available at http://irclogs.ubuntu.com/2011/01/29/%23ubuntu-classroom.html [15:01] thanks ClassBot ! }}}