Edgy

Revision 38 as of 2006-10-20 04:44:30

Clear message

This is now an accepted Ubuntu Summer of Code project. See [http://code.google.com/soc/ubuntu/appinfo.html?csaid=9FB004A7A81E0B27 the application page] for more information. Progress on this spec has been made, see [http://www.evalicious.com/blog my website] for more information.

Summary

Design and develop import functionality for the Ubiquity LiveCD installation program and debian-installer that allows the user to transfer existing user accounts as well as the settings and files that go with each individual account.

Rationale

The reality for most potential Ubuntu users is that they've bought their computer with Windows pre-installed or have already installed another Linux distribution. These users have probably already used their existing operating system for a decent period of time and have accumulated documents, email, settings, and other files that can be imported and translated for use in Ubuntu, but the existing Ubuntu graphical installer mercilessly destroys them. If users could install Ubuntu knowing that all of their existing Office documents would be placed in their Documents folder, their email would be imported into Evolution, and their bookmarks imported into Firefox, they would have much more incentive to make the switch, be it from Windows or another Linux distribution.

Use cases

  • Andrew is a long time Gentoo user, tired of having to manually configure his system, and wants a distribution that "just works". He'd like to copy over his home directory to the new system when he installs Ubuntu.
  • Dave is a busy parent who has been using Windows XP for the past year, and he and his children have accumulated a large number of pictures, music, documents, and bookmarks between their four user accounts. He'd like to be able to copy over all of the user accounts while selecting what files to keep and what to throw away when installing Ubuntu.
  • Joe is a CAD engineer who would like to switch his home computer to Ubuntu. While Ubuntu does not have software to read Autodesk files, he would nevertheless like to save his set. He does not have a CD burner and the files are too big for email attachments.

Scope

As the functionality will exist in both Ubiquity and debian-installer, it will affect both of those packages.

Design

+----------------------------------------+
|                                        |
| Would you like to import documents and |
| settings from your previous operating  |
| system?                                |
|                                        |
|  (o) Yes                               |
|          +---+                         |
|      [X] |   | Evan                    |
|          +---+                         |
|                                        |
|          +---+                         |
|      [X] | ? | Erica                   |
|          +---+                         |
|                                        |
|  ( ) No                                |
|                                        |
|                      [ Back ] [ Next ] |
|                                        |
+----------------------------------------+

If Ubuntu is being installed alongside another operating system, the above dialog is presented to the user. The user pictures are grabbed from the previous operating system. If a picture cannot be found, the no-picture-found picture is used. 

+----------------------------------------+
|                                        |
|              +---+                     |
|              |   |  Evan               |
|              +---+                     |
|                                        |
| Select the items you would like to     |
| import for this user.                  |
|                                        |
| +------------------------------------+ |
| |                                 |/\| |
| |   [X] My Documents              |  | |
| |   [X] My Music                  |  | |
| | - [X] Settings                  |  | |
| |       [X] AOL Instant Messenger |  | |
| |       [X] IE Bookmarks          |  | |
| |       [X] Firefox Bookmarks     |  | |
| |                                 |\/| |
| +------------------------------------+ |
|         [ Select All ] [ Select None ] |
|                                        |
|                                        |
| Type in the full name, username, and   |
| password for this account below.       |
|                                        |
| Full Name: [ Evan Dandrea            ] |
| User Name: [ evand                   ] |
| Password:  [                         ] |
| Confirm:   [                         ] |
|                                        |
| [ ] Administrator Account              |
|                                        |
|                                        |
|                      [ Back ] [ Next ] |
|                                        |
+----------------------------------------+

The user is presented with the above page for each user they selected to import. Again, the user picture is used along with the user name to clarify exactly which account the user is dealing with.

The user is presented with groupings of objects that they can select for import. All objects are selected by default.

On the same screen the user will fill out the account information for the user they're importing. In the case of Windows the Full Name and User Name fields will be automatically filled. In the case of Linux the password and confirm fields are filled with a set number of asterisks.

If the user checks the Administrator Account checkbox, the account will be added to sudoers.

Implementation

The import function will search the hard drive on which it is being installed for user accounts as well as the documents and settings that go along with each account, presenting its findings in a screen for each discovered user and allowing the user of the installer to select exactly which accounts and groupings of items they would like to import, with everything selected by default.

Configurations will be translated into their appropriate Ubuntu equivelent. AOL Instant Messenger accounts, for example, will be added to a .gaim/accounts.xml file in that user's home directory.

The transferring of data will be done by means of copying the files and settings the user has selected to the new Ubuntu filesystem after the LiveCD filesystem is copied.

Data:

If Ubuntu is being installed over a Linux system, the import function will select all of the home directories as well as the matching sections of /etc/passwd and /etc/shadow. However, if Ubuntu is being installed over Windows, the import tool will select each user from Documents and Settings, their My Documents folder, and what other files and settings it's preprogrammed to look for. As the development of the import function progresses, this should include:

Files:

  • Documents
  • Pictures
  • Email messages
  • Fonts
  • Music

Settings:

  • User accounts
  • Email clients
  • Instant messaging accounts
  • Bookmarks
  • Desktop background
  • Address book

Documentation

Documentation for the import functionality in Ubiquity will be written and submitted to the Documentation Team for review.

Outstanding issues

Comments

  • If Windows XP does not make a user an administrator by default, we should automatically check the “Make an Administrator” box in the interface for the accounts that are.
    • The first user created upon installation in Windows XP Home is an administrator
    • Note: Upon the release of Vista, User Access Mode will be configured so the default user will run with lowest privilges so this will change on the next version of Windows
  • Do we ignore the root account or do we only import it if /root has files in it? If we import it, we break the expected functionality of having to use sudo for root access.

    • On the other hand such users are probably in the habit of doing "sudo bash" or "sudo su -" so it may make little difference. -- SamLiddicott

  • Migrating user data from Windows means we'd have to detect for existing Windows partitions, mount the appropriate ones (a user could have several windows drives and only one of them could contains the Documents and settings folder. Also, most of the relevant data a windows user doesn't directly translates to linux settings, we would probably be able only to copy user data files, and import settings from programs that represent them the same on the filesystem be it a windows or linux file system.

    • Detecting Windows filesystems is easy with the aid of os-prober. -- ColinWatson

Issues regarding file/settings transfer in the case of "messy" windows installations.

  • Many XP-installations I have seen do not use seperate accounts for the users, or they have multiple, but devided along the lines of "dad-work" and "family". There is also the not-often-used "guest" account. User accounts should be created for each user in this case.
  • Users might not use "my documents" - on all systems I have ever installed, I've used a seperate partition (like D:/data). If you promise to import files and settings, how will you deal with files that have no obvious owner? A UI might be presented asking what to import to which user account.

    • Sometimes "My Documents" is NOT in c:\Application Data, its a shell folder, you MUST ask the registry where it actually is stored, you must also know the system mount points so that you can track down this location -- SamLiddicott

      • The Windows partition naming system is completely broken. C:\ doesn't even have to be the root partition, most of the names are determined at boot time (http://en.wikipedia.org/wiki/Drive_letter_assignment). Does anybody know of a project that can name drive letters from Linux? I couldn't find anywhere if this information is stored in the registry... --SamuelCormierIijima

    • Sometimes when multiple users share the same account, they have different Outlook Express or Thunderbird profiles, sometimes they have different accounts defined in the same profile, although outlook express user switching may be a carry over from Win95 that isn't used anymore? -- SamLiddicott

      • The UI will indeed ask the user which Ubuntu account they want to import the old operating system's account into, creating the new account if it does not already exist. -- EvanDandrea

  • On many pc's, all windows accounts are administrators, due to conviencience or lack of understanding of a proper permissions system.

The point here is that many windows pc's are messy when compared to a properly set up ubuntu installation, and ways should be found to import the files/settings for user convenience, but not the mess, or users will end up with an ubuntu system as unusable and unstable as their XP experience ever was. This is partially design, partially education. --HiddeBrugmans

More discussion

I've been working on a small program that does something like this. Right now it just imports IE bookmarks, check it out here: ["MigrationAssistance/Script"]. Pretty self explanatory, and its just a prototype.

One question: If a user has so many documents that a) it would take too long to copy to the new ubuntu partition or b) the partition isn't big enough, could we make a symbolic link? Thats what I did for my mom, I made her a "My Windows Documents" link on her desktop :-). The only problem is that if its NTFS its read only (until fuse becomes default)... --SamuelCormierIijima

  • I think it would be better to calculate how much space is required before-hand and prompt the user to remove some items to import or make the Ubuntu partition larger. As you mention, NTFS is read-only and symlinking would introduce unexpected problems. -- EvanDandrea

KillerKiwi - If the user has xp installed then could not some 'legal issue' software be safetly install? For instance MS Fonts, Mp3 codecs? As these things are licensed under windows xp? An option to copy over the users wallpaper would also be good, (The most common customization of any system)

  • Ok, I've updated the script. It now lets you choose if you want to import your background... Can anybody test? Smile :-) --SamuelCormierIijima

  • Alright, yet another update, you can now also import firefox bookmarks from windows Smile :-)

One more idea: suggest and install software as alternatives to programs the user has installed under windows.

Hey, I read on Colin's blog that you're doing this for Google SoC, congrats :-). I was wondering though, is it feasible to have the migration wizard in ubiquity? Especially for importing things like bookmarks and such, I'm not sure if the Firefox profiles are even created then, how would you go about working that out? Keep me posted, I'm interested in how this turns out :-). Good luck, --SamuelCormierIijima

  • Yup, there will be both debian-installer and Ubiquity front-ends to the d-i backend. Thanks! -- EvanDandrea

Only at install time? (Dual boot Q) Presuming this would be part of the live CD, would it not be possible to have a version that can run from Ubuntu installations, asking "Do you want to search other partitions for compatible Operating Systems and attempt to import (or even better, merge, in the case of bookmarks & mail) items from that OS?"

  • This is exactly what it will do. It also merges wherever necessary (Gaim, AIM, and Yahoo! IM all going into Gaim). -- EvanDandrea

I dual boot XP and Dapper, simply because I need XP sometimes, but for users who are dual booting & happy to switch it would be useful to have the option after install - though again the disk space requirements may be near impossible to resolve on some systems. Ideally GParted would be able to resize, merge & even convert partitions to achieve it, or perhaps to replace a dual boot Windows partition & migrate all at once the script could dump large files to an external USB drive or even create CD/DVDs of selected files, then remove the NTFS partition & create an new EXT3 one with the larger folders in it (symlinked?)?

  • As it stands right now, resizing partitions can often be dangerous and therefore migration-assistant will only copy documents and settings over for the time being. -- EvanDandrea

Also I second the call for suggesting equivalent programs, but I would go further and suggest generating a text file or script to sudo apt-get any obvious requirements other than applications, such as media codecs. As for things (FF profiles?) that might not exist when first installing Ubuntu, how about zip- (or any format-) archiving required files then having Ubiquity able to process them when FF first launches?

  • There has been much discussion about installing extra codecs during the main install in Ubuntu. I don't have time to dig up the link right now, but check some of the Ubuntu Developers Conference specs. -- EvanDandrea

It all sounds like a great project, well done on the Google SOC support! -CDE

You should get the location & name of user accounts & directories from the registry, not by guessing from the filesystem! It should also be possible to find the default applications that way (e.g. firefox, opera or IE, if all of them are installed). And probably other important information too (e.g. location of the "mail database").

[http://prdownloads.sourceforge.net/windowsir/regp_1_1.zip?download This script] and the source of [http://home.eunet.no/pnordahl/ntpasswd/ this program] could be useful.

-- -- JanClaeys DateTime(2006-07-05T09:26:37Z)

  • This is already being done in the Python version. I rewrote a part of it in Python, but I'm thinking about going back to the C version and using SWIG and family. See https://svn.bountysource.com/migrate --SamuelCormierIijima

  • Indeed, registry viewing code has been written for all of the bits of information we'll need to grab. -- EvanDandrea

  • Probably a much better way is to let a windows program extract that information because there are so many rules and exceptions to all that Windows stuff (and these are merely the ones that I've seen) that it's probably best to query WMI in a windows program, dump the information to disk and read that file in the installer. That way you can also dump the user email from Outlook and Outlook Express using their respective COM interfaces. This should be much more reliable than having to figure out how the file formats these programs use actually work (I'd expect them to change with every release, anyway). I know that this messes up the current design but it's the only reliable way I can think of. Oh, and keep in mind that some parts of the Windows user database are in a branch of the registry that is only stored in an encrypted file. [http://www2.futureware.at/~gregor/LMA-20060707.tar.gz This] might be useful proof of concept code. In it's current form it requires Python and win32libs to be installed to run (both could be provided as a part of a stand-alone bundle). -- GregorMückl

    • There is no way I'm accepting Windows code into the default installation environment, and I would like migration-assistant to be part of the default installation environment. I'd rather avoid Windows code even if it means the result is a little less reliable; it can always be made more reliable over time. -- ColinWatson

  • So far the registry has been able to give all the information the program needs. -- EvanDandrea

-- What about stuff like: - Language Settings - Screensaver settings - Regional Settings

If you could retrieve that information the installation process can be simplified some more, since you don't need to request it.

  • On the face of it importing the i18n settings sounds like a pretty good idea, but I have two concerns. First, the language selection occurs as the first step in the installer, long before migration-assistant runs. Second, if we were to preseed the installer with one of the previous operating system's language settings, we would be assuming that is what the user actually wants. Since we've already answered the question, the installer will not prompt them to make sure, which would kill off the benefit of reducing the size of the installer by a screen, and they wont have the opportunity to change their mind. -- EvanDandrea

  • Screensavers, on the other hand, are actually Windows binaries, and wouldn't be runnable without WINE. Even then we wouldn't be able to assure that they'd work in WINE. -- EvanDandrea

  • Since screensavers are executable, they're also a common vector of spyware on Windows systems. I'd be pretty wary of automatically migrating them under any circumstances. -- ColinWatson

Great project guys. -Nojlib2

-- PeterWhittaker, 2006/08/02

Are there particular reasons for limiting this to XP? Since XP is supported by MS, it might be more relevant to include/address earlier OS's, e.g., '98, '98 SE, Me, etc. I "upgraded" my laptop to Ubuntu from 2000; my home machine will eventually "upgrade" from '98 SE.

Also, email support should include Outlook. You may wish to consult http://outport.sourceforge.net as a starting point. (On our home machine, we had both Outlook and Outlook Express installed; on my laptop, Outlook only, Express never made it off the CD.) There are also tools for reading email passwords from Outlook, Express, etc., but I do not know if there are FLOSS variants.

  • Absolutely. Both Outlook and Outlook Express will be supported. Thanks for the link. From what I remember of my research, both programs store most settings in the registry. I think OE poorly encrypts passwords and puts them in the registry as well. So for pulling the accounts it should be a somewhat simple matter of translating the registry keys to gconf keys. -- EvanDandrea

Have you seen [http://versora.com/products/progression_desktop_linux.php Versora Progression Desktop] ? They seem to do everything we need (Not for free, of course). It could be a source of inspiration, anyway. -- AlfonsoDeCala

  • Future work: From an implementation point of view it's easiest to migrate documents and settings for someone who's dual-booting. But from a user's point of view, this is a lot less useful than migrating documents and settings for someone who won't have their Windows partition to read from. For these people, the installer should offer to copy their files to CDs, DVDs, or an external drive before blanking the Windows partition. -- MatthewPaulThomas

CategorySpec