GenBunToo

This idea has been languishing for a long time. Is anyone else who is subscribed to this page still interested in it? If so, please email me at oscartheduck@gmail.com

About GenBunToo AKA Ugentu

GenBunToo and Ugentu are a play on words for Ubuntu + Gentoo linux. The project will either be called one or the other, we are leaning toward Ugentu at the moment (which might be an african word meaning "some gentoo ideas inside of an ubuntu installation").

The project is an attempt to take the best ideas of Gentoo Linux and sort of mix them into the simplicity found in Ubuntu Linux without making the user do anything more complicated than what they do already.

Discussion

Please keep this wiki page clean by keeping discussion about Ugentu in this thread.

Status

Ugentu is not an official Ubuntu project; it is a hobby project of a few of the members of the UbuntuForums community. Currently the project is in planning and discussion stages. There is not even an alpha build for people to play with, so dont try apt-get'ing gen-bun-too or ugentu just yet.

Overview

A person who installs the ugentu package can use it to fetch the source for a package in the Ubuntu repository, build it with compile flags specifically optimized for his or her processor, and install it on their system as though it had been installed from the official binary package in the Ubuntu repositories.

Furthermore, a “sudoer” could issue a command that detected all the installed packages on his (or her) system and rebuilt them all from source (except for a few compile sensitive packages that would be left alone for stability reasons).

The first time this is done it could take an entire weekend to compile all those packages from source, but subsequent runs of the script would only compile new or updated packages so they would be significantly shorter.

An administrator could even use cron to run the system optimization script every night so that his (or her) system can be kept more or less in a continually optimized state with little or no interaction from the system maintainer.

An optional daemon will also be installed that will monitor the list of installed packages and optimize new ones as they are added / updated. There is currently some debate as to the need for a full blown daemon, becuase cron could be used as layed out above to run the system update script multiple times a day, resulting in much the same idea as what a daemon would do.

Examples

The package named "foo" could be installed/optimized by issuing the following command:

 sudo aptimize foo

All of the un-optimized packages on your system could be optimized by issuing the following command:

 sudo aptimize --system-update

Goals

This project wants to add optimized compilation to Ubuntu without adding all of the headaches and complexities that are common in source based distros.

We want the end user to have as small of a learning curve as possible.

We want it to be easy and hassle free to have a perpetually up to date optimized system, the sort of environment that gets set up once, but the benefits are reaped indefinitely.

We want the end resulting system to be a properly maintained Debian installation. In other words, we want to use dpkg to make sure our optimized packages are installed by the system in a way that it knows what is installed, and the normal apt-get / aptitude / synaptic system can still be used without sacrificing system stability.

We want users to be able to continue to use apt-get and synaptic, and after using thier install method of choice to be able to issue a simple command that optimizes all the new stuff.

We want people to be able to use the update-manager that comes with ubuntu as a way of staying up to date, and have this optimize the new packages with no extra effort from the user -- after it is setup.

Details

Installing the ugentu package would also install these packages...

build-essential fakeroot linux-kernel-headers

the package would include these files (explanations for each file can be found below)...

/usr/bin/ugentu/gcc
/usr/bin/aptimize
/usr/bin/aptimized
/usr/bin/ugentu-config
/usr/bin/ugentu-xconfig
/etc/ugentu/make.conf
/etc/ugentu/blacklist

/usr/bin/ugentu/gcc

a wrapper around the normal gcc that will be added to the $PATH in such a way that a normal call to gcc will actually call this wrapper application (we stole this idea from the apt-build project) which in turn will call the normal /usr/bin/gcc compiler to do the actual compiling, but first, it will substitute the compilation flags that would have been used with the new *aptimized* flags specified in /etc/ugentu/make.conf

/usr/bin/aptimize

sudo aptimize <package-name> needs to be functionally equivalent to sudo apt-get install <package-name> with the following two exceptions: The binaries will be optimized for your processor, and the -dev packages containing the header files needed to compile the source code will be installed as dependencies. **note, the installation of the -dev packages in a lasting manor is being discussed, and probably wont be a side effect.

apt-merge will probably be a bash script that does something like this...

$OLD_CWD = curdir?

cd /usr/src
apt-get update

# sorry if this is all wrong, I dont know dip about bash scripts, but I know what I want to have happen -- someone repair the syntax
# if $@ = "--system-update" goto SYSTEM_UPDATE    

for PACKAGE in $@ ;
do
    apt-get install $PACKAGE --yes 
#   if $PACKAGE is not in /etc/ugentu/blacklist then
      apt-get build-dep $PACKAGE --yes
      apt-get source $PACKAGE --build --yes
      dpkg --install *.deb
      rm *
#   fi
done

cd $OLD_CWD

#Exit script

SYSTEM_UPDATE:

for each PACKAGE in [All installed packages]
do
  if $PACKAGE is not already aptimized
      apt-get build-dep $PACKAGE --yes
      apt-get source $PACKAGE --build --yes
      dpkg --install *.deb
      rm *
  fi
done

cd $OLD_CWD

In English that means:

Download and install the binary version of the package, its dependencies, and the header files and libraries that are required to build the package from source.

Build the package (optimized by the gcc wrapper we put in place).

And install it.

then clean up after ourselves. by using pbuilder, all the extra -dev files are installed inside of a temporary folder and get flushed when we are done building. It removes the side effect of adding undesirable -dev packages, but it potentially takes longer (temporarily installing the same -dev packages over and over as it builds packages with similar -dev dependancies). We need to decide based on real world tests if it would be easier to determine which -dev packages are already installed, and then after the full script is run, remove any new ones that have been added as a side effect rather than use pbuilder. Once we know which is faster when many packages are aptimized at once, we can go with that method.

/usr/bin/aptimized

the aptimize-daemon can be run in the background, and it will do something similiar to aptimize --system-update, but as new packages are added or updated, it will just automatically start aptimizing them.

we are still in debate as to whether or not this daemon is worth the time writing if we can get cron to be a reasonably similar solution.

/usr/bin/ugentu-config

ugentu-config will be a simple script -- probably ncurses based, so it can be used on ubuntu-server's without X Windows -- that uses /proc/cpuinfo to detect the cpu type. Then generates a decent make.conf after asking the user to verify the auto-detection.

we may some day go so far as to add another config option: ugentu-xconfig that uses glade and python to make a pretty gui configuration alternative for people who are shell-shocked by text based dialogs.

/etc/ugentu/make.conf

a file that stores the CFLAGS that we want the ugentu/gcc wrapper to use when it runs the normal gcc compiler. Very similar to the way make.conf works in gentoo -- although the contents of the file will probably be much simpler.

/etc/ugentu/blacklist

This file will have the list of packages that should not be aptimized. This will include the following, and administrators can add packages to the blacklist on their particular system as needed:

the linux kernel
gcc-4.0
*-dbg

Packages that are discovered to not work properly when aptimized should be reported in the forum thread specified above so that the default blacklist installed with ugentu can include them, and so that the bugs can be looked into and resolved.

Comments/Thoughts

MattZimmerman: the correct way to implement this would be to fix packages to honour CFLAGS and CXXFLAGS, not try to patch them at runtime

Brentoboy: I thought that the "forced" CFLAGS in the source package was by design -- to make sure that the end result was the same thing as the one shipped as a binary. The intended reason to get the source instead is to fix a bug or something and then recompile knowing that you haven’t changed things other than your intended change. Or, at least that was my assumption when I discovered that it was not using my CFLAGS because there are CFLAGS in the source dir.

RAOF: It would probably be a good idea to use the pbuilder facilities, rather than raw dpkg-buildpackage. In particular, that means that all changes to the users buildchain could be confined to a chroot environment (just that little bit safer ;)), and the user's system wouldn't end up with practically every foo-dev package installed. This would change the proposed "emerge" script to (pseudocode):

cd /usr/src
for PACKAGE in $@ ;
do
    apt-get source $PACKAGE
    pbuilder build $PACKAGE.dsc
    dpkg --install *.deb
    rm *
done

I still think it would be a good idea to put the optimised packages in a local repository.

TODO

Here's a list of things we're going to have to create, and functionality we will have to include in order to implement this:

- The aptimize shell script that takes one or more packages as arguements and adds them in an aptimized state.

- Update the aptimize script to recognize if the first argument is --system-update and if it is, run a full system update instead of adding a list of packages.

- update the apt-build gcc wrapper so that it will work in our project, adding some status lines to the output explaining what we are going to do. Make sure it works with our make.conf file.

- build glade files for the ugentu GUI Requirements for GUI: * allow them to generate make.conf * allow them run a --system-update right now. * allow them to schedule cron times to do an unattended --system-update * launch the help file (probably the howto page in wiki.ubuntu.com/UgentuHowTo

- script the glade files with python and make them actually work

- ccache for faster recompilation?

Aptimize Feature Spec

Feel free to edit this list to your satisfaction. Try not to delete items, so that discussion about them can continue on the forums. Once consensus has been reached, deletions will be made. Additions are welcome!

1) a "package" called world (with obvious intentions)

2) oscar's symlink gcc call (again, easily done with a wrapper)

3) dependency handling

4) blacklist checking

5) special options to decide which dependencies are recompiled, and which aren't? similar to gentoo's e and D flags.

6) some way to gauge the progress of the script, and not just by the number of packages that have been compiled already... (this may require a patch in apt-build, or even make)

(This section added by YourDoom123).

Ubuntu Wiki