paul@katu:~/ubuntu/fix/bzip2-compression/original/tmp$ ls -l data*bz2 ../language-pack-gnome-en-base_20050826_all.deb | sort -k 5 -rw-r--r-- 1 paul paul 1216424 2005-09-16 14:10 data-repacked-by-alpha-r.tar.bz2 -rw-r--r-- 1 paul paul 1216878 2005-09-16 13:50 data-repacked-by-percentage.tar.bz2 -rw-r--r-- 1 paul paul 1228604 2005-09-16 14:09 data-repacked-by-alpha.tar.bz2 -rw-r--r-- 1 paul paul 1237817 2005-09-16 14:06 data-repacked-by-percentage-r.tar.bz2 -rw-r--r-- 1 paul paul 1502554 2005-09-16 13:39 data.tar.bz2 -rw-r--r-- 1 paul paul 1508720 2005-08-26 17:40 ../language-pack-gnome-en-base_20050826_all.deb
mkdir -p original cp -a /var/cache/apt/archives/language-pack-* original/ mv original/ bzip2-compression/ cd bzip2-compression/original/ mkdir -p tmp cd tmp ar x ../language-pack-gnome-en-base_20050826_all.deb data.tar.bz2 mkdir -p data ; cd data tar jxvf ../data.tar.bz2 find usr/ -type f -print0 | xargs -0 bzip2 -cv > /dev/null 2>repack-results sort -n -k 3 repack-results | awk '{print $1}' | sed -e 's/:$//' | xargs tar jcvf data-repacked-by-percentage.tar.bz2
find usr/ -type f | sort -t '/' -k 5 > last-part-alpha cat last-part-alpha | xargs tar jcvf ../data-repacked-by-alpha.tar.bz2 find usr/ -type f | sort -r -t '/' -k 5 > last-part-alpha-r cat last-part-alpha-r | xargs tar jcvf ../data-repacked-by-alpha-r.tar.bz2
Top 20 packages on the install CD come to 200MB:
$ find pool/ -type f -ls | awk '{print $7, $11}' | sort -n > cd-contents $ tail -20 cd-contents | awk '{x = x + $1; print } END {print x, "Total"}' 4807608 pool/main/r/rss-glx/rss-glx_0.7.5-4ubuntu1_i386.deb 4886658 pool/main/g/glibc/libc6_2.3.5-1ubuntu11_i386.deb 5094342 pool/main/g/gcc-4.0/libgcj6_4.0.1-4ubuntu6_i386.deb 5167552 pool/restricted/l/linux-restricted-modules-2.6.12/linux-restricted-modules-2.6.12-8-386_2.6.12.2-12_i386.deb 5381934 pool/main/t/ttf-kochi/ttf-kochi-mincho_1.0.20030809-3_all.deb 5602128 pool/main/t/ttf-arphic-bsmi00lp/ttf-arphic-bsmi00lp_2.10-6_all.deb 5646236 pool/main/x/xfonts-core/xfonts-base_6.8.2.1-3_all.deb 5918140 pool/main/l/linux-source-2.6.12/linux-headers-2.6.12-8_2.6.12-8.12_i386.deb 5984092 pool/main/t/ttf-arphic-bkai00mp/ttf-arphic-bkai00mp_2.10-6_all.deb 6522278 pool/main/h/hplip/hplip-data_0.9.4-3ubuntu1_all.deb 7280072 pool/main/f/foomatic-filters-ppds/foomatic-filters-ppds_20050720-1ubuntu1_all.deb 8440374 pool/main/f/firefox/firefox_1.0.6-1ubuntu10_i386.deb 8962800 pool/main/c/cupsys/cupsys_1.1.23-10ubuntu3_i386.deb 10334574 pool/main/m/mozilla-thunderbird/mozilla-thunderbird_1.0.6-0ubuntu6_i386.deb 10981938 pool/main/e/emacs21/emacs21-common_21.4a-1ubuntu1_all.deb 11504944 pool/main/t/ttf-baekmuk/ttf-baekmuk_2.2-1ubuntu1_all.deb 12661874 pool/main/m/mesa/libgl1-mesa-dri_6.3.2-0ubuntu5_i386.deb 17786728 pool/main/l/linux-source-2.6.12/linux-image-2.6.12-8-386_2.6.12-8.12_i386.deb 25643634 pool/main/o/openoffice.org2/openoffice.org2-common_1.9.125+2.0beta2-1ubuntu1_all.deb 31338372 pool/main/o/openoffice.org2/openoffice.org2-core_1.9.125+2.0beta2-1ubuntu1_i386.deb 199946278 Total
OOo2, Linux (+binary modules, +headers), Mesa, Emacs, 4x intriguing Truetype fonts, Core X fonts, Thunderbird, Firefox, Cups (+foomatic PPDs), GCJ/Java and Lots of OpenGL screensavers (rss-glx).
At a guess, Foomatic and linux-headers would reduce fairly well. Although it might be simpler to move everything not in the default install seed to bzip2 compression (since install speed is less important).
Foomatic
Foomatic contains PPDs (PostScript Printer Definitons) for 2599 individual printers. Each of these is incredibly similar but each file is individually gzipped.
$ find -name '*.gz' | wc -l 2599 $ find -name '*.gz' | xargs du -c -h | tail -1 1.2M total
Yet these files take up 13MB on disk because each one is occupying a 4k inode. After unpacking each PPD, then tar'ing the whole lot up again with gzip or bzip2 it uses:
-rw-r--r-- 1 paul paul 929986 2005-09-16 17:16 ../data-ppd-no-gz.tar.bz2 -rw-r--r-- 1 paul paul 2976299 2005-09-16 17:15 ../data-ppd-no-gz.tar.gz -rw-r--r-- 1 paul paul 7210215 2005-09-16 17:03 ../data.tar.gz
I think these PPDs should really be stored on-disk as a tarball or .zip; the inode/block count wastage is huge. ZIP has an advantage that it is directly addressible, but lacks cross-file compression savings. But uncompressing the PPDs, the raw on-disk usage goes up from 13MB to 120MB.
$ time find -type f -name \*.ppd | xargs gzip real 0m2.912s user 0m2.543s $ time find -type f -name \*.gz | xargs gunzip real 0m1.680s user 0m1.192s
Even the Bzip2 archive (because of the huge compression) is amazingly fast to decompress:
$ time tar tjf ../data-ppd-no-gz.tar.bz2 | grep HP | head -1 usr/share/ppd/HP/ real 0m1.446s user 0m0.042s
Solution: Store as a .tar.bz2 and teach CUPS how to go and fetch PPDs. Preferably abstract the fetching behind a script so that they can be transparently downloaded off the web or from the Printer itself. Possibility to remain 100% compatible by using a post-install to unpack the .tar.bz2 and recompress as individual files.
Saving: 7.2MB -> 0.9MB on CD and 13MB -> 0.9MB on disk. 87% saving on CD.
Linux Headers
$ find usr/ -type f -name '*.h' | xargs du -sch | tail -1 9.9M Header Files $ find usr/ -type f \! -name '*.h' | xargs du -sch | tail -1 6.4M Not Header Files $ find usr/ -type f -name 'Makefile' | xargs du -sch | tail -1 3.3M Of Which, Makefiles $ find usr/ -type f -name 'Kconfig*' | xargs du -sch | tail -1 2.5M Of Which Kconfigs $ find usr/ -type f \! -name 'Kconfig*' \! -name Makefile \! -name '*.h' | xargs du -sch | tail -1 684K Of Which Other/Uknown
9.9MB of headers. 6.4MB of Makefile/Kconfig. 176kB of the Unknowns are ARM assembly .S assembler files for every particular variation of embedded board/device. 160kB are files such as MAINTAINERS.gz and README.gz that have already been compressed; these unpack to 440kB on disk and compress back to 118kB using bzip2.
3.3MB of it is include/asm-arm/ so this probably warrants special attention.
So far, the best I have is:
Unpacking all *.gz files (280kB on-disk increase)---could be reclaimed by post-inst.
find usr/ -type f -name \*.gz | xargs gunzip find usr/ -type f | LANG=C sed -e 's-^.*/\([^/][^/]*\)-\1 &-' | sort | awk '{print $2}' | tar jcf data-by-filename.tar.bz2 -T -
-rw-r--r-- 1 paul paul 4555509 2005-09-16 19:21 data-by-filename.tar.bz2 -rw-r--r-- 1 paul paul 4823926 2005-09-16 17:45 data.tar.bzip2 -rw-r--r-- 1 paul paul 5916271 2005-09-16 17:43 data.tar.gz
Saving: 1.3MB on CD (18% with bzip2, 23% with bzip2 + fiddling), 280kB extra on disk.