Ubuntu Wiki

   1 [18:01] <kirkland> howdy all
   2 [18:02] <kirkland> this session is on dotdee
   3 [18:02] <kirkland> there will be a live streamed demo at: http://bit.ly/uclass
   4 [18:02] <kirkland> i invite you to join me there
   5 [18:02] <kirkland> the username and password is guest/guest
   6 [18:03] <kirkland> for those interested, this is a tool called ajaxterm
   7 [18:03] <kirkland> which embeds a terminal in a web browser
   8 [18:03] <kirkland> i've set up a byobu/screen session for the guest user (which is readonly for all of you)
   9 [18:03] <kirkland> I'll drive the demo
  10 [18:04] <kirkland> and annotate it here
  11 [18:04] <kirkland> alternatively, you can ssh guest@ec2-50-19-128-105.compute-1.amazonaws.com
  12 [18:04] <kirkland> with password guest
  13 [18:05] <kirkland> okay, on to dotdee :-)
  14 [18:05] <kirkland> if you've ever configured a Linux/UNIX system, you're probably familiar with the /etc directory
  15 [18:05] <kirkland> and inside of /etc, there are many directories that end in a ".d"
  16 [18:05] <kirkland> watch the terminal while I find a few in /etc
  17 [18:06] <kirkland> there's a bunch!
  18 [18:06] <kirkland> this is a very user friendly way of offering configuration to users
  19 [18:06] <kirkland> usually, files in a .d directory are concatenated, or executed sequentially
  20 [18:07] <kirkland> it give users quite a bit of flexibility for editing or adding configurations
  21 [18:07] <kirkland> to some software or service
  22 [18:07] <kirkland> it also helps developers and packagers of software
  23 [18:07] <kirkland> as it often allows them to drop snippets of configuration into place
  24 [18:07] <kirkland> but not every configuration file is setup in this way
  25 [18:08] <kirkland> in fact, most of them are really quite "flat"
  26 [18:08] <kirkland> a few months ago, I found myself repeatedly needing to converted some flat configuration files
  27 [18:08] <kirkland> to .d style ones
  28 [18:09] <kirkland> this was for software I was working with, as a developer/packager
  29 [18:09] <kirkland> but not software that I had written myself
  30 [18:09] <kirkland> ideally, I would just ask the upstream developers to change their flat .conf file
  31 [18:09] <kirkland> to a .d directory
  32 [18:09] <kirkland> and they would magically do it
  33 [18:09] <kirkland> and test it
  34 [18:09] <kirkland> and release it
  35 [18:09] <kirkland> immediately :-)
  36 [18:10] <kirkland> that rarely happens though :-P
  37 [18:10] <kirkland> so I wrote a little tool that would generically to that for me!
  38 [18:10] <kirkland> and that tool is called "dotdee"
  39 [18:10] <kirkland> so let's take a look!
  40 [18:10] <kirkland> over in the terminal, i'm going to install dotdee, which is already in Ubuntu oneiric
  41 [18:10] <kirkland> sudo apt-get install dotdee
  42 [18:11] <kirkland> for older ubuntu distros, you can 'sudo apt-add-repository ppa:dotdee/ppa'
  43 [18:11] <kirkland> and update, and install it there too
  44 [18:11] <kirkland> cool, so now i have a /usr/sbin/dotdee executable
  45 [18:11] <kirkland> let's take a flat file, and turn it into a dotdee directory!
  46 [18:12] <kirkland> I'm going to use /etc/hosts as my example
  47 [18:12] <kirkland> first, I need to "setup" the file
  48 [18:12] <kirkland> i can first verify that /etc/hosts is in fact a flat file,
  49 [18:12] <kirkland> -rw-r--r-- 1 root root 296 2011-07-13 12:14 /etc/hosts
  50 [18:13] <kirkland> I'm going to set it up like this:
  51 [18:13] <kirkland> sudo dotdee --setup /etc/hosts
  52 [18:13] <kirkland> INFO: [/etc/hosts] updated by dotdee
  53 [18:13] <kirkland> cool!
  54 [18:13] <kirkland> now let's look at what dotdee did....
  55 [18:13] <kirkland> $ ll /etc/hosts
  56 [18:13] <kirkland> lrwxrwxrwx 1 root root 27 2011-07-13 18:13 /etc/hosts -> /etc/alternatives/etc:hosts
  57 [18:13] <kirkland> so /etc/hosts is now a symlink
  58 [18:13] <kirkland> pointing to an alternatives link
  59 [18:14] <kirkland> $ ll /etc/alternatives/etc:hosts
  60 [18:14] <kirkland> lrwxrwxrwx 1 root root 22 2011-07-13 18:13 /etc/alternatives/etc:hosts -> /etc/dotdee//etc/hosts
  61 [18:14] <kirkland> which is pointing to a flat file in /etc/dotdee
  62 [18:14] <kirkland> $ ll /etc/dotdee//etc/hosts
  63 [18:14] <kirkland> -r--r--r-- 1 root root 296 2011-07-13 18:13 /etc/dotdee//etc/hosts
  64 [18:14] <kirkland> if I look at the contents of that file, I see exactly what I had before
  65 [18:15] <kirkland> but let's go to that directory, /etc/dotdee
  66 [18:15] <kirkland> inside of /etc/dotdee, there is a file structure that mirrors the same file structure on the system
  67 [18:15] <kirkland> importantly, we now have a .d directory
  68 [18:15] <kirkland> that refers exclusively to our newly managed file
  69 [18:16] <kirkland> namely, /etc/dotdee/etc/hosts.d
  70 [18:16] <kirkland> in that directory, all we have is a file called 50-original
  71 [18:16] <kirkland> which is the original contents of our /etc/hosts
  72 [18:16] <kirkland> but let's say we want to "append" a host to that fie
  73 [18:16] <kirkland> file
  74 [18:16] <kirkland> let's create a new file in this directory
  75 [18:16] <kirkland> and call it 60-googledns
  76 [18:17] <kirkland> so I edit the new file (as root)
  77 [18:17] <kirkland> add the entry, 8.8.8.8 googledns
  78 [18:17] <kirkland> and I'm going to write the file
  79 [18:17] <kirkland> but before i write the file, let me split the screen
  80 [18:18] <kirkland> so that we can watch it get updated, automatically, in real time!
  81 [18:18] <kirkland> so i ran 'watch -n1 cat /etc/hosts'
  82 [18:18] <kirkland> which is just printing that file every 1 second
  83 [18:18] <kirkland> now i'm going to save our 60-googledns file
  84 [18:18] <kirkland> and voila!
  85 [18:19] <kirkland> we have a new entry appended to our /etc/hosts
  86 [18:19] <kirkland> through the magic of inotify :-)
  87 [18:19] <kirkland> which is a daemon that monitors filesystem events
  88 [18:19] <kirkland> dotdee comes with a configuration (which is dotdee managed, of course!) that adds and removes patterns
  89 [18:19] <kirkland> as you setup/remove dotdee management
  90 [18:20] <kirkland> let's prepend a host
  91 [18:20] <kirkland> we'll call this one 40-foo
  92 [18:20] <kirkland> and see it land at the beginning of the file
  93 [18:20] <kirkland> bingo
  94 [18:20] <kirkland> now it's at the beginning of our /etc/hosts
  95 [18:22] <kirkland> okay
  96 [18:22] <kirkland> so adding/removing a flat text file is one way of affecting our managed file
  97 [18:23] <kirkland> flat text files are just appended or prepended, based on its alpha-numeric positioning
  98 [18:23] <kirkland> but there are 2 other ways as well!
  99 [18:23] <kirkland> you can also put executables in this .d directory
 100 [18:23] <kirkland> which operate on the standard in and out
 101 [18:23] <kirkland> if you want to modify a flat file by "processing" it
 102 [18:24] <kirkland> for instance
 103 [18:24] <kirkland> let's make this file all uppercase
 104 [18:25] <kirkland> whoops
 105 [18:25] <kirkland> okay, there we go
 106 [18:25] <kirkland> which brings me to the --update command :-)
 107 [18:26] <kirkland> dotdee --update can be called against any managed file
 108 [18:26] <kirkland> to update it immediately
 109 [18:26] <kirkland> in case the inotify bit didn't pick up the change
 110 [18:26] <kirkland> in any case, i just did that, and now our /etc/hosts is all uppercase
 111 [18:26] <kirkland> because of our 70-uppercase executable
 112 [18:27] <kirkland> what happens if we move it from 70-uppercase to 10-uppercase?
 113 [18:27] <kirkland> or, rather, how about 51-uppercase?
 114 [18:27] <kirkland> see the output now
 115 [18:28] <kirkland> note that 51-uppercase was applied against the "current state" of the output, as of position 51
 116 [18:28] <kirkland> but 60- was applied afterward
 117 [18:28] <kirkland> so it wasn't affected
 118 [18:28] <kirkland> so that's two ways we can affect the contents of the file
 119 [18:28] <kirkland> a) flat text files, b) scripts that process stdin and write to stdout
 120 [18:29] <kirkland> the third way is patches or diff files
 121 [18:29] <kirkland> given that this is a developer audience, we're probably familiar with quilt
 122 [18:29] <kirkland> and directories of patches
 123 [18:29] <kirkland> this is particularly useful if you need to do some 'surgery' on a file
 124 [18:29] <kirkland> let's say I want to "insert" a line into the middle of this file
 125 [18:30] <kirkland> into the middle of 50-original, for instance
 126 [18:31] <kirkland> okay, so i've added "10.9.8.7        hello-there" to the middle of a copy of this file
 127 [18:31] <kirkland> and i'm going to use diff -up to generate a patch
 128 [18:31] <kirkland> there we go
 129 [18:31] <kirkland> okay, let's put that in this .d dir
 130 [18:32] <kirkland> note that I have to add .patch or .diff as the file extension
 131 [18:32] <kirkland> and now i can cat /etc/hosts and see that the patch has been applied!
 132 [18:33] <kirkland> i could stack a great number of these patches here
 133 [18:33] <kirkland> much like a quilt directory
 134 [18:33] <kirkland> okay, so now let's undo this configuration
 135 [18:34] <kirkland> oh, first
 136 [18:34] <kirkland> sudo dotdee --list /etc/hosts
 137 [18:34] <kirkland> /etc/hosts
 138 [18:34] <kirkland> $ echo $?
 139 [18:34] <kirkland> 0
 140 [18:34] <kirkland> this verifies that /etc/hosts is in fact dotdee managed
 141 [18:34] <kirkland> if i try this against some other file
 142 [18:35] <kirkland> $ sudo dotdee --list /boot/vmlinuz-3.0-3-virtual
 143 [18:35] <kirkland> ERROR: [/boot/vmlinuz-3.0-3-virtual] is not managed by dotdee
 144 [18:35] <kirkland> (I don't recommend dotdee'ing your kernel :-)
 145 [18:35] <kirkland> but we can undo our /etc/hosts
 146 [18:35] <kirkland> $ sudo dotdee --undo /etc/hosts
 147 [18:35] <kirkland> update-alternatives: using /etc/dotdee//etc/hosts.d/50-original to provide /etc/hosts (etc:hosts) in auto mode.
 148 [18:35] <kirkland> INFO: [/etc/hosts] has been restored
 149 [18:35] <kirkland> INFO: You may want to manually remove [/etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d]
 150 [18:36] <kirkland> and now our /etc/hosts is back to being whatever we saved in 50-original
 151 [18:36] <kirkland> so ...
 152 [18:36] <kirkland> that's how it works
 153 [18:36] <kirkland> and the /etc/hosts example is only marginally useful
 154 [18:37] <kirkland> what I would *really* like to use it for is in configuration file management in Debian/Ubuntu packaging
 155 [18:37] <kirkland> in the case where the upstream daemon or utility has a single, flat .conf file
 156 [18:37] <kirkland> but I really would prefer it to be a .d directory
 157 [18:37] <kirkland> so i just did a find on /etc
 158 [18:38] <kirkland> sudo find /etc/ -type f -name "*.conf"
 159 [18:38] <kirkland> and chose one at random
 160 [18:38] <kirkland>  /etc/fonts/fonts.conf
 161 [18:38] <kirkland> which happens to be XML
 162 [18:38] <kirkland> and I just thought about this
 163 [18:38] <kirkland> i should have mentioned it in our previous section
 164 [18:39] <kirkland> XML is tougher than a linear file, like a shell script
 165 [18:39] <kirkland> in that you can't just append, or prepend text
 166 [18:39] <kirkland> you have to surgically insert the bits you want
 167 [18:39] <kirkland> in which case the latter two methods I mentioned, the executable and the diff/patch will be your friend!
 168 [18:40] <kirkland> okay
 169 [18:40] <kirkland> now I would *really* like to see dpkg learn just a little bit about dotdee
 170 [18:41] <kirkland> I'd like for it to be able to determine *if* a file is managed by dotdee
 171 [18:41] <kirkland> (easy to check using dotdee --list, or just checking if the file is a symlink itself)
 172 [18:41] <kirkland> and if so, then it would use $(dotdee --original ...) to find the 50-original file path
 173 [18:42] <kirkland> and dpkg would write its changes to that location (the 50-original file)
 174 [18:42] <kirkland> such that local admin, or even other packages, could dabble in the .d directory, without causing conffile conflicts or .dpkg-original files
 175 [18:43] <kirkland> okay, anyway, let's take a break for questions
 176 [18:43] <kirkland> I think I've demo'd most of what I'd like to show you
 177 [18:43] <kirkland> any questions?
 178 [18:44] <kirkland> <coalitians> Question: Is there any issues when you upgrade or update due to dotdee?
 179 [18:44] <kirkland> coalitians: great question !
 180 [18:44] <kirkland> coalitians: right, so that's what I was saying about dpkg needing to "learn" about dotdee
 181 [18:44] <kirkland> coalitians: let's look at an example over in our test system
 182 [18:45] <kirkland> coalitians: I'm going to dotdee --setup that font xml
 183 [18:47] <kirkland> coalitians: okay, as you can see, i've made a change
 184 [18:47] <kirkland> coalitians: let's upgrade (or reinstall) the package that owns this file
 185 [18:47] <kirkland> $ sudo apt-get install --reinstall fontconfig-config
 186 [18:47] <kirkland> lrwxrwxrwx  1 root root   38 2011-07-13 18:45 fonts.conf -> /etc/alternatives/etc:fonts:fonts.conf
 187 [18:47] <kirkland> -rw-r--r--  1 root root 5287 2011-07-01 12:12 fonts.conf.dpkg-new
 188 [18:48] <kirkland> unfortunately, dpkg dump fonts.conf.dpkg-new here :-(
 189 [18:48] <kirkland> i have toyed with another inotify/iwatch regex that would look for these :-)
 190 [18:48] <kirkland> slurp them up, and move them over to 50-original
 191 [18:48] <kirkland> which works reasonably well
 192 [18:49] <kirkland> except that I don't yet have the interface for the merge/conffile questions, like dpkg does
 193 [18:49] <kirkland> coalitians: so to answer your question, upgrades are not yet handled terribly gracefully
 194 [18:49] <kirkland> coalitians: and would take a little work within dpkg itself
 195 [18:49] <kirkland> coalitians: sorry;  but great question!
 196 [18:49] <kirkland> any others?
 197 [18:50] <kirkland> I see roughly 19# in the byobu session :-)
 198 [18:50] <kirkland> that's what the red-on-white 19# means
 199 [18:50] <kirkland> 19 ssh sessions
 200 [18:50] <kirkland> did the web interface work for anyone?
 201 [18:50] <kirkland> this is the first time I've tried it