CheckboxConvertRegistryIntoDBusBackend

Differences between revisions 4 and 5
Revision 4 as of 2009-06-01 22:01:13
Size: 9678
Editor: modemcable178
Comment: Initial release of spec karmic-qa-checkbox-convert-registry-into-dbus-backend.
Revision 5 as of 2009-06-01 22:13:07
Size: 7277
Editor: modemcable178
Comment: Removed notes taken during the session copied while drafting this spec.
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:

Registry is a singleton: no reason to have more than one instance
Thus it can be exposed through DBUS and used by any applications that might want it
Also, it may be time-consuming, so we should only gather this info once and then make it available to all comers

Issue: some information should be publicly available while others should be private. (E.g. HAL info can be public, but if we gathered, say, a user's gconf info, that should not be public.)

Issue: currently the interface is very coarse; it should be enriched to expose strongly-typed variables in a tree structure (like HAL). Should support additional pluggable information.

Use case: apport (it would like to be able to use the system information that the registry gathers).
Feature that will be needed: expose system ID. (This is currently generated during checkbox run.)

Marc wants to be able to recurse through information gathered for apport... e.g. if a Rhythmbox bug is reported, recurse through the required packages for Rhythmbox and see that alsa-base is in the tree. Since
+alsa-base has apport hooks, we can gather all of its information as well.

Arguments for exposing this interface:
        apport
        Marc: "it just feels right"
        Individual scripts can use this data within Checkbox (rather than having to do e.g. lshal as they do now)
                Dave: this will require scripts to have a DBUS interface! :(
                        This should be mitigated by having a generalized interface to the registry
                                This would be useful in general as well... useful for determining where to find things in the registry e.g. when writing 'requires' fields for Checkbox scripts

Arguments against:
        depends on DBUS (duh) but -server doesn't have this yet
                or does it? It apparently has the system bus but not the session bus...
                        does this necessarily affect the registry?
        some people may not be comfortable exposing information to users that is normally accessible only to root
                need to run this by the security team
                might be able to add a policykit profile to this
                        Marc doesn't want to have to prompt the user when doing this
                        could we expose this to a group that contains only 'real' users (but not e.g. guest, apache, etc.)
        currently the queries to this are evaluated as Python expressions
                Martin Owen has a Perl script that illustrates some of the checks that need to happen when parsing this sort of data (that he will give to Marc)
                eeejay: flatten the tree and allow the user to pass a regular expression
                        run the regex on the .flatten method
                If this is all in SQLite, we could just let people run SQL queries
                Martin Owen: Xapien (sp?) or Lucene (sp?)

Questions:
        persistence of data? How long? We also have to listen for changes, as the system changes (hw plugged in, etc.)...
                this might need to happen in a later phase
                store under e.g. SQLite? (so that queries do not have to walk the whole registry tree)

Action items:
        Determine if -server will have DBUS by default in Karmic
                (Though could this be worked around through an additional abstraction layer?)

-----

(Another thing that would be cool to have would be a general DBUS browser GUI.)
Line 86: Line 35:
 * Test scripts could make use of a command line tool in order to query
   the registry instead of having to access the DBus backend directly.
Line 92: Line 43:
   * Apport would like to make use of the information gathered in a consistent
   way by the registry, such as the system id for example.
Line 121: Line 74:
Since the registry would run under the system DBus, it would need to
expose a query interface which does not consist of eval'ing a Python
expression. Instead, the registry should be stored in an SQL table
and expose an SQL query interface.
Line 137: Line 95:

As for the query interface, it would be preferable to use a lightweight
SQL database such as SQLite. In order to support recursive searching,
the database should be populated as if it were recursive. For example,
the registry key:

 * foo.bar.baz.blah

Should be populated as:

 * foo.bar.baz.blah
 * bar.baz.blah
 * baz.blah

This would make it possible to search for:

 * baz.blah = 'meh'
Line 188: Line 163:
Depending on how long the backend is expected to keep running, it might
become necessary to start listening for events such as new packages
installed, new hardware plugged, etc.

Summary

The registry in Checkbox should be expressed as a DBus backend.

Release Note

Applications needing consistent access to system information should use a central location. Ideally, this location should be exposed in a language agnostic way so that any application can access the same information.

Rationale

During the Jaunty cycle, Checkbox was extended so that parts of the registry needing root privileges to gather information could be run by the system DBus. This experience has shown that it would be worthwhile to expose the whole registry through DBus so that it could be loaded independently of the application.

Furthermore, the registry was originally designed with the singleton pattern in mind. This was successfully implemented at the application layer but, with the system DBus, it would be possible to implement this pattern at the system layer which would be even more appropriate.

User stories

  • Test scripts could use the same registry as Checkbox instead of having
    • to re-parse the output of various commands such as lshal, lshw, etc.
  • Test scripts could make use of a command line tool in order to query
    • the registry instead of having to access the DBus backend directly.
  • Checkbox could run more quickly by deferring processing of system
    • information to the DBus backend asynchronously.
  • Users could query the registry themselves, either through a DBus browser
    • or the command line to gather information about their system consistenly.
  • Contributers could write better test definitions by running queries
    • against the registry from the command line for the requires field.
  • Apport would like to make use of the information gathered in a consistent
    • way by the registry, such as the system id for example.

Assumptions

As of writing this specification, the ubuntu-server team doesn't seem to have any compelling reason to run DBus by default. Therefore, if we would like Checkbox installed by default at least during the development cycle, it would be preferable not to enforce DBus as a dependency. Instead, it should be possible to implement the backend as a library which could be imported directly when the application is run as root.

Also, the information gathered by the DBus backend should be sensitive to the user accessing it. For example, it should only expose gconf information for the user rather than for all users.

Design

The design of moving the registry from the core to a DBus backend will affect both the core and the plugins.

As far as the core is concerned, the concept of the registry should be exposed as a singleton accessed by a global handler. This handler will in turn abstract whether the registry is being accessed as a library when run as root or through DBus when run as a normal user.

As for the plugins, they currently access the registry as an attribute of the plugin manager. Instead, this attribute should be pulled out and the registry should then be accessed using the handler described above.

Since the registry would run under the system DBus, it would need to expose a query interface which does not consist of eval'ing a Python expression. Instead, the registry should be stored in an SQL table and expose an SQL query interface.

Implementation

Part of the implementation has already been completed during the Jaunty cycle by exposing a coarse grained interface to some registries through the DBus backend. This interface exposed a very simple interface returning the string representation of registries which deferred parsing to the client application. Instead, the interface should be refined so that the backend could become responsible for the parsing. The implication is that the interface would then have to become much finer grained and define types for all the information exposed.

Furthermore, the current implementation already supports the concept of accessing the registry directly by the application when run as root. So, this abstraction might actually be reusable or might need some refinement. This would require further exploration or might be revealed during implementation.

As for the query interface, it would be preferable to use a lightweight SQL database such as SQLite. In order to support recursive searching, the database should be populated as if it were recursive. For example, the registry key:

  • foo.bar.baz.blah

Should be populated as:

  • foo.bar.baz.blah
  • bar.baz.blah
  • baz.blah

This would make it possible to search for:

  • baz.blah = 'meh'

UI Changes

As far as the current command line and graphical interfaces are concerned, there should be no noticeable changes from moving the registry behind the DBus backend.

As far as the API is concerned, the plugins should become slightly simpler to implement. The reason is mostly that the contributers will need to know less about private attributes inherited from the Plugin class.

Last, it would be convenient to provide another command line application in order to query the registry. Instead of solely relying on the test definitions to query the registry, it would be quite useful to be able to perform queries like this:

  • checkbox_registry 'package.name = "apache2"'

This should return all the information known by the registry about the package named "apache2" assuming it is installed.

Code Changes

Interestingly, the existing regitries themselves should not need any changes at all. Instead, the code changes should strictly consist of wrapping the registries behind an abstraction layer which could either be called directly by the application or through the DBus backend. So, most of the implementation will consist of extending the current coarse grained interface to a much finer grained interface.

As far as the plugins are concerned, it would actually be possible to make the changes backward compatible with the current implementation. However, it would be preferable to change the plugins in order to access the registry singleton instead of using private attributes from the plugin manager.

Migration

Optionally, it would be desirable for the plugins to be migrated in order to use the registry singleton. As far as the current registries are concerned, as mentionned above, they should not need any changes.

Test/Demo Plan

  1. Run the command: checkbox_registry 'package.name = "apache2"'
  2. This should return a package entry from the registry

Unresolved issues

It might be worthwhile to explore existing DBus browsers which could expose the information provided by the registry in a convenient way.

Depending on how long the backend is expected to keep running, it might become necessary to start listening for events such as new packages installed, new hardware plugged, etc.


CategorySpec

QATeam/Specs/CheckboxConvertRegistryIntoDBusBackend (last edited 2009-06-01 22:15:35 by modemcable178)