CheckboxConvertRegistryIntoDBusBackend

Differences between revisions 3 and 4
Revision 3 as of 2009-05-25 15:00:54
Size: 2625
Editor: 80
Comment: Fixed URL to blueprint
Revision 4 as of 2009-06-01 22:01:13
Size: 9678
Editor: modemcable178
Comment: Initial release of spec karmic-qa-checkbox-convert-registry-into-dbus-backend.
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
Registry is a singleton: no reason to have more than one instance
Thus it can be exposed through DBUS and used by any applications that might want it
Also, it may be time-consuming, so we should only gather this info once and then make it available to all comers

Issue: some information should be publicly available while others should be private. (E.g. HAL info can be public, but if we gathered, say, a user's gconf info, that should not be public.)

Issue: currently the interface is very coarse; it should be enriched to expose strongly-typed variables in a tree structure (like HAL). Should support additional pluggable information.

Use case: apport (it would like to be able to use the system information that the registry gathers).
Feature that will be needed: expose system ID. (This is currently generated during checkbox run.)

Marc wants to be able to recurse through information gathered for apport... e.g. if a Rhythmbox bug is reported, recurse through the required packages for Rhythmbox and see that alsa-base is in the tree. Since
+alsa-base has apport hooks, we can gather all of its information as well.

Arguments for exposing this interface:
        apport
        Marc: "it just feels right"
        Individual scripts can use this data within Checkbox (rather than having to do e.g. lshal as they do now)
                Dave: this will require scripts to have a DBUS interface! :(
                        This should be mitigated by having a generalized interface to the registry
                                This would be useful in general as well... useful for determining where to find things in the registry e.g. when writing 'requires' fields for Checkbox scripts

Arguments against:
        depends on DBUS (duh) but -server doesn't have this yet
                or does it? It apparently has the system bus but not the session bus...
                        does this necessarily affect the registry?
        some people may not be comfortable exposing information to users that is normally accessible only to root
                need to run this by the security team
                might be able to add a policykit profile to this
                        Marc doesn't want to have to prompt the user when doing this
                        could we expose this to a group that contains only 'real' users (but not e.g. guest, apache, etc.)
        currently the queries to this are evaluated as Python expressions
                Martin Owen has a Perl script that illustrates some of the checks that need to happen when parsing this sort of data (that he will give to Marc)
                eeejay: flatten the tree and allow the user to pass a regular expression
                        run the regex on the .flatten method
                If this is all in SQLite, we could just let people run SQL queries
                Martin Owen: Xapien (sp?) or Lucene (sp?)

Questions:
        persistence of data? How long? We also have to listen for changes, as the system changes (hw plugged in, etc.)...
                this might need to happen in a later phase
                store under e.g. SQLite? (so that queries do not have to walk the whole registry tree)

Action items:
        Determine if -server will have DBUS by default in Karmic
                (Though could this be worked around through an additional abstraction layer?)

-----

(Another thing that would be cool to have would be a general DBUS browser GUI.)
Line 10: Line 61:
Technical details of extracting the registry into it's own DBus backend, along with identify other projects who may be interested in using this. The registry in Checkbox should be expressed as a DBus backend.
Line 14: Line 65:
This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.
Applications needing consistent access to system information should use a
central location. Ideally, this location should be exposed in a language
agnostic way so that any application can access the same information.
Line 20: Line 71:
This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified. During the Jaunty cycle, Checkbox was extended so that parts of the
registry needing root privileges to gather information could be run by
the system DBus. This experience has shown that it would be worthwhile
to expose the whole registry through DBus so that it could be loaded
independently of the application.

Furthermore, the registry was originally designed with the singleton
pattern in mind. This was successfully implemented at the application
layer but, with the system DBus, it would be possible to implement this
pattern at the system layer which would be even more appropriate.
Line 24: Line 84:
 * Test scripts could use the same registry as Checkbox instead of having
   to re-parse the output of various commands such as lshal, lshw, etc.
 * Checkbox could run more quickly by deferring processing of system
   information to the DBus backend asynchronously.
 * Users could query the registry themselves, either through a DBus browser
   or the command line to gather information about their system consistenly.
 * Contributers could write better test definitions by running queries
   against the registry from the command line for the requires field.
 
Line 25: Line 94:

As of writing this specification, the ubuntu-server team doesn't seem to
have any compelling reason to run DBus by default. Therefore, if we would
like Checkbox installed by default at least during the development cycle,
it would be preferable not to enforce DBus as a dependency. Instead,
it should be possible to implement the backend as a library which could
be imported directly when the application is run as root.

Also, the information gathered by the DBus backend should be sensitive
to the user accessing it. For example, it should only expose gconf
information for the user rather than for all users.
Line 28: Line 108:
You can have subsections that better describe specific parts of the issue. The design of moving the registry from the core to a DBus backend will
affect both the core and the plugins.

As far as the core is concerned, the concept of the registry should be
exposed as a singleton accessed by a global handler. This handler will
in turn abstract whether the registry is being accessed as a library
when run as root or through DBus when run as a normal user.

As for the plugins, they currently access the registry as an attribute
of the plugin manager. Instead, this attribute should be pulled out
and the registry should then be accessed using the handler described
above.
Line 32: Line 123:
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like: Part of the implementation has already been completed during the Jaunty
cycle by exposing a coarse grained interface to some registries through
the DBus backend. This interface exposed a very simple interface returning
the string representation of registries which deferred parsing to the
client application. Instead, the interface should be refined so that the
backend could become responsible for the parsing. The implication is that
the interface would then have to become much finer grained and define
types for all the information exposed.

Furthermore, the current implementation already supports the concept
of accessing the registry directly by the application when run as
root. So, this abstraction might actually be reusable or might need some
refinement. This would require further exploration or might be revealed
during implementation.
Line 36: Line 140:
Should cover changes required to the UI, or specific UI that is required to implement this As far as the current command line and graphical interfaces are concerned,
there should be no noticeable changes from moving the registry behind the
DBus backend.

As far as the API is concerned, the plugins should become slightly simpler
to implement. The reason is mostly that the contributers will need to know
less about private attributes inherited from the Plugin class.

Last, it would be convenient to provide another command line application
in order to query the registry. Instead of solely relying on the test
definitions to query the registry, it would be quite useful to be able
to perform queries like this:

 * checkbox_registry 'package.name = "apache2"'

This should return all the information known by the registry about the
package named "apache2" assuming it is installed.
Line 40: Line 160:
Code changes should include an overview of what needs to change, and in some cases even the specific details. Interestingly, the existing regitries themselves should not need any changes
at all. Instead, the code changes should strictly consist of wrapping the
registries behind an abstraction layer which could either be called directly
by the application or through the DBus backend. So, most of the implementation
will consist of extending the current coarse grained interface to a much finer
grained interface.

As far as the plugins are concerned, it would actually be possible to make the
changes backward compatible with the current implementation. However, it would
be preferable to change the plugins in order to access the registry singleton
instead of using private attributes from the plugin manager.
Line 44: Line 174:
Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.
Optionally, it would be desirable for the plugins to be migrated in
order to use the registry singleton. As far as the current registries
are concerned, as mentionned above, they should not need any changes.
Line 51: Line 180:
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.
 1. Run the command: checkbox_registry 'package.name = "apache2"'
 2. This should return a package entry from the registry
Line 57: Line 185:
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

== BoF agenda and discussion ==

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
It might be worthwhile to explore existing DBus browsers which could
expose the information provided by the registry in a convenient way.

Registry is a singleton: no reason to have more than one instance Thus it can be exposed through DBUS and used by any applications that might want it Also, it may be time-consuming, so we should only gather this info once and then make it available to all comers

Issue: some information should be publicly available while others should be private. (E.g. HAL info can be public, but if we gathered, say, a user's gconf info, that should not be public.)

Issue: currently the interface is very coarse; it should be enriched to expose strongly-typed variables in a tree structure (like HAL). Should support additional pluggable information.

Use case: apport (it would like to be able to use the system information that the registry gathers). Feature that will be needed: expose system ID. (This is currently generated during checkbox run.)

Marc wants to be able to recurse through information gathered for apport... e.g. if a Rhythmbox bug is reported, recurse through the required packages for Rhythmbox and see that alsa-base is in the tree. Since +alsa-base has apport hooks, we can gather all of its information as well.

Arguments for exposing this interface:

  • apport Marc: "it just feels right" Individual scripts can use this data within Checkbox (rather than having to do e.g. lshal as they do now)
    • Dave: this will require scripts to have a DBUS interface! Sad :(

      • This should be mitigated by having a generalized interface to the registry
        • This would be useful in general as well... useful for determining where to find things in the registry e.g. when writing 'requires' fields for Checkbox scripts

Arguments against:

  • depends on DBUS (duh) but -server doesn't have this yet
    • or does it? It apparently has the system bus but not the session bus...
      • does this necessarily affect the registry?
    some people may not be comfortable exposing information to users that is normally accessible only to root
    • need to run this by the security team might be able to add a policykit profile to this
      • Marc doesn't want to have to prompt the user when doing this could we expose this to a group that contains only 'real' users (but not e.g. guest, apache, etc.)
    currently the queries to this are evaluated as Python expressions
    • Martin Owen has a Perl script that illustrates some of the checks that need to happen when parsing this sort of data (that he will give to Marc) eeejay: flatten the tree and allow the user to pass a regular expression
      • run the regex on the .flatten method
      If this is all in SQLite, we could just let people run SQL queries Martin Owen: Xapien (sp?) or Lucene (sp?)

Questions:

  • persistence of data? How long? We also have to listen for changes, as the system changes (hw plugged in, etc.)...
    • this might need to happen in a later phase store under e.g. SQLite? (so that queries do not have to walk the whole registry tree)

Action items:

  • Determine if -server will have DBUS by default in Karmic
    • (Though could this be worked around through an additional abstraction layer?)


(Another thing that would be cool to have would be a general DBUS browser GUI.)

Summary

The registry in Checkbox should be expressed as a DBus backend.

Release Note

Applications needing consistent access to system information should use a central location. Ideally, this location should be exposed in a language agnostic way so that any application can access the same information.

Rationale

During the Jaunty cycle, Checkbox was extended so that parts of the registry needing root privileges to gather information could be run by the system DBus. This experience has shown that it would be worthwhile to expose the whole registry through DBus so that it could be loaded independently of the application.

Furthermore, the registry was originally designed with the singleton pattern in mind. This was successfully implemented at the application layer but, with the system DBus, it would be possible to implement this pattern at the system layer which would be even more appropriate.

User stories

  • Test scripts could use the same registry as Checkbox instead of having
    • to re-parse the output of various commands such as lshal, lshw, etc.
  • Checkbox could run more quickly by deferring processing of system
    • information to the DBus backend asynchronously.
  • Users could query the registry themselves, either through a DBus browser
    • or the command line to gather information about their system consistenly.
  • Contributers could write better test definitions by running queries
    • against the registry from the command line for the requires field.

Assumptions

As of writing this specification, the ubuntu-server team doesn't seem to have any compelling reason to run DBus by default. Therefore, if we would like Checkbox installed by default at least during the development cycle, it would be preferable not to enforce DBus as a dependency. Instead, it should be possible to implement the backend as a library which could be imported directly when the application is run as root.

Also, the information gathered by the DBus backend should be sensitive to the user accessing it. For example, it should only expose gconf information for the user rather than for all users.

Design

The design of moving the registry from the core to a DBus backend will affect both the core and the plugins.

As far as the core is concerned, the concept of the registry should be exposed as a singleton accessed by a global handler. This handler will in turn abstract whether the registry is being accessed as a library when run as root or through DBus when run as a normal user.

As for the plugins, they currently access the registry as an attribute of the plugin manager. Instead, this attribute should be pulled out and the registry should then be accessed using the handler described above.

Implementation

Part of the implementation has already been completed during the Jaunty cycle by exposing a coarse grained interface to some registries through the DBus backend. This interface exposed a very simple interface returning the string representation of registries which deferred parsing to the client application. Instead, the interface should be refined so that the backend could become responsible for the parsing. The implication is that the interface would then have to become much finer grained and define types for all the information exposed.

Furthermore, the current implementation already supports the concept of accessing the registry directly by the application when run as root. So, this abstraction might actually be reusable or might need some refinement. This would require further exploration or might be revealed during implementation.

UI Changes

As far as the current command line and graphical interfaces are concerned, there should be no noticeable changes from moving the registry behind the DBus backend.

As far as the API is concerned, the plugins should become slightly simpler to implement. The reason is mostly that the contributers will need to know less about private attributes inherited from the Plugin class.

Last, it would be convenient to provide another command line application in order to query the registry. Instead of solely relying on the test definitions to query the registry, it would be quite useful to be able to perform queries like this:

  • checkbox_registry 'package.name = "apache2"'

This should return all the information known by the registry about the package named "apache2" assuming it is installed.

Code Changes

Interestingly, the existing regitries themselves should not need any changes at all. Instead, the code changes should strictly consist of wrapping the registries behind an abstraction layer which could either be called directly by the application or through the DBus backend. So, most of the implementation will consist of extending the current coarse grained interface to a much finer grained interface.

As far as the plugins are concerned, it would actually be possible to make the changes backward compatible with the current implementation. However, it would be preferable to change the plugins in order to access the registry singleton instead of using private attributes from the plugin manager.

Migration

Optionally, it would be desirable for the plugins to be migrated in order to use the registry singleton. As far as the current registries are concerned, as mentionned above, they should not need any changes.

Test/Demo Plan

  1. Run the command: checkbox_registry 'package.name = "apache2"'
  2. This should return a package entry from the registry

Unresolved issues

It might be worthwhile to explore existing DBus browsers which could expose the information provided by the registry in a convenient way.


CategorySpec

QATeam/Specs/CheckboxConvertRegistryIntoDBusBackend (last edited 2009-06-01 22:15:35 by modemcable178)