ARMDeveloperEnvironment

Differences between revisions 13 and 44 (spanning 31 versions)
Revision 13 as of 2010-05-20 19:59:28
Size: 6263
Editor: bd7a3fd5
Comment:
Revision 44 as of 2010-06-08 10:57:12
Size: 16327
Editor: fw-unat
Comment: s/tbd/larc all around
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
 * '''Launchpad Entry''': [[https://blueprints.launchpad.net/ubuntu-arm/+spec/arm-m-development-tools]]  * '''Launchpad Entry''': [[https://blueprints.launchpad.net/ubuntu/+spec/arm-m-development-tools]]
Line 5: Line 5:
 * '''Contributors''': GuilhermeSalgado  * '''Contributors''': GuilhermeSalgado, JamesWestby
Line 10: Line 10:
We want to provide a simple yet powerful set of tools to allow ARM developers to easily create/manage archives and generate images. These tools will be implemented as a layer on top of utilities/frameworks that work on a lower level and thus require more knowledge to be operated.

These tools will have a high-level interface with sensible defaults so that developers don't have to learn the specifics of each utility/framework in order to get started.

== Release Note ==

?
We want to provide a simple yet powerful set of tools to allow ARM developers to easily create/manage archives (including package uploads/builds) and generate images.
Line 20: Line 14:
We need to allow ARM developers to easily create/maintain derivative archives for use during the development of new devices. They should be able to branch archives, make changes to them and manage deltas without having to learn the myriad of tools that are currently needed to perform such tasks. Organizations need to generate their own images for testing
and evaluation purposes, possibly including software that supports unreleased hardware or software with restricted redistribution rights. To generate such images they need to maintain an integrated set of software packages that can be installed on to their devices.

== Definitions ==

{{{larc}}} is the command that allow developers to use the tools described here.

See the Definitions section of [[Specs/M/ARMArchiveBranching|ARMArchiveBranching]] and [[Specs/M/ARMArchiveFeatures|ARMArchiveFeatures]]
Line 24: Line 25:
 * Ted wants to package the latest version of the Foo project, which depends on the latest version of Bar and Baz, which in turn are not yet available in Ubuntu. Instead of packaging Bar and Baz, Joe can use these dev tools to easily branch off of two existing PPAs that already have packages for the latest version of Bar and Baz. (This use case was invented to try and explain why we might want multiple parents, as discussed during the UDS session)

 * A partner is developing a custom netbook UI but need to keep it private until it's finalized, so they use the tools to create a slim archive off of the main Ubuntu archive and do any UI changes (or add new packages) on their new archive, which is only accessible by themselves. The slim archive is made possible (instead of a full one) because it should be binary compatible with the main Ubuntu archive. (Not yet clear whether or not the new archive should be hosted on LP, like a private PPA; probably need to allow it to live in LP or outside)

 * YAP (Yet Another Partner) is working on optimizing their new (not-yet-released) chip, but for that they need a version of GCC newer than the one on the Ubuntu archive. They want to create a new (private) archive where they'll upload the new GCC version, but upgrading to that new version of GCC is known to break binary compatibility, so it must be possible for them to easily rebuild all packages using the new GCC.

 * YAP also has a separate team working on the UI for a device which will use their new chip, so they want to have yet another archive, based on the one containing the new GCC, where they'll make their UI changes without affecting other users of the archive containing the new GCC. The tools should allow them to do that as well.

 * During the development of YAP's latest device, it should be possible for them to easily see the changes done to the upstream archive since both archives diverged. They should also be able to review those changes and pull the ones they want into their archive.

 * YAP has finished development of a new device and want to upstream the changes they've done when developing. They need to first see what are those changes, decide which ones should go upstream and submit them. Similarly, they should be able to easily pull some/all changes from the upstream archive.
 1. Ted wants to generate an image for marvel doves, which require non-free software packages that are not included in Ubuntu but instead are packaged in a marvel PPA. Ted must be able to create a new archive (by [[Specs/M/ARMArchiveBranching|branching]] the Ubuntu archive and marvel's PPA) and generate the image using the new archive.
 {{{
# Create a new archive containing the "standard" platform of Ubuntu's main archive.
larc branch http://archives.ubuntu.com/main https://archives.yap.com/marvel --platform=standard
# Append all packages from the PPA containing the private bits to the newly created archive.
larc append-archive https://launchpad.net/~yap/+archive/private https://archives.yap.com/marvel
# Generate an image to test the marvel doves.
larc gen-image https://archives.yap.com/marvel
 }}}

 1. A partner is experimenting with a custom netbook UI but doesn't want to include it in their main archive until they've done some more testing, so they use {{{larc}}} to create a slim archive and do any UI changes (or add new packages) there. Once they decide the UI has had enough testing, they push the changes from the slim archive back to their main one. (The slim archive could be hosted on LP, like a [private] PPA, as soon enough we'll be able to upload packages through sftp).
 {{{
# Create a new empty archive.
larc create https://archives.yap.com/new-netbook-ui
# Create a workspace associated with the new archive.
larc make-workspace https://archives.yap.com/new-netbook-ui
# Hack on an existing package or create a new one.
larc edit-package unity
cd unity
vi # That's how you do it, right? ;)
larc build # Make sure it builds correctly with your changes.
# Push the package to the archive.
larc push
 }}}

 1. YAP (Yet Another Partner) is working on optimizing their new (not-yet-released) chip, but for that they need a version of GCC newer than the one on the Ubuntu archive. They want to create a new (private) archive where they'll upload the new GCC version, but upgrading to that new version of GCC is known to break binary compatibility, so it must be possible for them to easily rebuild all packages using the new GCC and generate images out of the new binaries to ensure the resulting system works as expected.
 {{{
larc branch https://archives.u.c/main https://archives.yap.com/new-gcc --platform=standard
# Upload an already prepared gcc-4.5 package.
larc push gcc-4.5.dsc https://archives.yap.com/new-gcc
larc rebuild https://archives.yap.com/new-gcc # This is going to take ages!
larc gen-image https://archives.yap.com/new-gcc
 }}}

 1. YAP also has a separate team working on the UI for a device which will use their new chip, so they want to have yet another archive, based on the one containing the new GCC, where they'll make their UI changes without affecting other users of the archive containing the new GCC. {{{larc}}} must allow them to do that as well.
 {{{
larc branch https://archives.yap.com/new-gcc https://archives.yap.com/new-gcc-and-UI --platform=standard
 }}}

 1. During the development of YAP's latest device, it should be possible for them to easily see the delta between their archive and its upstream. They should also be able to review those changes and pull/push changes from/to the upstream archive.
 {{{
# Notice how we don't need to specify the parent as the archive has that information.
larc show-delta https://archives.yap.com/new-device
 }}}

 1. At the end of the development of their new device, YAP's engineers want to freeze their archive so that all package uploads have to be reviewed by their release team before they're accepted.
 {{{
larc freeze https://archives.yap.com/new-device
 }}}
Line 38: Line 77:
== Assumptions ==

 * Archives can be refered to by their URLs. (I think this is how we're going to tell the tools the archives they'll operate on)

 * There are two kinds of new archive that can be created: a 'full' archive which starts out containing every package in its parent archive at the version at the time of creation (the debian/ubuntu relationship is a bit like this) or an archive that starts out empty and expects that a user will have another archive to satisfy most package requests (the relationship between the primary archive and a PPA is like this)
Line 46: Line 79:
Goals:
 * provide a way to create new archives by branching a subset of ubuntu packages
 * make it easy to change these archives
 * visualize (through a web tool, probably) deltas between archives and their (direct or no?) upstreams/downstreams
 * manage these deltas by pulling/pushing changes from/to upstreams
 * Must allow either short/adhoc (e.g. upload a new upstream version of gstreamer, rebuild things against it and test that everything still works) or long running experiments (e.g. hardware enablement and the like, where you may take months to get things how you like before dropping it all in to Ubuntu at once when the hardware is announced)
 * deep hierarchies: common archive -> project archive -> project variant archive etc.
 * automatic superseding and merging, to keep the archive in sync with its upstream. Must also be able to stop the automatic superseding/merging at any point (e.g. when freezing for a release).
 * Need to be able to freeze an archive so that all uploads are held for approval by the release team.
 * a way of enforcing version number rules in a particular archive would be good. this is because we'll have to use fairly odd version numbering in downstream archives, so it'd be nice to have archives enforce their own rules for version numbering.
The goal here is to provide a high-level interface with sensible defaults, abstracting some of the low-level complexity of the underlying utilities/frameworks.

{{{larc}}} should be transparent and allow people to access its inner workings. We should aim to ensure it is always possible to complete a task manually.

Users must be able to run {{{larc}}} on either a desktop or a server. Also, it must not be tied to Launchpad, although it will take advantage of anything provided by Launchpad whenever desirable.

{{{larc}}} will be able to operate on both local and remote archives, for both read and write operations. When operating on remote archives it will use the HTTP RESTFUL API of the service hosting the archive (wich can be either Launchpad.net or a [[Specs/M/ARMArchiveBranching|vostok]] instance). To upload packages we'll use sftp.

{{{larc}}} must not try to enforce any sort of version number rules, but it will provide appropriate version numbers by default so that users don't need to worry about that.

The following are some of the main tasks that {{{larc}}} must facilitate.

 * Interacting with an archive
 * Modifying packages
 * Image building
 * Updating a derived archive

=== Interacting with an archive ===

This category covers operations on remote archives, such as finding current versions of packages, removing packages, requesting rebuilds, freezing an archive, and indeed branching an archive.

This should make use of an HTTP API exposed by the archive management software that allows querying and modifications.

For each logical operation that a user may wish to do there should be a command or subcommand provided by {{{larc}}}, which it will then map to the necessary API calls. Therefore most of the logic about how the operations work will be in the archive management software, and {{{larc}}} just needs to handle making the correct API call, presenting the information to the user, and handling error conditions.

We will strive to get good coverage of typical operations in the {{{tdb}}} tool, but asking users to go to the web UI of the archive management service is acceptable, and will be necessary if new features are added that old versions of {{{tdb}}} do not support.


=== Modifying packages ===

One of the most common operations will be modifying a package. {{{larc}}} will provide commands that make it easy to get a copy of the current version of the package, make changes to it, build it locally for testing, and then commit the change.

Ideally we should support both pushing the change directly to an archive, and for submission for review for those that either don't have upload rights to the archive, or would like peer review before making the change.

While editing files and the like won't be abstracted, {{{larc}}} can provide wrapper commands for test-building a package, adding a changelog entry and uploading, merging a new upstream version, and merging from a parent archive, amongst others.

There should also be a way to submit the change to a parent archive for review very easily, so that the change can be made in the archive that the developer is targeting, but also start making its way in to parent archives at the same time.

We may also want to provide environments other than the host system in order to build packages. chroots or virtual machines are important as soon as you are building for a target that is different to the host in terms of package versions and the like. Clearly when building for a different architecture this needs to be done, in which case we should interface with UbuntuSpec:arm-m-xdeb-cross-compilation-environment.

=== Image building ===

{{{larc}}} will tie in to the results of UbuntuSpec:arm-m-image-building-tool such that a developer can easily build themselves a test image, including results of their test builds.

In addition, {{{larc}}} will tie in to UbuntuSpec:arm-m-image-building-console such that they can also request image builds from a service. Here it would be needed to host all the packages remotely so that the image building service could make use of them.

Furthermore for the developer's workflow it would be ideal if they could submit a package to build in the archive, and at the same time queue an image build request that would start if and when the package built successfully. That would save them having to switch context too often.

=== Updating a derived archive ===

{{{larc}}} will also allow the developer to make use of the features of the archives described in UbuntuSpec:arm-m-archive-branching. It will allow them to visualise the difference between an archive and its parent, and then act on the result as well.

Crucially it should allow them to do three things:

  * Request a sync of a package.
  * Submit a change to the parent archive in the appropriate manner.
  * Merge a package from the parent where the package was modified in both archives.

As the last two operations can't be done through the web UI described in the other spec it is important that {{{larc}}} make that part easy for developers to do.

=== Workspaces ===

A workspace is an area on disk that {{{larc}}} can create which encapsulates a specific configuration. This allows {{{larc}}} to infer lots of information when it is in a workspace and save the developer time in remembering and typing some of those details.

A workspace will be tied to a certain archive, and so by default act on that archive.

The developer can maintain as many workspaces as they like locally, and {{{cd}}} between them in order to work on different archives.

In addition to this the workspace can contain a local archive that can be used in addition to the remote one. This is used to allow the developer to do things like build an image containing some test changes, or to build two packages locally where the second depends on some new API just added to the first, all without having to upload experimental changes to the archive for others to see. We may also want to allow developers to push packages from their development archive to a PPA for sharing with others, or even use a PPA for this if they like.
Line 59: Line 151:
=== Interacting with an archive ===

Operations accessing remote locations through HTTP will need to be authenticated, so {{{larc}}} will have a per-user registry of credentials that can be used for OAuth authentication. There will be one set of credentials for each remote location, but when such credentials don't exist {{{larc}}} will guide the user, via their web browser, through the process of obtaining them.

Once it has the credentials it can make the necessary authenticated API calls to perform the requested operation.

It must understand enough of the response that it will receive from the server to present the results meaningfully to the developer, and also provide useful error messages where possible.

=== Workspaces ===

Any directory containing a .larc.conf file is considered a workspace and {{{larc}}} will use the archive specified there when one is not explicitly provided. The file will also be able to store other configuration defaults for that workspace, such as extra archives (e.g. PPAs) that should be included in images built from that workspace.

Workspaces are most useful when modifying packages or building images, so they can also cache .deb packages (for image building) and bazaar working trees for the source packages in the archive.

We need to properly lay out bazaar working trees and source/binary packages (resulting from test builds) in workspaces, to keep things sane. Here is an example of how that could look.

 {{{
  .
  |-- workspace root
  | |-- unity (bzr working tree)
  | | `-- README.txt
  | |-- gcc (bzr working tree)
  | | `-- HACKING.txt
  | `-- testbuilds
  | |-- Packages (makes it an archive that image builders can use)
  | |-- unity-N.NN-x86.deb
  | `-- gcc-N.NN-x86.deb
 }}}

=== Modifying packages ===

To get the source for a given package we'll use bazaar and fetch the branch associated with the given package, placing it in the current directory.

To generate a binary package from a bazaar working tree, we'll use bzr-builddeb to build a source package from the tree and then build that into a binary package.

When pushing the given bazaar working tree to its parent branch, the facilities of the target archive management system will define how the binary package building happens:
  * If it has full building facilities then we request a build of the branch to a source/binary package.
  * If it can only build binaries then we build a source package locally and upload that, which will then trigger a binary build.
  * Otherwise we also upload source and binary packages.

=== Image building ===

For building images we'll use the [[Specs/M/ARMImageBuildingTool|image building tool]] to generate an image containing the packages specified by the user.

In order to build an image we'll need to fetch lots of binary packages from the archive, so it makes sense to cache these binary packages locally for further images built. These will be cached in the workspace.

=== Updating a derived archive ===

When requesting a sync of a package, submitting changes to the parent or merging a package from the parent, {{{larc}}} will just make the appropriate HTTP request and let the archive management systems take care of the rest.

When submitting changes to the parent of merging a package from the parent, {{{larc}}} assumes there are no conflicts between the changes. ('''Is it reasonable to assume that? -- GuilhermeSalgado''')

To help users doing the above, {{{larc}}} can show the list of modified packages in each category, and then allow the user to choose one to work on.
Line 62: Line 207:
Most tools will probably have no UI other than their command-line arguments, except for showing deltas between archives, in which case we'll probably want a new UI in Launchpad. That UI should also allow to push/pull part of the delta upstream/downstream.
Mostly {{{larc}}} will just have command-line arguments and status output.

There will be sometimes when it may need to present the user a list of options, so we should design how that would look.
Line 67: Line 213:
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.
There is clearly a lot of testing that will be required. We will make a lot of use of unit testing, and also dogfooding by developers.

Each feature should be explicitly tested as it is included though, and integration tests included where possible.
Line 73: Line 219:
 * We'll need multiple OAuth credentials (e.g. for launchpad.net, archives.yap.com, etc), so we need to figure out which credentials to use depending on the arguments given.

 * If we go with OAuth for the authentication, it means the user will need a browser to obtain the OAuth credentials, so it will be tricky to run {{{larc}}} on a server. To workaround that we can either copy existing credentials to the server or ask the user for their password and do the oauth dance ourselves (a la ground control). The latter is a really nasty trick, IMO.

 * For long running operations, do we want to try and design some sort of progress report or should we rely just on an email sent by vostok when it's done?

 * Maybe the caching of binary packages should be the responsibility of [[Specs/M/ARMImageBuildingTool|ARMImageBuildingTool]]?

 * Is supporting the local archive in a workspace in all the tools going to be a lot of work? Should there be a way for it to be transparently included instead?

Summary

We want to provide a simple yet powerful set of tools to allow ARM developers to easily create/manage archives (including package uploads/builds) and generate images.

Rationale

Organizations need to generate their own images for testing and evaluation purposes, possibly including software that supports unreleased hardware or software with restricted redistribution rights. To generate such images they need to maintain an integrated set of software packages that can be installed on to their devices.

Definitions

larc is the command that allow developers to use the tools described here.

See the Definitions section of ARMArchiveBranching and ARMArchiveFeatures

User stories

  1. Ted wants to generate an image for marvel doves, which require non-free software packages that are not included in Ubuntu but instead are packaged in a marvel PPA. Ted must be able to create a new archive (by branching the Ubuntu archive and marvel's PPA) and generate the image using the new archive.

    # Create a new archive containing the "standard" platform of Ubuntu's main archive.
    larc branch http://archives.ubuntu.com/main https://archives.yap.com/marvel --platform=standard
    # Append all packages from the PPA containing the private bits to the newly created archive.
    larc append-archive https://launchpad.net/~yap/+archive/private https://archives.yap.com/marvel
    # Generate an image to test the marvel doves.
    larc gen-image https://archives.yap.com/marvel
  2. A partner is experimenting with a custom netbook UI but doesn't want to include it in their main archive until they've done some more testing, so they use larc to create a slim archive and do any UI changes (or add new packages) there. Once they decide the UI has had enough testing, they push the changes from the slim archive back to their main one. (The slim archive could be hosted on LP, like a [private] PPA, as soon enough we'll be able to upload packages through sftp).

    # Create a new empty archive.
    larc create https://archives.yap.com/new-netbook-ui
    # Create a workspace associated with the new archive.
    larc make-workspace https://archives.yap.com/new-netbook-ui
    # Hack on an existing package or create a new one.
    larc edit-package unity
    cd unity
    vi  # That's how you do it, right? ;)
    larc build  # Make sure it builds correctly with your changes.
    # Push the package to the archive.
    larc push
  3. YAP (Yet Another Partner) is working on optimizing their new (not-yet-released) chip, but for that they need a version of GCC newer than the one on the Ubuntu archive. They want to create a new (private) archive where they'll upload the new GCC version, but upgrading to that new version of GCC is known to break binary compatibility, so it must be possible for them to easily rebuild all packages using the new GCC and generate images out of the new binaries to ensure the resulting system works as expected.
    larc branch https://archives.u.c/main https://archives.yap.com/new-gcc --platform=standard 
    # Upload an already prepared gcc-4.5 package.
    larc push gcc-4.5.dsc https://archives.yap.com/new-gcc
    larc rebuild https://archives.yap.com/new-gcc  # This is going to take ages!
    larc gen-image https://archives.yap.com/new-gcc
  4. YAP also has a separate team working on the UI for a device which will use their new chip, so they want to have yet another archive, based on the one containing the new GCC, where they'll make their UI changes without affecting other users of the archive containing the new GCC. larc must allow them to do that as well.

    larc branch https://archives.yap.com/new-gcc https://archives.yap.com/new-gcc-and-UI --platform=standard 
  5. During the development of YAP's latest device, it should be possible for them to easily see the delta between their archive and its upstream. They should also be able to review those changes and pull/push changes from/to the upstream archive.
    # Notice how we don't need to specify the parent as the archive has that information.
    larc show-delta https://archives.yap.com/new-device
  6. At the end of the development of their new device, YAP's engineers want to freeze their archive so that all package uploads have to be reviewed by their release team before they're accepted.
    larc freeze https://archives.yap.com/new-device

Some of the features described here will depend on DerivedArchiveRebuild

Design

The goal here is to provide a high-level interface with sensible defaults, abstracting some of the low-level complexity of the underlying utilities/frameworks.

larc should be transparent and allow people to access its inner workings. We should aim to ensure it is always possible to complete a task manually.

Users must be able to run larc on either a desktop or a server. Also, it must not be tied to Launchpad, although it will take advantage of anything provided by Launchpad whenever desirable.

larc will be able to operate on both local and remote archives, for both read and write operations. When operating on remote archives it will use the HTTP RESTFUL API of the service hosting the archive (wich can be either Launchpad.net or a vostok instance). To upload packages we'll use sftp.

larc must not try to enforce any sort of version number rules, but it will provide appropriate version numbers by default so that users don't need to worry about that.

The following are some of the main tasks that larc must facilitate.

  • Interacting with an archive
  • Modifying packages
  • Image building
  • Updating a derived archive

Interacting with an archive

This category covers operations on remote archives, such as finding current versions of packages, removing packages, requesting rebuilds, freezing an archive, and indeed branching an archive.

This should make use of an HTTP API exposed by the archive management software that allows querying and modifications.

For each logical operation that a user may wish to do there should be a command or subcommand provided by larc, which it will then map to the necessary API calls. Therefore most of the logic about how the operations work will be in the archive management software, and larc just needs to handle making the correct API call, presenting the information to the user, and handling error conditions.

We will strive to get good coverage of typical operations in the tdb tool, but asking users to go to the web UI of the archive management service is acceptable, and will be necessary if new features are added that old versions of tdb do not support.

Modifying packages

One of the most common operations will be modifying a package. larc will provide commands that make it easy to get a copy of the current version of the package, make changes to it, build it locally for testing, and then commit the change.

Ideally we should support both pushing the change directly to an archive, and for submission for review for those that either don't have upload rights to the archive, or would like peer review before making the change.

While editing files and the like won't be abstracted, larc can provide wrapper commands for test-building a package, adding a changelog entry and uploading, merging a new upstream version, and merging from a parent archive, amongst others.

There should also be a way to submit the change to a parent archive for review very easily, so that the change can be made in the archive that the developer is targeting, but also start making its way in to parent archives at the same time.

We may also want to provide environments other than the host system in order to build packages. chroots or virtual machines are important as soon as you are building for a target that is different to the host in terms of package versions and the like. Clearly when building for a different architecture this needs to be done, in which case we should interface with arm-m-xdeb-cross-compilation-environment.

Image building

larc will tie in to the results of arm-m-image-building-tool such that a developer can easily build themselves a test image, including results of their test builds.

In addition, larc will tie in to arm-m-image-building-console such that they can also request image builds from a service. Here it would be needed to host all the packages remotely so that the image building service could make use of them.

Furthermore for the developer's workflow it would be ideal if they could submit a package to build in the archive, and at the same time queue an image build request that would start if and when the package built successfully. That would save them having to switch context too often.

Updating a derived archive

larc will also allow the developer to make use of the features of the archives described in arm-m-archive-branching. It will allow them to visualise the difference between an archive and its parent, and then act on the result as well.

Crucially it should allow them to do three things:

  • Request a sync of a package.
  • Submit a change to the parent archive in the appropriate manner.
  • Merge a package from the parent where the package was modified in both archives.

As the last two operations can't be done through the web UI described in the other spec it is important that larc make that part easy for developers to do.

Workspaces

A workspace is an area on disk that larc can create which encapsulates a specific configuration. This allows larc to infer lots of information when it is in a workspace and save the developer time in remembering and typing some of those details.

A workspace will be tied to a certain archive, and so by default act on that archive.

The developer can maintain as many workspaces as they like locally, and cd between them in order to work on different archives.

In addition to this the workspace can contain a local archive that can be used in addition to the remote one. This is used to allow the developer to do things like build an image containing some test changes, or to build two packages locally where the second depends on some new API just added to the first, all without having to upload experimental changes to the archive for others to see. We may also want to allow developers to push packages from their development archive to a PPA for sharing with others, or even use a PPA for this if they like.

Implementation

Interacting with an archive

Operations accessing remote locations through HTTP will need to be authenticated, so larc will have a per-user registry of credentials that can be used for OAuth authentication. There will be one set of credentials for each remote location, but when such credentials don't exist larc will guide the user, via their web browser, through the process of obtaining them.

Once it has the credentials it can make the necessary authenticated API calls to perform the requested operation.

It must understand enough of the response that it will receive from the server to present the results meaningfully to the developer, and also provide useful error messages where possible.

Workspaces

Any directory containing a .larc.conf file is considered a workspace and larc will use the archive specified there when one is not explicitly provided. The file will also be able to store other configuration defaults for that workspace, such as extra archives (e.g. PPAs) that should be included in images built from that workspace.

Workspaces are most useful when modifying packages or building images, so they can also cache .deb packages (for image building) and bazaar working trees for the source packages in the archive.

We need to properly lay out bazaar working trees and source/binary packages (resulting from test builds) in workspaces, to keep things sane. Here is an example of how that could look.

  •   .
      |-- workspace root
      |   |-- unity  (bzr working tree)
      |   |   `-- README.txt
      |   |-- gcc    (bzr working tree)
      |   |   `-- HACKING.txt
      |   `-- testbuilds
      |       |-- Packages (makes it an archive that image builders can use)
      |       |-- unity-N.NN-x86.deb
      |       `-- gcc-N.NN-x86.deb

Modifying packages

To get the source for a given package we'll use bazaar and fetch the branch associated with the given package, placing it in the current directory.

To generate a binary package from a bazaar working tree, we'll use bzr-builddeb to build a source package from the tree and then build that into a binary package.

When pushing the given bazaar working tree to its parent branch, the facilities of the target archive management system will define how the binary package building happens:

  • If it has full building facilities then we request a build of the branch to a source/binary package.
  • If it can only build binaries then we build a source package locally and upload that, which will then trigger a binary build.
  • Otherwise we also upload source and binary packages.

Image building

For building images we'll use the image building tool to generate an image containing the packages specified by the user.

In order to build an image we'll need to fetch lots of binary packages from the archive, so it makes sense to cache these binary packages locally for further images built. These will be cached in the workspace.

Updating a derived archive

When requesting a sync of a package, submitting changes to the parent or merging a package from the parent, larc will just make the appropriate HTTP request and let the archive management systems take care of the rest.

When submitting changes to the parent of merging a package from the parent, larc assumes there are no conflicts between the changes. (Is it reasonable to assume that? -- GuilhermeSalgado)

To help users doing the above, larc can show the list of modified packages in each category, and then allow the user to choose one to work on.

UI Changes

Mostly larc will just have command-line arguments and status output.

There will be sometimes when it may need to present the user a list of options, so we should design how that would look.

Test/Demo Plan

There is clearly a lot of testing that will be required. We will make a lot of use of unit testing, and also dogfooding by developers.

Each feature should be explicitly tested as it is included though, and integration tests included where possible.

Unresolved issues

  • We'll need multiple OAuth credentials (e.g. for launchpad.net, archives.yap.com, etc), so we need to figure out which credentials to use depending on the arguments given.
  • If we go with OAuth for the authentication, it means the user will need a browser to obtain the OAuth credentials, so it will be tricky to run larc on a server. To workaround that we can either copy existing credentials to the server or ask the user for their password and do the oauth dance ourselves (a la ground control). The latter is a really nasty trick, IMO.

  • For long running operations, do we want to try and design some sort of progress report or should we rely just on an email sent by vostok when it's done?
  • Maybe the caching of binary packages should be the responsibility of ARMImageBuildingTool?

  • Is supporting the local archive in a workspace in all the tools going to be a lot of work? Should there be a way for it to be transparently included instead?


CategorySpec

Specs/M/ARMDeveloperEnvironment (last edited 2010-06-08 10:57:12 by fw-unat)