Bileto

Differences between revisions 65 and 126 (spanning 61 versions)
Revision 65 as of 2015-08-30 03:34:36
Size: 29597
Editor: 1
Comment:
Revision 126 as of 2021-02-10 09:39:58
Size: 37582
Editor: sil2100
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## page was renamed from citrain/LandingProcess
Line 3: Line 4:
= TODO =

Some information can be found at [[citrain/FAQ]] and [[citrain/NewbieGuide]] that should probably be merged here.

= Landing changes through the CI Train =

Hello lander! If you got here, you're probably looking for information about how to use the CI Train or what to do to get your project landed to Ubuntu Touch. For general information about the usage of the train, make your way to the [[#landing|Landing your change to Ubuntu]] section below. If you already know the basics of the train and just want to know which of the tools and processes to use for your specific landing, the [[#approaches|Which landing method to use?]] section should help you.

= Landing changes through Bileto =

Hello lander! If you got here, you're probably looking for information about how to use Bileto (formerly known as the "CI Train") or what to do to get your project landed to Ubuntu. For general information, make your way to the [[#landing|Landing your change to Ubuntu]] section below. If you already know the basics and just want to know which of the tools and processes to use for your specific landing, the [[#approaches|Which landing method to use?]] section should help you.
Line 12: Line 10:

'''IMPORTANT!''' With the sunset of the Ubuntu Touch project, the main use-case of Bileto as a MP-based workflow became much less common (and useful), and now its purpose shifted more towards a manual package-staging helper with ephemeral PPAs and autopkgtests runners. For more information about this particular use case, see the [[#manual|Manual upload]] section of the documentation.
Line 16: Line 17:
The train provides two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is managed by the CI Train. There are two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is maintained by Canonical.
Line 20: Line 21:
 * If this is your first time running a landing, ask the '''trainguards''' in '''#ubuntu-ci-eng''' to give you the required permissions for operating the CI Train.
 * I
f this is the first time this source package has ever been released in Ubuntu, refer to the [[https://wiki.ubuntu.com/DailyRelease/InlinePackaging|Inline Packaging]] guide to bring your packaging up to the train's expectations.
 * Open the [[https://requests.ci-train.ubuntu.com|CI Train requests page]].
 * Sign in with SSO, and you'll be presented with a form to fill out.
 * If this is your first time running a landing, ask the '''trainguards''' in '''#ubuntu-ci-eng''' to give you the required permissions for operating Bileto.
 *
If this is the first time this source package has ever been released with Bileto, refer to the [[https://wiki.ubuntu.com/DailyRelease/InlinePackaging|Inline Packaging]] guide to meet the minimum packaging standards.
 * Open the [[https://bileto.ubuntu.com|tickets page]].
 * Sign in with SSO, click '''Create New Request''', and you'll be presented with a form to fill out.
Line 25: Line 26:
   * Test plans to run: each project handled by CI Train '''needs''' to have a test plan that documents how you, the lander, intend to test the given component(s). It's best to present this in the form of a wiki page or other document that can be referred to outside the train. See [[#QA]] for more details.    * Test plans to run: this mandatory field documents how you, the lander, intend to test the given component(s). Generally you would put a URL to a document describing steps to test that your component functions correctly, as this information will generally not change much from one ticket to the next. See [[#QA]] for more details.
Line 27: Line 28:
   * Manual Download URLs: This optional field contains click packages / tarball updates that don't require a PPA for testing.
   * Name of assigned silo: Leave this blank, the train will fill it out for you.
   * Status: Leave this blank also, the train will fill it out as well.
   * Landers: List of people who care about this landing. These names will be pinged on IRC when silo status changes, so use IRC nicks if possible (the field defaults to containing your launchpad nickname, I recommend setting an IRC highlight for your lp nick if it's different from your IRC nick).
   * Sync Source: This is a special field used only when you are copying packages from one series to another. Hover your mouse over the field to see some example syntax.
   * Destination PPA: select or type in the PPA to which the packages should be released instead of the main archive - currently mostly used only for cases when landing to the stable overlay PPA is needed. See [[#overlay|Landing to the Stable Overlay PPA]]. If you are doing a touch-specific landing for vivid, you'll want to specify '''ci-train-ppa-service/ubuntu/stable-phone-overlay'''.
   * Additional source packages to land: here you list the 'additional' source packages that you would like to upload to the silo PPA besides what is already handled by MPs. Can be used for projects that Ubuntu is not upstream for or which have no bzr trunk.
   * Manual Download URLs: This optional field contains click packages / device tarball updates that don't require a PPA for testing, but instead must be downloaded manually by the tester.
   * Landers: List of people who care about this landing. These names will be pinged on IRC when ticket status changes, so use IRC nicks if possible.
   * Source Package Names: You can leave this blank and the correct value will auto-fill based on the contents of the PPA.
Line 35: Line 32:
   * Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select [[dual|Dual-Landing]].
   * QA signoff status: the rule is: any stable-phone targets like vivid or 'dual' landings by default require it. See [[#QA]] for more information. Set to either 'No QA Needed' or 'QA Required' to start. See [[#qastatus|QA Signoff Status field]] for more information.
 * Once the request is filled out and you've clicked 'Save', you can assign the silo in jenkins by clicking 'Assign' from the menu that hovers beneath your row when you mouse over it, and then 'Build' on the resulting jenkins job.
 * Once your silo is assigned, the hover menu will grow three new links, 'Build', 'Publish', and 'Merge&Clean'. Use the 'Build' button to start building packages from the specified merge requests.
   * If you have added some 'Additional source packages to land', you need to provide the trainguards or Ubuntu core-dev with the required source package so that they can upload those to your silo. Any core-developer should be able to dput packages to the silo PPAs as well.
   * Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select '''yakkety+xenial+vivid'''.
   * QA signoff: Values "N/A", "Required", or "Ready" will be automatically set as appropriate for your landing. The rule is: any stable-phone targets like vivid or 'yakkety+xenial+vivid' landings by default require it. See [[#QA]] for more information.
 * Once the request is filled out and you've clicked 'Create New Request', you can assign a PPA by clicking 'Assign' from the menu at the bottom of your new ticket.
 * Once your PPA is assigned, the menu will show some new buttons. Use the 'Build' button to start building packages from your merge proposals.
   * If you have manual packages not built from merges, you need a trainguard or Ubuntu core-dev to upload those to your PPA.
Line 41: Line 38:
 * If your packages pass your testing criteria, mark the silo as tested:
   * Get all your MP's top-approved. If you don't use the feature in your team's processes, just mark it as top-approved yourself.
   * Hover over your landing request and click 'Edit' in the hover menu.
   * Set the QA Signoff Status dropdown to either 'Publish without QA' or 'Ready for QA' as appropriate (generally speaking if your silo only targets the ubuntu devel release, you don't need QA, but if you target the stable release or 'dual' then you do need QA). See [[#qastatus|QA Signoff Status field]] for more information.
 * In case your silo needs QA sign-off, wait for the QA Team to finish testing on the silo. This will be indicated in the requests and the dashboard.
 * The landing team will now publish the silo to the archive if everything is OK packaging-wise. Note: ''Merge Proposals need to be top-approved latest at this point, otherwise landing team cannot publish the silo.''

This is the standard process. When you prepare a list of MPs to land and provide them to CI Train, the infrastructure will merge all of them together and build a source package using bzr bd. This package is then uploaded to the silo PPA that has been assigned to your landing. CI Train also automatically generates the debian/changelog for you using the MPs' commit-messages. Every landing creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time.
After all changes land into the archive, during the 'Merge & Clean' step these merges are all finally merged into trunk.
 * If your packages pass your testing criteria, mark the ticket as tested:
   * Get all your MP's top-approved. From Bileto's perspective, you are free to approve your own merges, however in some teams this is frowned upon, so you must get your branches reviewed by your teammates.
   * Set 'Lander Signoff' to 'Approved'.
 * Britney will now automatically inspect your PPA and perform various installability checks and autopkgtests. Once britney approves your ticket, it will automatically be marked ready for QA, if appropriate.
 * If your ticket says "QA Signoff: Ready" you must wait for a QA person to approve your ticket.
 * Once your ticket has been approved by both Britney and QA (or if QA is N/A), you can publish your own ticket. Landers are permitted to publish their own PPAs so long as there are no packaging changes, and no manual sources in the PPA. In the event that you do have either packaging changes or manual sources, it is your responsibility to find a core dev to publish the PPA for you.

This is the standard process. When you prepare a list of MPs to land, the infrastructure will merge all of them together and build a source package using dpkg-buildpackage. This source package is then uploaded to the PPA that has been assigned to your ticket. The debian/changelog is automatically generated for you using the Commit Message field of each MP. Every ticket creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time.
After all changes land into the archive, these merges are automatically merged into trunk.
Line 55: Line 52:
 1. Build packages in silo PPA
 1. Test packages from the silo PPA on real hardware
 1. Mark silo as tested
 1. (in some cases) Wait for silo to be QA signed-off
 1. Wait for silo to be published

<<Anchor(qastatus)>>
=== The QA Signoff Status field ===

The current Bileto implementation has one field to indicate the overall sign-off status of a landing. This means that information of if the silo needs QA sign-off or not is in the same place as the information if the silo has been tested by the lander and is ready to publish. This field is a drop-down list named 'QA Signoff Status'. Possible statuses:

 * '''No QA Needed''' - the silo is '''not''' tested yet by the lander, but once it is, it does not require additional QA sign-off.
 * '''Publish without QA''' - the silo that did not require QA sign-off is now tested by the lander and is ready to be published.
 * '''QA Required''' - the silo is '''not''' tested yet by the lander and once it is, it will require an additional QA sign-off.
 * '''Ready for QA''' - the silo is now tested by the lander and is ready for QA sign-off. Please note that only now the silo appears on the QA queue.
 * '''QA Granted''' - the silo has passed both the lander testing and QA sign-off and is ready to publish.
 * '''QA Failed''' - the silo failed QA sign-off.

=== CI Train for manual source uploads ===

Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released as a silo: either because of the QA sign-off requirement for touch, or simply because they're needed by a landing, mixed-in with normal MP-based requests.

In this case, the "Additional source packages to land" field needs to be used.

 1. In your request, put the name of the source package in the Additional source packages to land field (space-separated)
 1. Fill in the rest of the request as normal
   * Remember you can mix merge-proposals and source uploads at will
   * Currently source uploads are not supported for dual-landings and sync silos
 1. Silo gets assigned
 1. Build packages in PPA
 1. Test packages from the PPA on real hardware
 1. Mark ticket as Lander Signoff: Approved
 1. Wait for Britney to mark the ticket Approved.
 1. (in some cases) Wait for QA to mark the ticket Approved
 1. Publish the ticket yourself
 1. (in some cases) Find a core dev to ACK packaging changes and publish the ticket for you



<<Anchor(manual)>>
=== Manual Source Uploads ===

Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released through Bileto: either because of the QA sign-off requirement for touch, or simply because they're needed by a ticket, mixed-in with normal MP-based packages.

 1. Create a new ticket. In this case most fields are optional and do not have to be filled up
 1. Press 'Build' on the empty ticket - this creates the PPA assigned to the ticket
Line 86: Line 71:
   * If you're a core-dev: simply use dput to upload the package to the silo PPA
   * In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you
 1. Once the package is uploaded, run the Build job with WATCH_ONLY set
   * This will make the train scan the PPA for new uploads and check if they build or built correctly
 1. Proceed normally with rest
   * If you're a core-dev: simply use dput to upload the package to the PPA
   * In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you (one option is to dput to your personal PPA and then core-dev or trainguard can copy the package from there).
 1. Proceed normally with building & testing


=== Manual Source Uploads for the Security Pocket ===

Sometimes performing an upload that is meant to go to the respective -security pocket is required. Bileto PPAs try to be as close to the archive as possible, so that one can freely bin-copy packages directly to the Ubuntu archive. By default, Bileto PPAs are configured to build with -proposed enabled (like any packages built in the Proposed pocket), but the settings of package dependencies can be manually adjusted per-PPA (Bileto will not touch these settings on its own):

 1. Create a ticket for manual uploads, as usual
   * Be sure to select the 'Target Series' to something that's not the overlay PPA
 1. Once the PPA is created, go to the PPA's main view and then to the 'Edit PPA dependencies' settings
 1. Switch the Ubuntu dependencies to 'Security' and make sure no other PPA dependencies are present
 1. Press Save and proceed normally with building & testing
Line 95: Line 90:
The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. In other words, it's the replacement for the ubuntu-rtm/14.09 series. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid. The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.
Line 99: Line 94:
By default, whenever a landing request is set for vivid and the packages affected (or it's obvious from the landing description) are touch-specific, you should set {{{ci-train-ppa-service/ubuntu/stable-phone-overlay}}} as the value for the 'Destination PPA' field. This means that once the package is published it will be copied to the overlay PPA instead of migrating to the vivid archive.

=== Syncing from and to the Stable Overlay PPA ===

A "sync" silo should only be used if you want to copy an existing package between wily and the overlay ppa (in either direction). If you have a single package that you want to land in both targets, please use [[#dual|Dual-Landings]].

'''WARNING!''' Sync silos should only be used for packages that are released through the train and include the train-generated version number. For packages that are either released through train source uploads or standard archive-uploads, please copy the packages manually with version numbers changed. See the rationale for this in [[citrain/SyncSilos]].

Currently syncing can work in both ways - from vivid to wily and the other way around. The recommended way is to first land your change to wily and then syncing it back to vivid-overlay - as per the standard procedure of first developing for devel. This can be done in a few ways. The recommended one is:

 1. Request two silos - one for vivid-overlay and one for wily
   * Put your merges to the wily silo
   * For the second request only put the silo number of the first request in the 'sync' field.
 1. A trainguard will first assign the main silo and then configure the second silo to fetch (sync) source packages from the first one
 1. Build your main silo
 1. Test your main silo
 1. Once you're happy with your main silo, go to the secondary one and press build
   * The build job will sync packages from the first silo and rebuild them for the new series
 1. Test your secondary silo
   * If there are issues with the silo on the other platform, you will have to rebuild both silos and re-test again
 1. Once you are happy with both silos, set both of them as ready

Another possible way, but less safe as one might forget about syncing their work to the other archive, is this:

 1. Perform a normal landing to wily and complete it the normal way.
 1. Request a new silo, a sync silo - no merges, only 'sync:ubuntu,wily' in the sync field and fill out the list of additional sources with the source packages you want synced.
 1. Build and test the silo
 1. Mark as tested and wait for QA sign-off

All sync possibilities are briefly overviewed on [[citrain/SyncSilos]].
For a trio ticket such as yakkety+xenial+vivid, the destination field is ignored. The the first series '''never''' goes to the stable overlay PPA, and the others '''always''' do.
Line 131: Line 97:
== Dual-landing for stable and devel ==

The current development series is wily. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep wily in sync, the CI Train offers functionality for performing dual landings to both overlay and wily through one silo. This is called dual-landing.

'''NOTICE!''' Remember that all dual landings are required to go through the [[QA|QA sign-off process]] because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your silo won't be released as a dual landing. What the trainguards can do for you is release the wily version and then request you to sync your work back to the vivid stable-overlay once the gates are open.

 1. Fill in a standard landing request
   * The filled merges need to point to your trunk (devel) branch, as the merges will be built against wily and then duplicated to vivid
 1. In the Target Series drop-down select "dual"
 1. Wait for silo assignment
 1. Once assigned, build the silo
   * The build job
will create 2 source packages - one for wily, one for vivid - and upload them to the silo PPA
   * The primary target is always wily
   * Your merges wil
l be built for wily (with the changelog entries generated for wily), but then the sources will be duplicated (with re-written versions) for vivid
   * If a package build fails for one series, it needs to be fi
xed in the merge and packages in all series in the silo need to be rebuilt
== Trio landing for stable and devel ==

The current development series is yakkety. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep yakkety and xenial in sync, Bileto allows tickets to be built for all three together automatically.

'''NOTICE!''' Remember that all trio landings are required to go through the [[QA|QA sign-off process]] because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your ticket won't be released as a trio landing. What the trainguards can do for you is release the yakkety version and then request you to sync your work back to the vivid stable-overlay once the gates are open.

 1. Fill in a standard ticket
   * The merges need to point to your trunk (devel) branch, as the merges will be built against yakkety and then copied to xenial and vivid
 1. In the Target Series drop-down select "yakkety+xenial+vivid"
 1. Assign your own PPA
 1. Build the ticket
 
* The build job will create 3 source packages - one for yakkety, one for xenial, and one for vivid - and upload them to the PPA
   * The prim
ary target is always yakkety
   * Your merges
will be built for yakkety (with the changelog entries generated for yakkety), but then the sources will be duplicated (with re-written versions) for xenial and vivid
Line 147: Line 112:
   * Please test both wily and vivid
 1. Mark the silo as tested
 1. Wait for silo to be QA signed-off
 1. Wait for the silo to be published and auto-merged
   * Once published, the silo merge and clean step will be blocked until '''all''' packages migrate to their destinations

<<Anchor(approaches)>>
== Which landing method to use? ==

Seeing how many different scenarios and landing techniques are available, sometimes it might be confusing as to which methods to use to land your change to the right place. Currently there are two main landing areas: devel and stable. Devel is a place for all the risky changes that are not yet ready for mainstream usage while stable is what all the shipped user phones are using.

=== My project only has stable features ===

In this case, and probably in most other ones, the best way is to use Dual-landing. Your project should only have one trunk and all your merges should be targeting it.

=== My project had only stable features but now it will diverge ===

Up until now you probably used dual-landings, which is good, but now you need to release different things to devel and stable.

Currently this is a bit of a tricky situation. Normally you should now diverge your trunk into two branches: leave the current one as devel and create a stable one for stable. But we need to trick CI Train into something before this can work. Since you used trunk to dual-release, the current top-most version of the branch has a devel (currently wily) version number, while stable uses a smaller versioning number. This can cause issues in the train saying things like: cannot release a version lower than the last one.

To deal with this, after splitting and creating your stable branch, you need to modify the bzr branch by hand. You need to change the latest debian/changelog entry's version number to correspond to the currently released stable version. This can look for example like this - current version in trunk 0.3+15.10.20150601-0ubuntu1, needs to be changed to its stable counterpart, so something like: 0.3+15.04.20150601-0ubuntu1.

Once that's done, land individual changes to stable and/or devel whenever you want using the standard procedures. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.

=== My project develops rapidly and only selective changes are good for stable ===

In this case it's best to start off by simply creating two trunks (bzr branches) from the start. Now, use standard landings to land for the devel and stable branches. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.


=== My project only targets stable, but I have risky features I want to test first without breaking user experience ===

For such a scenario it's best to use syncs. You only have one trunk basically targeting both devel and stable, but in fact the stable versions will be always slightly behind in the archives. Simply release your risky changes to the devel version (wily currently), test them for as long as you wish (in the meantime you can release new versions with fixes to devel) and once you feel it's ready for stable - request a sync silo from devel to stable.

Note that you can use syncs and dual-landings intertwined. For instance, you can release your project using dual-landing for everything that's safe, but switch to syncs once you feel like the current features are risky. Then, once you stabilize your devel branch, you can use dual-landings once again.


== Modifying your existing landing ==

Sometimes after a landing is prepared, a silo assigned and the packages built, during testing the lander can notice that some changes are missing and need to be added for the feature to be complete. This usually requires additional merges or sources included in the silo. CI Train handles this case, but requires certain actions.

Whenever a merge or source is added to the landing request after assignment, the merge and source is '''not''' automatically noticed by CI Train. A '''silo reconfigure''' is needed first.

To reconfigure, please 'Edit' your request and follow the ''Assign'' link present in your landing line. The train will detect that the assignment has the same requestid as an existing silo and reconfigure it instead of assigning a new one.

Also, an important notice for both landers and trainguards: when removing an entire package from a silo configuration, the packages are not automatically removed from the PPA. You must contact a trainguard (or ubuntu core dev) to remove the package from the silo PPA for you.

== Dealing with twin packages (-gles) ==

Some packages in our archives come with so-called package twins - those are usually packages that build for normally un-buildable architectures. Upstreams use those for packages such as qtubuntu and qtmir which normally do not build for desktop architectures. Therefore the *-gles packages have been created for the use by the emulator.

To keep both the normal and -gles packages in sync, the notion of twin packages has been introduced. If the CI Train detects that a silo has such a package in its landing, it will not allow it normally building before the -gles counterpart gets configured in the silo as well. Currently it's a rather troublesome process, but upstream developers invented a rather easy workflow to make it less problematic.

=== Recommended workflow ===

This scenario overviews what to do when you know your projects require twin package uploads.

 * When preparing the request, besides all the standard MRs, also include all the required -gles package names as Additional source packages to land (e.g. qtubuntu = qtubuntu-gles)
 * Build the silo normally
 * Once the main packages build (e.g. qtubuntu), prepare sync MRs for the -gles branches in the following way:
   * Target the -gles specific trunk (e.g. qtubuntu-gles)
   * In this branch, modify the changelog to have '''exactly the same''' version number as the main packages in the PPA (e.g. qtubuntu)
   * Also, modify the debian/watch file to point to the silo where the packages were built
   * Example MR: https://code.launchpad.net/~bzoltan/ubuntu-ui-toolkit/landing_2015-04-17-gles/+merge/256689
   * Add the newly created MRs to the silo request, remove the -gles names from Additional sources
   * Reconfigure the silo
   * Build the -gles packages in the silo


== CI Train Operation ==

=== Aborting ongoing builds ===

This is a use case that is very poorly supported by the train. The build job has two major parts, the 'preparing packages' part and the 'building' part (this is reflected in the silo status when you run the build job). During the 'preparing packages' phase, all silo state for the packages you've selected is wiped out, and recreated. If you cancel the build job during this phase, the silo will be left in an inconsistent/broken state, meaning that the packages that are in the PPA will be orphaned, and there will be no way to publish this silo as the necessary branches to merge to trunk will be missing. Don't ever cancel a build in this state, it's a bad idea, and you'll be forced to rebuild again anyway.

If the silo status says 'building', that means that the merges are finished being prepared and the train is simply polling launchpad to determine whether the packages have succeeded in building or not. In this state it is safe to cancel the build, however doing so will not actually stop the packages from continuing to build in the PPA, you're just cancelling jenkins from polling launchpad about the build status. The only real reason to want to do this is if you noticed you made a mistake and need to rebuild right away again immediately (as jenkins prevents two build jobs from running simultaneously).

'''WARNING!''' You can '''not''' abort the build job before the packages are prepared and uploaded to the PPA! The CI Train pretty much guarantees that after a build job is aborted early its state will be missing critical pieces. This will make the silo completely broken, requiring a reconfigure, reassignment or manual trainguard intervention.

=== Abandon a landing ===

In order to free a silo from jenkins and give up on the landing request, click 'Merge & Clean' on your landing request. This opens a Jenkins job where you must select '''ONLY_FREE_SILO''' and then click 'Build'. Once that job is complete, the silo will be freed, and the request will be set to 'Abandoned' for you.

If you want to abandon a request which does not have a silo assigned, just click 'Edit' and manually set the status to 'Abandoned' and then click 'Save'.
   * Please test all three
 1. Set Lander Signoff: Approved
 1. Wait for Britney to approve the ticket
 1. Wait for QA to approve the ticket
 1. Publish it yourself once everything is approved.

=== Handling Symbols ===

Because of the gcc5 C++ ABI break, symbols file maintenance has become more difficult. The compiler generates differently-mangled names for the old and new C++ ABI, so it is not possible to use a single symbols file for both 4.9 and 5.x ABIs.

Rather than maintaining two symbols files, it is better to maintain a symbols file only for vivid and to use a {{{shlibs}}} file for xenial. Nothing is lost with this approach because the exact same code is compiled for both distros. Therefore, any difference in symbols files between 4.9 and 5.x is due only to changes in the compiler (name mangling and inlining). Or, to spell it out more clearly: a symbols file for xenial cannot detect any ABI breaks that won't also be detected by the symbols file for vivid.

A workable way to keep a single source tree for both vivid and xenial is to generate the symbols file for vivid, and to generate a {{{shlibs}}} file for xenial during the build. (Also see below--it is possible to ditch the symbols files altogether.) In addition, you need to generate all the other debian files, such as {{{control}}}, {{{*.click-hook}}}, and the various {{{*.install}}} files. The hook you need is called {{{debian/bileto_pre_release_hook}}}:

Bileto will invoke [[http://bazaar.launchpad.net/~unity-team/unity-scopes-api/devel/view/head:/debian/bileto_pre_release_hook|a shell script that generates the files depending on the series]]. It's the usual hackery with {{{sed}}} and the like, but it gets the job done quite effectively. This script runs after all your merges have been merged but before the 'Releasing ...' commit is committed, so any changes you make in this script will be included in the release commit.

The script reads the current version number from a {{{VERSION}}} file in the debian directory. The version file contains something like {{{1.0.1}}}. (By definition, we are ABI compatible if the minor and major version number stay the same---your rules may be different.) The script then parses out the three version components and ''derives'' an soversion number from that. (Our {{{CMakeLists.txt}}} reads the version info from the same {{{VERSION}}} file so, when the version number changes, we change it once in that file and everything else for the build and the packaging adjusts automagically. (For example, we use this to correctly set the soname when we build a shared library, and substitute the soversion into the {{{.pc}}} file for {{{pkgconfig}}}.)

The script uses {{{sed}}} to modify the various debian files. For example, {{{control}}} is generated from {{{control.in}}}. The {{{control.in}}} file contains substitution markers that {{{sed}}} looks for. For example:
{{{
Package: libunity-scopes@UNITY_SCOPES_SOVERSION@
}}}

This then turns into {{{libunity-scopes3}}} on vivid, and {{{libunity-scopes1.0}}} on xenial and later.

Generating the other files uses similar tricks to get the correct xenial or vivid version and soname into the right spots.

For unity-scopes-api, we have ditched symbols files altogether and use [[https://sourceware.org/libabigail/|abigail]] instead. While the symbols file check is better than nothing, it does not detect ''many'' ABI breaks, such as changing the return type of a function, re-ordering struct members, changing an enumeration, making a non-virtual method virtual (plus many, many others). [[https://sourceware.org/libabigail/|abigail]] does a much better job at this and provides a far better assurance that the ABI is indeed still intact.

ABI compliance testing is implemented as normal unit test, so it runs every time we build. You can find the relevant scripts [[http://bazaar.launchpad.net/~unity-team/unity-scopes-api/devel/files/head:/test/abi-compliance/|here]]. The test requires a baseline ABI dump for each architecture. One easy way to get at these files is to make a temporary ticket that generates the ABI dump and adds it to the binary package. Once the PPA is built, you can grab the binary package for each architecture, pull out the dump file, check that into the source tree for future use, and then throw the PPA away.

ABI compliance checking is fast enough to tolerate. (For unity-scopes-api, which is large, it adds about 6 minutes to our total build time in Jenkins on Arm.) For most packages, that time will be much less. The added safety and no need to maintain symbols files any longer are well worth the wait though. Note that, when you do decide to break ABI, you will need to re-generate the new baseline ABI dumps and add them to the source tree.


== Removing Packages from your Ticket ==

 1. You must remove the MPs from the merges field, and remove the package name from the Source Package Names field.
 1. You must ask a trainguard or core-dev to remove the package from the PPA.


== Bileto Operation ==

=== Abandon a ticket ===

If your ticket is a failure and you want to give up on it, you must Abandon it. Abandoning a ticket will delete all the packages in the associated PPA, delete all the diffs that have been uploaded to swift, and set the status to 'Abandoned', which closes the ticket.

If you change your mind about an abandonment, it is possible to build abandoned tickets. They will spring back to life with a new PPA created for them, however there is a delay between when a PPA gets deleted and when it's name is free to be re-used again so you may need to wait an hour between abandoning and building the same ticket. If you're in a hurry you can create a new ticket and copy the MPs over.
Line 236: Line 163:
Under certain circumstances your landing may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are: Under certain circumstances your ticket may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:
Line 246: Line 173:
If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the '''Test plans to run''' column of the landing spreadsheet so that QA have a good basis on which to build their additional testing. Here are some tips: If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the '''Test plans''' field of the ticket so that QA have a good basis on which to build their additional testing. Here are some tips:
Line 249: Line 176:
 * Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the silo is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.  * Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the ticket is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
Line 253: Line 180:
Lastly, if you don't use the citrain tool for installing the silo then you need to mention what steps you used to install the silo, since citrain is the assumed way. Lastly, if you don't use the {{{bileto}}} tool for installing the PPA then you need to mention what steps you used to install the PPA, since {{{bileto}}} is the assumed way.
Line 261: Line 188:
= IRC Notifications =

All CI Train silo status changes are pinged in {{{#ubuntu-ci-eng}}} on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about. As a convenience, when making a new request, Bileto will insert your launchpad nick into the landers field. Unfortunately, Bileto doesn't know your IRC nick though, only launchpad nick, so if those differ, the easiest thing for you to do might be to set an IRC highlight for your launchpad nick. Then when the bot pings the status message in the channel you'll see the highlight. Either that, or just make sure to change your launchpad nick to your IRC nick every time you make a landing request.
= Branching for a Stable Series =

{{{
dch: fatal error at line 1080:
New version specified (0.2+15.04.20170331-0ubuntu1) is less than
the current version number (0.2+17.10.20160901.3-0ubuntu2)! Use -b to force.
}}}

Are you completely flummoxed by this error message? You've come to the right place!

This error message happens when you attempt to build a release for a stable series, using the devel series trunk. Don't do that! Because:

 * If Bileto let you do this, the end result would be that your devel-series trunk no longer matches the package that is actually released in devel series.

 * The next time you attempt to do a devel-series release, this stable update will also be released to devel series, which probably isn't what you intended (if it is what you intended, you should be using a dual/trio ticket and release to all series at the same time, which avoids this issue).

There are two possible solutions:

 * Just use a dual/trio series ticket, this will upload your MP for both the devel series and the stable series, keeping the devel trunk consistent with the devel series.

 * If you're really, really sure that you really just want to release your MP for the stable series only, you must branch trunk for the stable series (will assume Vivid for this example):

   1. Find your package in the stable overlay PPA for vivid.
   1. Download the DSC file and extract the debian/changelog file from it.
   1. Look in the trunk commit history to find the commit that corresponds to the release in vivid. It can be the most recent one but isn't necessarily, eg, if there have been some devel-only releases. Then do:
   {{{
   bzr branch lp:yourproject -r $REVNO vivid && bzr push -d vivid lp:~teamowner/yourproject/vivid
   }}}
   1. Confirm that the most recent commit at {{{lp:~teamowner/yourproject/vivid}}} has the commit message "Releasing $VERSION" such that $VERSION corresponds to what's in the vivid overlay PPA.
   1. Take the debian/changelog you extracted earlier and copy it over the one on your new trunk, and commit that with a message such as "Branching trunk for Vivid." This is necessary because the version numbers don't actually match, so you should see a big diff where many historical releases of your project have their version numbers changed from, eg, {{{0.2+16.10...}}} to {{{0.2+15.04...}}}
   1. Now create a new MP proposing your original branch that caused this whole problem, and target it at this new trunk you've just created, and build that.

Once you've done all this, be careful not to do another dual/trio ticket, as that will clobber everything you've done here, releasing all your diverged devel releases back to the stable release.

= Reverting a Landing =

Did somebody discover a regression '''AFTER''' a ticket was fully QA'd, published, and merged? Never fear, the Revert button is here! Reverting a landing will re-upload the second-most-recent upload of a source package, with a new version number stamped on it, so the upload will be accepted over the newest one with the regression. Here's what to do:

 1. Start a new ticket. The original ticket with the MPs on it cannot be used for the revert.
 1. On the new ticket, manually type in the names of the source packages to the sources field.
 1. Make sure your series selection matches that of the original ticket. If the original landing was a trio ticket, the revert ticket should be a trio ticket so that it's reverted in all series equally for consistency. But if you had a vivid-only phone landing, be sure to make the revert a vivid-only ticket. Reverting in a trio ticket does not mean "reupload the second-most-recent of each series", it means "take the second-most-recent from yakkety and upload it to yakkety, xenial, and vivid". This is only what you want if the landing you're reverting was also a trio.
 1. Click the revert button. This behaves much like the Build button and will show you a log of the packages built.
 1. QA the ticket as per usual
 1. If QA confirms the regression is fixed, publish as usual!

In this process, nothing will be merged to the project trunk (hence no MPs), this allows upstream developers to work on fixing their trunks uninterrupted while the revert goes directly to users. While you technically can revert a revert, this is not recommended; instead the upstream landers should do a fresh landing with their fix for the regression.

= Interpreting Ticket Statuses =

Bileto monitors PPA statuses at a per-package level, which can lead to some detailed status messages. Fortunately, in many common cases, irrelevant details are omitted for clarity. Here are some examples:

 * '''Successfully built''' or '''Failed to build'''
    When there is no package list, it means that all packages have this same status. Error states such as '''Failed to build''' or '''Dependency wait''' will appear if that status appears on any arch, but '''Successfully built''' will appear only if it succeeded on all arches.

 * '''Failed to build (xenial/foo-package). Successfully built (xenial/bar-package, xenial/grill-package).'''
    When there are multiple sentences with series/packages in brackets, the message applies to the packages named in the brackets. So in this example foo-package failed to build in xenial while the other two succeeded.

 * '''Proposed pocket (yakkety/foo-package). Release pocket (vivid/foo-package, xenial/foo-package).'''
    This means that the ticket has been published but has not yet made it all the way to Ubuntu archive yet. When publishing a trio ticket, the vivid and xenial packages are copied immediately to the overlay PPA, so it will indicate "Release pocket" (the fact that it's the release pocket of the overlay PPA and not the release pocket of the vivid archive is implied by the fact that it's a trio landing. trio landings '''never''' target vivid archive). Once the yakkety package finishes migrating through Proposed and into Release, the ticket will be auto-merged, meaning all your merges will be merged to trunk automatically.

 * '''Release pocket (vivid/foo-package, xenial/foo-package). Successfully built (yakkety/foo-package).'''
    This means that the vivid and xenial packages have been published but the yakkety package has not been published, most likely due to a bug. "Successfully built" implies that the package is not in any upload queue or proposed pocket or anything, because if it was, the status would say so. Please notify trainguards immediately if you see this!

 * '''Needs rebuild due to new commits (yakkety/foo-package). Successfully built (vivid/foo-package, xenial/foo-package).'''
    That's weird! Why doesn't the vivid/xenial ones reflect that it also needs to be rebuilt to match the yakkety one? Well the class that implements the secondary series does not have any knowledge of MPs or commits, so when unbuilt commits are discovered in the yakkety MPs, the vivid/xenial bits have no way to also check that. So this is the expected status when a trio ticket has new commits. Rest assured that when you rebuild the package, all series will be rebuilt, you cannot build one without the other so they can never get out of sync.

== Common Status Messages ==

=== Ready to build ===
    No package is in the PPA, mostly likely because the PPA is newly created, but also possibly because the packages have been deleted.

=== PPA/bzr version mismatch ===
    The version number in the PPA does not match the version number in [[https://code.launchpad.net/~ci-train-bot|the bzr branch]]. Can also be '''PPA/git version mismatch''' if you are using git. A number of different situations can trigger this:

 * The upload to the PPA has failed. Launchpad batch-processes PPA uploads so even if the log indicates a successful upload, launchpad can still reject it asynchronously after the fact. Generally the conditions under which launchpad would reject an upload are predictable so if this happens it is almost always a bug in Bileto that made it generate a bad version number for the package that launchpad doesn't like (eg, if it's lower than the existing one in the PPA).
 * The upload to the PPA is just really slow. At times launchpad can be under heavy load and your package just might take a while to appear in the PPA.
 * The push to the bzr/git branch failed. This has never happened as far as I know, but if it did happen it would cause this error message.

In short, this is a little racy. If you see this message, wait 20 minutes. If it doesn't go away after 20 minutes then some investigation is required. Check that the PPA really contains your package. Generally this can be fixed by simply doing another build, but please do wait 20 minutes and investigate the issue instead of just blindly clicking rebuild every time you see this.

=== Bad merges ===
    This happens if you have merges that are either Rejected or Superceded. Click the "Status" link to see the full log, which indicates which merges need attention.

=== Needs rebuild due to new commits ===
    Bileto has detected new commits on either the source or target branches of your merges. Click the "Status" link to see the list of where new commits have been detected. Note that this message won't appear if the only new commits are Launchpad Translation auto-commits.

=== Needs rebuild due to burned version number ===
    This happens when the same source package is built in two different tickets on the same day, such that they both have the same version number, and one of those tickets has been published. The other ticket sees that the destination archive has a matching version number, but the package has differing contents, so the current version number is now "burned" (eg, already used up). Usually this message will shortly be followed by the '''new commits''' message after the other ticket merges, so take this as an early warning that you'll soon need to rebuild (but not yet!).

=== Release/Updates/Proposed pocket ===
    This ticket has been published and is on it's way to the archive! The ticket will automatically merge once all packages are found to be in either the Release or Updates pocket. It is safe to manually finalize when packages are in the Proposed pocket, however this is discouraged because you most likely want to do this when something is stuck in Proposed, and finalize is just ignoring the problem. Much better to fix the problem and let the package migrate normally.

=== NEW/UNAPPROVED queue ===
    If your package has never been released in ubuntu before, it will find itself in the NEW queue, and if the archive is frozen (or you're doing an SRU), it will be in the UNAPPROVED queue. This is not something that will migrate on it's own and does require manual intervention to prod along, so be sure to ask for help if you see this state. '''WARNING:''' it is not safe to finalize the ticket in this state! If you merge, the packages will be lost and your release will never make it to the archive.

=== DONE queue ===
    Apologies for this cryptic message, this happens when the package has been uploaded to the {{{-proposed}}} archive but the ticket destination is set to a PPA instead of the ubuntu archive. "DONE" means that the upload to the ubuntu archive was successful, but it doesn't give the expected "Release pocket" or "Proposed pocket" message because it's looking at the overlay PPA instead, and the package isn't there. This situation can be resolved by copying the package from your PPA into the overlay PPA.

=== Destination version is missing from changelog ===

When you see this message, click through to the bileto log and it will show you the diff between what is in the destination archive and what is on your project trunk. There are three scenarios that cause this message to appear:

 1. Your ticket was already published but you rebuilt the ticket since then. Bileto is "stateless" and so isn't able to recognize when a package in the destination archive is a previous build from the same ticket. If this is the case, you must join '''#ubuntu-release''' on freenode and ask an archive admin to delete the offending package from -proposed [as requested by slangasek]. Once the package is deleted this message will go away and you'll be able to submit the ticket for qa again.
 1. Some other ticket containing the same source package was published but hasn't yet merged to trunk. In this case no action is required yet, simply wait for the other ticket to merge to trunk, at which point your status will update to 'Needs rebuild due to new commits' and then you should rebuild your ticket.
 1. A manual upload has been uploaded to the destination archive, not using bileto. This can happen for a variety of reasons, typically an archive admin has uploaded a trivial fix for some packaging issue, but didn't use bileto to do it. In this case you must inspect the status log which will show you the exact diff that is missing from your trunk, you'll need to commit that to your trunk manually and then rebuild in order to continue. If you ignore this message in this case, it means that publishing your ticket will result in reverting the work of the archive admin. Don't do that!

The first two scenarios will be characterized by version numbers that end with '''-0ubuntu1''' while scenario three will typically have some other version number, such as '''-0ubuntu2''' or '''-0ubuntu1.1'''. Unfortunately there is no way to automatically distinguish between the first two scenarios, you'll need to Just Know whether you rebuilt an already-published ticket or not.


=== Failed to build ===
    Your package was uploaded to the PPA but did not build successfully! Click the "Status" link to see the full list of what packages/arches failed and links to the failure logs.

=== Diff missing ===
    Most likely you uploaded manual sources to your PPA, although this can appear on MP builds if there was an error generating diffs, which can happen from time to time. Please click the '''Diff''' button on your ticket to resolve this.

=== Successfully built ===
    Absolutely no problems could be (automatically) found with your package build. Congrats! Test it out and publish it!


= Notifications =

== IRC ==

All ticket status changes are pinged in {{{#ubuntu-ci-eng}}} on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about.

== Email ==

Important behavioural changes in Bileto are occasionally announced via [[https://launchpad.net/~bileto-announce|the mailing list]]. Please join the team & subscribe to the mailing list so you stay up to date with important announcements.
Line 267: Line 319:
By default, the train will generate a debian/changelog entry for you, based on the text you enter into the "Commit Message" field of the merge proposals in the silo. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series '''UNRELEASED''' so that the train can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, the train will make a new empty changelog message which isn't what you want.

= Common Errors =

== [source_package_name] [version] is missing from the changelog, which has up to [lesser_version]. Please sync destination version back to trunk. ==

This happens when an ubuntu core developer makes a manual upload of your package to ubuntu without using ci train. 99% of the time this would be no-change rebuilds during library transitions, or other trivial fixes for packaging issues. Before you can build the silo, it is necessary to sync the manual distro upload back to your trunk, so that they match. The first thing to do is check the launchpad page for your source package (for example [[https://launchpad.net/ubuntu/+source/unity8|here's the page for unity8]]. On that page look for a version number that ends in "0ubuntu2" or similar (the train only ever generates versions ending in "0ubuntu1"). Click through that page to get to the "available diffs" for that version, download that diff, and then commit that directly to your trunk. '''Do not''' make a merge proposal with this diff and include it in the silo, that does not work and is a backwards approach. You want to copy what's in distro into your trunk, silos are for getting branches into distro.
By default, a debian/changelog entry will be automatically generated for you, based on the text you enter into the "Commit Message" field of the merge proposals in the ticket. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series '''UNRELEASED''' so that Bileto can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, your entry will be ignored and a new one will be created on top of it which is not what you want.

= Command-Line Tool =

Installing a PPA onto an Ubuntu Touch device is a bit of a chore because you have to set the device writable, pin the ticket PPA higher than the overlay PPA, etc. Fortunately there is a tool to automate this.

{{{
$ sudo add-apt-repository ppa:phablet-team/ubuntu/tools
$ sudo apt-get install phablet-tools-bileto
$ bileto help
}}}

= How to Coordinate the Publishing of Multiple Large Landings =

Sometimes, there will be a few large tickets wanting to publish around the same time, eg mir and unity8. When this happens, best practices is to do an image build in between publishings, because this makes it easier to bisect where regressions came from. It's a little bit tricky however, because certain tools report information differently and there's a race condition that you can fail if you're not careful. Best to follow these steps:

1. Publish one of the tickets.

2. Wait for it to migrate. Eventually, queuebot will ping that the request is '''Release pocket'''.

3. It's not yet safe to kick the image build, however. You must now use {{{rmadison}}} to confirm that the packages are truly out of the -proposed pocket and in the release pocket.

4. Once rmadison confirms publication has completed, now kick the image build.

5. Wait until the image build has completed, to ensure the build succeeds and doesn't need to be retried or fixed.

6. Now publish the second ticket.

= Renaming a Source Package =

Are you trying to rename a source package and getting weird errors? Don't do that!

Just kidding, but here's what's going on. Bileto inspects the *trunk* (merge target) in order to discover what source package name to use for building. It is therefore not possible to propose merges that change the source package name of your project. You'll need to push a direct trunk commit which changes the source package name, and then put other changes into an MP which can then build packages with the new name. You'll need to ensure the first line of your debian/changelog '''on trunk''' contains your desired source package name.

Landing changes through Bileto

Hello lander! If you got here, you're probably looking for information about how to use Bileto (formerly known as the "CI Train") or what to do to get your project landed to Ubuntu. For general information, make your way to the Landing your change to Ubuntu section below. If you already know the basics and just want to know which of the tools and processes to use for your specific landing, the Which landing method to use? section should help you.

Enjoy.

IMPORTANT! With the sunset of the Ubuntu Touch project, the main use-case of Bileto as a MP-based workflow became much less common (and useful), and now its purpose shifted more towards a manual package-staging helper with ephemeral PPAs and autopkgtests runners. For more information about this particular use case, see the Manual upload section of the documentation.

Landing your change to Ubuntu

There are two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is maintained by Canonical.

Whenever you want to request a release of your component to the archive:

  • If this is your first time running a landing, ask the trainguards in #ubuntu-ci-eng to give you the required permissions for operating Bileto.

  • If this is the first time this source package has ever been released with Bileto, refer to the Inline Packaging guide to meet the minimum packaging standards.

  • Open the tickets page.

  • Sign in with SSO, click Create New Request, and you'll be presented with a form to fill out.

    • Description of landing: write here an overview of what your landing is about - is it fixing bugs? Adding some features? Try being informative without being over-verbose.
    • Test plans to run: this mandatory field documents how you, the lander, intend to test the given component(s). Generally you would put a URL to a document describing steps to test that your component functions correctly, as this information will generally not change much from one ticket to the next. See #QA for more details.

    • Merge proposals to land: here you list all the links to the merge proposals you want to land in this landing. Please note, this must be a newline-separated list of MP links, not branch links. More on this later.

    • Manual Download URLs: This optional field contains click packages / device tarball updates that don't require a PPA for testing, but instead must be downloaded manually by the tester.
    • Landers: List of people who care about this landing. These names will be pinged on IRC when ticket status changes, so use IRC nicks if possible.
    • Source Package Names: You can leave this blank and the correct value will auto-fill based on the contents of the PPA.
    • Target distribution: select which distribution you want to land into. Currently only ubuntu is supported but if ubuntu-rtm ever makes a comeback you'd select that here.
    • Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select yakkety+xenial+vivid.

    • QA signoff: Values "N/A", "Required", or "Ready" will be automatically set as appropriate for your landing. The rule is: any stable-phone targets like vivid or 'yakkety+xenial+vivid' landings by default require it. See #QA for more information.

  • Once the request is filled out and you've clicked 'Create New Request', you can assign a PPA by clicking 'Assign' from the menu at the bottom of your new ticket.
  • Once your PPA is assigned, the menu will show some new buttons. Use the 'Build' button to start building packages from your merge proposals.
    • If you have manual packages not built from merges, you need a trainguard or Ubuntu core-dev to upload those to your PPA.
  • After the packages finish building, it is now your responsibility to test the packages from the PPA on a real device. More information about this is in the #QA section. Remember: you are responsible not only to test that the fix works, but also to test that your change didn't break anything else. Use the Test Plan.

  • If your packages pass your testing criteria, mark the ticket as tested:
    • Get all your MP's top-approved. From Bileto's perspective, you are free to approve your own merges, however in some teams this is frowned upon, so you must get your branches reviewed by your teammates.
    • Set 'Lander Signoff' to 'Approved'.
  • Britney will now automatically inspect your PPA and perform various installability checks and autopkgtests. Once britney approves your ticket, it will automatically be marked ready for QA, if appropriate.
  • If your ticket says "QA Signoff: Ready" you must wait for a QA person to approve your ticket.
  • Once your ticket has been approved by both Britney and QA (or if QA is N/A), you can publish your own ticket. Landers are permitted to publish their own PPAs so long as there are no packaging changes, and no manual sources in the PPA. In the event that you do have either packaging changes or manual sources, it is your responsibility to find a core dev to publish the PPA for you.

This is the standard process. When you prepare a list of MPs to land, the infrastructure will merge all of them together and build a source package using dpkg-buildpackage. This source package is then uploaded to the PPA that has been assigned to your ticket. The debian/changelog is automatically generated for you using the Commit Message field of each MP. Every ticket creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time. After all changes land into the archive, these merges are automatically merged into trunk.

So, the general steps in the landing process from the lander's perspective are:

  1. Fill in a landing request
  2. Assign it yourself
  3. Build packages in PPA
  4. Test packages from the PPA on real hardware
  5. Mark ticket as Lander Signoff: Approved
  6. Wait for Britney to mark the ticket Approved.
  7. (in some cases) Wait for QA to mark the ticket Approved
  8. Publish the ticket yourself
  9. (in some cases) Find a core dev to ACK packaging changes and publish the ticket for you

Manual Source Uploads

Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released through Bileto: either because of the QA sign-off requirement for touch, or simply because they're needed by a ticket, mixed-in with normal MP-based packages.

  1. Create a new ticket. In this case most fields are optional and do not have to be filled up
  2. Press 'Build' on the empty ticket - this creates the PPA assigned to the ticket
  3. Prepare the source packages locally
  4. Get the package uploaded to the PPA:
    • If you're a core-dev: simply use dput to upload the package to the PPA
    • In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you (one option is to dput to your personal PPA and then core-dev or trainguard can copy the package from there).
  5. Proceed normally with building & testing

Manual Source Uploads for the Security Pocket

Sometimes performing an upload that is meant to go to the respective -security pocket is required. Bileto PPAs try to be as close to the archive as possible, so that one can freely bin-copy packages directly to the Ubuntu archive. By default, Bileto PPAs are configured to build with -proposed enabled (like any packages built in the Proposed pocket), but the settings of package dependencies can be manually adjusted per-PPA (Bileto will not touch these settings on its own):

  1. Create a ticket for manual uploads, as usual
    • Be sure to select the 'Target Series' to something that's not the overlay PPA
  2. Once the PPA is created, go to the PPA's main view and then to the 'Edit PPA dependencies' settings
  3. Switch the Ubuntu dependencies to 'Security' and make sure no other PPA dependencies are present
  4. Press Save and proceed normally with building & testing

Landing your change to the Stable Overlay PPA

The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.

How this works is: all current devel builds (and soon all stable builds) are built from vivid + the stable overlay PPA. This means we get all the goodness from vivid-updates and vivid-security, while we keep the dynamics of publishing touch packages for our purposes without having to go through the SRU process.

For a trio ticket such as yakkety+xenial+vivid, the destination field is ignored. The the first series never goes to the stable overlay PPA, and the others always do.

Trio landing for stable and devel

The current development series is yakkety. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep yakkety and xenial in sync, Bileto allows tickets to be built for all three together automatically.

NOTICE! Remember that all trio landings are required to go through the QA sign-off process because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your ticket won't be released as a trio landing. What the trainguards can do for you is release the yakkety version and then request you to sync your work back to the vivid stable-overlay once the gates are open.

  1. Fill in a standard ticket
    • The merges need to point to your trunk (devel) branch, as the merges will be built against yakkety and then copied to xenial and vivid
  2. In the Target Series drop-down select "yakkety+xenial+vivid"
  3. Assign your own PPA
  4. Build the ticket
    • The build job will create 3 source packages - one for yakkety, one for xenial, and one for vivid - and upload them to the PPA
    • The primary target is always yakkety
    • Your merges will be built for yakkety (with the changelog entries generated for yakkety), but then the sources will be duplicated (with re-written versions) for xenial and vivid
  5. Test the generated packages on real hardware
    • Please test all three
  6. Set Lander Signoff: Approved
  7. Wait for Britney to approve the ticket
  8. Wait for QA to approve the ticket
  9. Publish it yourself once everything is approved.

Handling Symbols

Because of the gcc5 C++ ABI break, symbols file maintenance has become more difficult. The compiler generates differently-mangled names for the old and new C++ ABI, so it is not possible to use a single symbols file for both 4.9 and 5.x ABIs.

Rather than maintaining two symbols files, it is better to maintain a symbols file only for vivid and to use a shlibs file for xenial. Nothing is lost with this approach because the exact same code is compiled for both distros. Therefore, any difference in symbols files between 4.9 and 5.x is due only to changes in the compiler (name mangling and inlining). Or, to spell it out more clearly: a symbols file for xenial cannot detect any ABI breaks that won't also be detected by the symbols file for vivid.

A workable way to keep a single source tree for both vivid and xenial is to generate the symbols file for vivid, and to generate a shlibs file for xenial during the build. (Also see below--it is possible to ditch the symbols files altogether.) In addition, you need to generate all the other debian files, such as control, *.click-hook, and the various *.install files. The hook you need is called debian/bileto_pre_release_hook:

Bileto will invoke a shell script that generates the files depending on the series. It's the usual hackery with sed and the like, but it gets the job done quite effectively. This script runs after all your merges have been merged but before the 'Releasing ...' commit is committed, so any changes you make in this script will be included in the release commit.

The script reads the current version number from a VERSION file in the debian directory. The version file contains something like 1.0.1. (By definition, we are ABI compatible if the minor and major version number stay the same---your rules may be different.) The script then parses out the three version components and derives an soversion number from that. (Our CMakeLists.txt reads the version info from the same VERSION file so, when the version number changes, we change it once in that file and everything else for the build and the packaging adjusts automagically. (For example, we use this to correctly set the soname when we build a shared library, and substitute the soversion into the .pc file for pkgconfig.)

The script uses sed to modify the various debian files. For example, control is generated from control.in. The control.in file contains substitution markers that sed looks for. For example:

Package: libunity-scopes@UNITY_SCOPES_SOVERSION@

This then turns into libunity-scopes3 on vivid, and libunity-scopes1.0 on xenial and later.

Generating the other files uses similar tricks to get the correct xenial or vivid version and soname into the right spots.

For unity-scopes-api, we have ditched symbols files altogether and use abigail instead. While the symbols file check is better than nothing, it does not detect many ABI breaks, such as changing the return type of a function, re-ordering struct members, changing an enumeration, making a non-virtual method virtual (plus many, many others). abigail does a much better job at this and provides a far better assurance that the ABI is indeed still intact.

ABI compliance testing is implemented as normal unit test, so it runs every time we build. You can find the relevant scripts here. The test requires a baseline ABI dump for each architecture. One easy way to get at these files is to make a temporary ticket that generates the ABI dump and adds it to the binary package. Once the PPA is built, you can grab the binary package for each architecture, pull out the dump file, check that into the source tree for future use, and then throw the PPA away.

ABI compliance checking is fast enough to tolerate. (For unity-scopes-api, which is large, it adds about 6 minutes to our total build time in Jenkins on Arm.) For most packages, that time will be much less. The added safety and no need to maintain symbols files any longer are well worth the wait though. Note that, when you do decide to break ABI, you will need to re-generate the new baseline ABI dumps and add them to the source tree.

Removing Packages from your Ticket

  1. You must remove the MPs from the merges field, and remove the package name from the Source Package Names field.
  2. You must ask a trainguard or core-dev to remove the package from the PPA.

Bileto Operation

Abandon a ticket

If your ticket is a failure and you want to give up on it, you must Abandon it. Abandoning a ticket will delete all the packages in the associated PPA, delete all the diffs that have been uploaded to swift, and set the status to 'Abandoned', which closes the ticket.

If you change your mind about an abandonment, it is possible to build abandoned tickets. They will spring back to life with a new PPA created for them, however there is a delay between when a PPA gets deleted and when it's name is free to be re-used again so you may need to wait an hour between abandoning and building the same ticket. If you're in a hurry you can create a new ticket and copy the MPs over.

QA Sign-off Needed

Under certain circumstances your ticket may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:

  • For all landings to the stable images (vivid + overlay PPA)

There is an exception to this rule if your landing contains an isolated bug fix. Right now there is not a strict definition of this term, but as guidance an isolated bug fix landing is:

  • Only one or a couple of bug fixes
  • A small change and not impacting a large number of sub-components of that package
  • A change in a package that does not have a lot of reverse dependencies, since this makes it much harder to assess the impact of the landing.

If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the Test plans field of the ticket so that QA have a good basis on which to build their additional testing. Here are some tips:

  • Never just test that the bug(s) alone are fixed - remember the potential for regressions
  • Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the ticket is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
  • Take into consideration the dependencies and reverse dependencies involved - ask yourself, who could I break? Ideally your package's test plan should have some guidance on this (for example see the Dependents/Clients section media-hub's test plan

  • If you only provide a link to the test plan, then bear in mind that QA will assume you've run all the tests in that plan and they have passed. If tests then fail for a reason you haven't mentioned then it will cause confusion and delay the landing, so make sure to record all cases where you have not run a test or a test has failed (and why).

Lastly, if you don't use the bileto tool for installing the PPA then you need to mention what steps you used to install the PPA, since bileto is the assumed way.

Isolated bugfixes for stable - confirmation

Whenever a landing is targeting a stable branch and is marked by the lander as not requiring QA sign-off because of an isolated bugfix, the Landing Team tries to verify if this indeed falls into this category. The Landing Team either by themselves or with the help of the QA team look at the landing description, looking for any rationale for this exception. Merge requests or diffs are also examined in case that's needed.

The best practice is to directly ping the QA team with a question if your change can go through without a full QA sign-off. The QA Team is here to ensure that things landing into stable are as safe as possible.

Branching for a Stable Series

dch: fatal error at line 1080:
New version specified (0.2+15.04.20170331-0ubuntu1) is less than
the current version number (0.2+17.10.20160901.3-0ubuntu2)!  Use -b to force.

Are you completely flummoxed by this error message? You've come to the right place!

This error message happens when you attempt to build a release for a stable series, using the devel series trunk. Don't do that! Because:

  • If Bileto let you do this, the end result would be that your devel-series trunk no longer matches the package that is actually released in devel series.
  • The next time you attempt to do a devel-series release, this stable update will also be released to devel series, which probably isn't what you intended (if it is what you intended, you should be using a dual/trio ticket and release to all series at the same time, which avoids this issue).

There are two possible solutions:

  • Just use a dual/trio series ticket, this will upload your MP for both the devel series and the stable series, keeping the devel trunk consistent with the devel series.
  • If you're really, really sure that you really just want to release your MP for the stable series only, you must branch trunk for the stable series (will assume Vivid for this example):
    1. Find your package in the stable overlay PPA for vivid.
    2. Download the DSC file and extract the debian/changelog file from it.
    3. Look in the trunk commit history to find the commit that corresponds to the release in vivid. It can be the most recent one but isn't necessarily, eg, if there have been some devel-only releases. Then do:
         bzr branch lp:yourproject -r $REVNO vivid && bzr push -d vivid lp:~teamowner/yourproject/vivid
    4. Confirm that the most recent commit at lp:~teamowner/yourproject/vivid has the commit message "Releasing $VERSION" such that $VERSION corresponds to what's in the vivid overlay PPA.

    5. Take the debian/changelog you extracted earlier and copy it over the one on your new trunk, and commit that with a message such as "Branching trunk for Vivid." This is necessary because the version numbers don't actually match, so you should see a big diff where many historical releases of your project have their version numbers changed from, eg, 0.2+16.10... to 0.2+15.04...

    6. Now create a new MP proposing your original branch that caused this whole problem, and target it at this new trunk you've just created, and build that.

Once you've done all this, be careful not to do another dual/trio ticket, as that will clobber everything you've done here, releasing all your diverged devel releases back to the stable release.

Reverting a Landing

Did somebody discover a regression AFTER a ticket was fully QA'd, published, and merged? Never fear, the Revert button is here! Reverting a landing will re-upload the second-most-recent upload of a source package, with a new version number stamped on it, so the upload will be accepted over the newest one with the regression. Here's what to do:

  1. Start a new ticket. The original ticket with the MPs on it cannot be used for the revert.
  2. On the new ticket, manually type in the names of the source packages to the sources field.
  3. Make sure your series selection matches that of the original ticket. If the original landing was a trio ticket, the revert ticket should be a trio ticket so that it's reverted in all series equally for consistency. But if you had a vivid-only phone landing, be sure to make the revert a vivid-only ticket. Reverting in a trio ticket does not mean "reupload the second-most-recent of each series", it means "take the second-most-recent from yakkety and upload it to yakkety, xenial, and vivid". This is only what you want if the landing you're reverting was also a trio.
  4. Click the revert button. This behaves much like the Build button and will show you a log of the packages built.
  5. QA the ticket as per usual
  6. If QA confirms the regression is fixed, publish as usual!

In this process, nothing will be merged to the project trunk (hence no MPs), this allows upstream developers to work on fixing their trunks uninterrupted while the revert goes directly to users. While you technically can revert a revert, this is not recommended; instead the upstream landers should do a fresh landing with their fix for the regression.

Interpreting Ticket Statuses

Bileto monitors PPA statuses at a per-package level, which can lead to some detailed status messages. Fortunately, in many common cases, irrelevant details are omitted for clarity. Here are some examples:

  • Successfully built or Failed to build

    • When there is no package list, it means that all packages have this same status. Error states such as Failed to build or Dependency wait will appear if that status appears on any arch, but Successfully built will appear only if it succeeded on all arches.

  • Failed to build (xenial/foo-package). Successfully built (xenial/bar-package, xenial/grill-package).

    • When there are multiple sentences with series/packages in brackets, the message applies to the packages named in the brackets. So in this example foo-package failed to build in xenial while the other two succeeded.
  • Proposed pocket (yakkety/foo-package). Release pocket (vivid/foo-package, xenial/foo-package).

    • This means that the ticket has been published but has not yet made it all the way to Ubuntu archive yet. When publishing a trio ticket, the vivid and xenial packages are copied immediately to the overlay PPA, so it will indicate "Release pocket" (the fact that it's the release pocket of the overlay PPA and not the release pocket of the vivid archive is implied by the fact that it's a trio landing. trio landings never target vivid archive). Once the yakkety package finishes migrating through Proposed and into Release, the ticket will be auto-merged, meaning all your merges will be merged to trunk automatically.

  • Release pocket (vivid/foo-package, xenial/foo-package). Successfully built (yakkety/foo-package).

    • This means that the vivid and xenial packages have been published but the yakkety package has not been published, most likely due to a bug. "Successfully built" implies that the package is not in any upload queue or proposed pocket or anything, because if it was, the status would say so. Please notify trainguards immediately if you see this!
  • Needs rebuild due to new commits (yakkety/foo-package). Successfully built (vivid/foo-package, xenial/foo-package).

    • That's weird! Why doesn't the vivid/xenial ones reflect that it also needs to be rebuilt to match the yakkety one? Well the class that implements the secondary series does not have any knowledge of MPs or commits, so when unbuilt commits are discovered in the yakkety MPs, the vivid/xenial bits have no way to also check that. So this is the expected status when a trio ticket has new commits. Rest assured that when you rebuild the package, all series will be rebuilt, you cannot build one without the other so they can never get out of sync.

Common Status Messages

Ready to build

  • No package is in the PPA, mostly likely because the PPA is newly created, but also possibly because the packages have been deleted.

PPA/bzr version mismatch

  • The version number in the PPA does not match the version number in the bzr branch. Can also be PPA/git version mismatch if you are using git. A number of different situations can trigger this:

  • The upload to the PPA has failed. Launchpad batch-processes PPA uploads so even if the log indicates a successful upload, launchpad can still reject it asynchronously after the fact. Generally the conditions under which launchpad would reject an upload are predictable so if this happens it is almost always a bug in Bileto that made it generate a bad version number for the package that launchpad doesn't like (eg, if it's lower than the existing one in the PPA).
  • The upload to the PPA is just really slow. At times launchpad can be under heavy load and your package just might take a while to appear in the PPA.
  • The push to the bzr/git branch failed. This has never happened as far as I know, but if it did happen it would cause this error message.

In short, this is a little racy. If you see this message, wait 20 minutes. If it doesn't go away after 20 minutes then some investigation is required. Check that the PPA really contains your package. Generally this can be fixed by simply doing another build, but please do wait 20 minutes and investigate the issue instead of just blindly clicking rebuild every time you see this.

Bad merges

  • This happens if you have merges that are either Rejected or Superceded. Click the "Status" link to see the full log, which indicates which merges need attention.

Needs rebuild due to new commits

  • Bileto has detected new commits on either the source or target branches of your merges. Click the "Status" link to see the list of where new commits have been detected. Note that this message won't appear if the only new commits are Launchpad Translation auto-commits.

Needs rebuild due to burned version number

  • This happens when the same source package is built in two different tickets on the same day, such that they both have the same version number, and one of those tickets has been published. The other ticket sees that the destination archive has a matching version number, but the package has differing contents, so the current version number is now "burned" (eg, already used up). Usually this message will shortly be followed by the new commits message after the other ticket merges, so take this as an early warning that you'll soon need to rebuild (but not yet!).

Release/Updates/Proposed pocket

  • This ticket has been published and is on it's way to the archive! The ticket will automatically merge once all packages are found to be in either the Release or Updates pocket. It is safe to manually finalize when packages are in the Proposed pocket, however this is discouraged because you most likely want to do this when something is stuck in Proposed, and finalize is just ignoring the problem. Much better to fix the problem and let the package migrate normally.

NEW/UNAPPROVED queue

  • If your package has never been released in ubuntu before, it will find itself in the NEW queue, and if the archive is frozen (or you're doing an SRU), it will be in the UNAPPROVED queue. This is not something that will migrate on it's own and does require manual intervention to prod along, so be sure to ask for help if you see this state. WARNING: it is not safe to finalize the ticket in this state! If you merge, the packages will be lost and your release will never make it to the archive.

DONE queue

  • Apologies for this cryptic message, this happens when the package has been uploaded to the -proposed archive but the ticket destination is set to a PPA instead of the ubuntu archive. "DONE" means that the upload to the ubuntu archive was successful, but it doesn't give the expected "Release pocket" or "Proposed pocket" message because it's looking at the overlay PPA instead, and the package isn't there. This situation can be resolved by copying the package from your PPA into the overlay PPA.

Destination version is missing from changelog

When you see this message, click through to the bileto log and it will show you the diff between what is in the destination archive and what is on your project trunk. There are three scenarios that cause this message to appear:

  1. Your ticket was already published but you rebuilt the ticket since then. Bileto is "stateless" and so isn't able to recognize when a package in the destination archive is a previous build from the same ticket. If this is the case, you must join #ubuntu-release on freenode and ask an archive admin to delete the offending package from -proposed [as requested by slangasek]. Once the package is deleted this message will go away and you'll be able to submit the ticket for qa again.

  2. Some other ticket containing the same source package was published but hasn't yet merged to trunk. In this case no action is required yet, simply wait for the other ticket to merge to trunk, at which point your status will update to 'Needs rebuild due to new commits' and then you should rebuild your ticket.
  3. A manual upload has been uploaded to the destination archive, not using bileto. This can happen for a variety of reasons, typically an archive admin has uploaded a trivial fix for some packaging issue, but didn't use bileto to do it. In this case you must inspect the status log which will show you the exact diff that is missing from your trunk, you'll need to commit that to your trunk manually and then rebuild in order to continue. If you ignore this message in this case, it means that publishing your ticket will result in reverting the work of the archive admin. Don't do that!

The first two scenarios will be characterized by version numbers that end with -0ubuntu1 while scenario three will typically have some other version number, such as -0ubuntu2 or -0ubuntu1.1. Unfortunately there is no way to automatically distinguish between the first two scenarios, you'll need to Just Know whether you rebuilt an already-published ticket or not.

Failed to build

  • Your package was uploaded to the PPA but did not build successfully! Click the "Status" link to see the full list of what packages/arches failed and links to the failure logs.

Diff missing

  • Most likely you uploaded manual sources to your PPA, although this can appear on MP builds if there was an error generating diffs, which can happen from time to time. Please click the Diff button on your ticket to resolve this.

Successfully built

  • Absolutely no problems could be (automatically) found with your package build. Congrats! Test it out and publish it!

Notifications

IRC

All ticket status changes are pinged in #ubuntu-ci-eng on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about.

Email

Important behavioural changes in Bileto are occasionally announced via the mailing list. Please join the team & subscribe to the mailing list so you stay up to date with important announcements.

Generated debian/changelog

By default, a debian/changelog entry will be automatically generated for you, based on the text you enter into the "Commit Message" field of the merge proposals in the ticket. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series UNRELEASED so that Bileto can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, your entry will be ignored and a new one will be created on top of it which is not what you want.

Command-Line Tool

Installing a PPA onto an Ubuntu Touch device is a bit of a chore because you have to set the device writable, pin the ticket PPA higher than the overlay PPA, etc. Fortunately there is a tool to automate this.

$ sudo add-apt-repository ppa:phablet-team/ubuntu/tools
$ sudo apt-get install phablet-tools-bileto
$ bileto help

How to Coordinate the Publishing of Multiple Large Landings

Sometimes, there will be a few large tickets wanting to publish around the same time, eg mir and unity8. When this happens, best practices is to do an image build in between publishings, because this makes it easier to bisect where regressions came from. It's a little bit tricky however, because certain tools report information differently and there's a race condition that you can fail if you're not careful. Best to follow these steps:

1. Publish one of the tickets.

2. Wait for it to migrate. Eventually, queuebot will ping that the request is Release pocket.

3. It's not yet safe to kick the image build, however. You must now use rmadison to confirm that the packages are truly out of the -proposed pocket and in the release pocket.

4. Once rmadison confirms publication has completed, now kick the image build.

5. Wait until the image build has completed, to ensure the build succeeds and doesn't need to be retried or fixed.

6. Now publish the second ticket.

Renaming a Source Package

Are you trying to rename a source package and getting weird errors? Don't do that!

Just kidding, but here's what's going on. Bileto inspects the *trunk* (merge target) in order to discover what source package name to use for building. It is therefore not possible to propose merges that change the source package name of your project. You'll need to push a direct trunk commit which changes the source package name, and then put other changes into an MP which can then build packages with the new name. You'll need to ensure the first line of your debian/changelog on trunk contains your desired source package name.

Bileto (last edited 2021-02-10 09:39:58 by sil2100)