Landing changes through Bileto
Hello lander! If you got here, you're probably looking for information about how to use Bileto (formerly known as the "CI Train") or what to do to get your project landed to Ubuntu. For general information, make your way to the Landing your change to Ubuntu section below. If you already know the basics and just want to know which of the tools and processes to use for your specific landing, the Which landing method to use? section should help you.
Landing your change to Ubuntu
There are two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is maintained by Canonical.
Whenever you want to request a release of your component to the archive:
If this is your first time running a landing, ask the trainguards in #ubuntu-ci-eng to give you the required permissions for operating Bileto.
If this is the first time this source package has ever been released with Bileto, refer to the Inline Packaging guide to meet the minimum packaging standards.
Open the tickets page.
Sign in with SSO, click Create New Request, and you'll be presented with a form to fill out.
- Description of landing: write here an overview of what your landing is about - is it fixing bugs? Adding some features? Try being informative without being over-verbose.
Test plans to run: this mandatory field documents how you, the lander, intend to test the given component(s). Generally you would put a URL to a document describing steps to test that your component functions correctly, as this information will generally not change much from one ticket to the next. See #QA for more details.
Merge proposals to land: here you list all the links to the merge proposals you want to land in this landing. Please note, this must be a newline-separated list of MP links, not branch links. More on this later.
- Manual Download URLs: This optional field contains click packages / device tarball updates that don't require a PPA for testing, but instead must be downloaded manually by the tester.
- Landers: List of people who care about this landing. These names will be pinged on IRC when ticket status changes, so use IRC nicks if possible.
- Source Package Names: You can leave this blank and the correct value will auto-fill based on the contents of the PPA.
- Target distribution: select which distribution you want to land into. Currently only ubuntu is supported but if ubuntu-rtm ever makes a comeback you'd select that here.
Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select yakkety+xenial+vivid.
QA signoff: Values "N/A", "Required", or "Ready" will be automatically set as appropriate for your landing. The rule is: any stable-phone targets like vivid or 'yakkety+xenial+vivid' landings by default require it. See #QA for more information.
- Once the request is filled out and you've clicked 'Create New Request', you can assign a PPA by clicking 'Assign' from the menu at the bottom of your new ticket.
- Once your PPA is assigned, the menu will show some new buttons. Use the 'Build' button to start building packages from your merge proposals.
- If you have manual packages not built from merges, you need a trainguard or Ubuntu core-dev to upload those to your PPA.
After the packages finish building, it is now your responsibility to test the packages from the PPA on a real device. More information about this is in the #QA section. Remember: you are responsible not only to test that the fix works, but also to test that your change didn't break anything else. Use the Test Plan.
- If your packages pass your testing criteria, mark the ticket as tested:
- Get all your MP's top-approved. From Bileto's perspective, you are free to approve your own merges, however in some teams this is frowned upon, so you must get your branches reviewed by your teammates.
- Set 'Lander Signoff' to 'Approved'.
- Britney will now automatically inspect your PPA and perform various installability checks and autopkgtests. Once britney approves your ticket, it will automatically be marked ready for QA, if appropriate.
- If your ticket says "QA Signoff: Ready" you must wait for a QA person to approve your ticket.
- Once your ticket has been approved by both Britney and QA (or if QA is N/A), you can publish your own PPA. Landers are permitted to publish their own PPAs so long as there are no packaging changes, and no manual sources in the silo. In the event that you do have either packaging changes or manual sources, it is your responsibility to find a core dev to publish the PPA for you.
This is the standard process. When you prepare a list of MPs to land, the infrastructure will merge all of them together and build a source package using dpkg-buildpackage. This source package is then uploaded to the PPA that has been assigned to your ticket. The debian/changelog is automatically generated for you using the MPs' Commit Message fields. Every ticket creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time. After all changes land into the archive, these merges are automatically merged into trunk.
So, the general steps in the landing process from the lander's perspective are:
- Fill in a landing request
- Assign it yourself
- Build packages in PPA
- Test packages from the PPA on real hardware
- Mark ticket as Lander Signoff: Approved
- Wait for Britney to mark the ticket Approved.
- (in some cases) Wait for QA to mark the ticket Approved
- Publish the ticket yourself
- (in some cases) Find a core dev to ACK packaging changes and publish the ticket for you
Manual Source Uploads
Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released through Bileto: either because of the QA sign-off requirement for touch, or simply because they're needed by a ticket, mixed-in with normal MP-based packages.
- Prepare the source packages locally
- Get the package uploaded to the PPA:
- If you're a core-dev: simply use dput to upload the package to the PPA
- In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you (one option is to dput to your personal PPA and then core-dev or trainguard can copy the package from there).
Proceed normally with building & testing
Landing your change to the Stable Overlay PPA
The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.
How this works is: all current devel builds (and soon all stable builds) are built from vivid + the stable overlay PPA. This means we get all the goodness from vivid-updates and vivid-security, while we keep the dynamics of publishing touch packages for our purposes without having to go through the SRU process.
For a trio ticket such as yakkety+xenial+vivid, the destination field is ignored. The the first series never goes to the stable overlay PPA, and the others always do.
Trio landing for stable and devel
The current development series is yakkety. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep yakkety and xenial in sync, Bileto allows tickets to be built for all three together automatically.
NOTICE! Remember that all trio landings are required to go through the QA sign-off process because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your silo won't be released as a trio landing. What the trainguards can do for you is release the yakkety version and then request you to sync your work back to the vivid stable-overlay once the gates are open.
- Fill in a standard ticket
- The merges need to point to your trunk (devel) branch, as the merges will be built against yakkety and then copied to xenial and vivid
- In the Target Series drop-down select "yakkety+xenial+vivid"
- Assign your own PPA
- Build the ticket
- The build job will create 3 source packages - one for yakkety, one for xenial, and one for vivid - and upload them to the PPA
- The primary target is always yakkety
- Your merges will be built for yakkety (with the changelog entries generated for yakkety), but then the sources will be duplicated (with re-written versions) for xenial and vivid
- Test the generated packages on real hardware
- Please test all three
- Set Lander Signoff: Approved
- Wait for Britney to approve the ticket
- Wait for QA to approve the ticket
- Publish it yourself once everything is approved.
Because of the gcc5 C++ ABI break, symbols file maintenance has become more difficult. The compiler generates differently-mangled names for the old and new C++ ABI, so it is not possible to use a single symbols file for both 4.9 and 5.x ABIs.
Rather than maintaining two symbols files, it is better to maintain a symbols file only for vivid and to use a shlibs file for xenial. Nothing is lost with this approach because the exact same code is compiled for both distros. Therefore, any difference in symbols files between 4.9 and 5.x is due only to changes in the compiler (name mangling and inlining). Or, to spell it out more clearly: a symbols file for xenial cannot detect any ABI breaks that won't also be detected by the symbols file for vivid.
A workable way to keep a single source tree for both vivid and xenial is to generate the symbols file for vivid, and to generate a shlibs file for xenial during the build. (Also see below--it is possible to ditch the symbols files altogether.) In addition, you need to generate all the other debian files, such as control, *.click-hook, and the various *.install files. The hook you need is called debian/bileto_pre_release_hook:
The train will invoke a shell script that generates the files depending on the series. It's the usual hackery with sed and the like, but it gets the job done quite effectively. This script runs after all your merges have been merged but before the 'Releasing ...' commit is committed, so any changes you make in this script will be included in the release commit.
The script reads the current version number from a VERSION file in the debian directory. The version file contains something like 1.0.1. (By definition, we are ABI compatible if the minor and major version number stay the same---your rules may be different.) The script then parses out the three version components and derives an soversion number from that. (Our CMakeLists.txt reads the version info from the same VERSION file so, when the version number changes, we change it once in that file and everything else for the build and the packaging adjusts automagically. (For example, we use this to correctly set the soname when we build a shared library, and substitute the soversion into the .pc file for pkgconfig.)
The script uses sed to modify the various debian files. For example, control is generated from control.in. The control.in file contains substitution markers that sed looks for. For example:
This then turns into libunity-scopes3 on vivid, and libunity-scopes1.0 on xenial and later.
Generating the other files uses similar tricks to get the correct xenial or vivid version and soname into the right spots.
For unity-scopes-api, we have ditched symbols files altogether and use abigail instead. While the symbols file check is better than nothing, it does not detect many ABI breaks, such as changing the return type of a function, re-ordering struct members, changing an enumeration, making a non-virtual method virtual (plus many, many others). abigail does a much better job at this and provides a far better assurance that the ABI is indeed still intact.
ABI compliance testing is implemented as normal unit test, so it runs every time we build. You can find the relevant scripts here. The test requires a baseline ABI dump for each architecture. One easy way to get at these files is to make a temporary ticket that generates the ABI dump and adds it to the binary package. Once the PPA is built, you can grab the binary package for each architecture, pull out the dump file, check that into the source tree for future use, and then throw the PPA away.
ABI compliance checking is fast enough to tolerate. (For unity-scopes-api, which is large, it adds about 6 minutes to our total build time in Jenkins on Arm.) For most packages, that time will be much less. The added safety and no need to maintain symbols files any longer are well worth the wait though. Note that, when you do decide to break ABI, you will need to re-generate the new baseline ABI dumps and add them to the source tree.
Removing Packages from your Ticket
- You must remove the MPs from the merges field, and remove the package name from the Source Package Names field.
- You must ask a trainguard or core-dev to remove the package from the PPA.
CI Train Operation
Abandon a ticket
If your ticket is a failure and you want to give up on it, you must Abandon it. Abandoning a ticket will delete all the packages in the associated PPA, delete all the diffs that have been uploaded to swift, and set the status to 'Abandoned', which closes the ticket.
QA Sign-off Needed
Under certain circumstances your ticket may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:
- For all landings to the stable images (vivid + overlay PPA)
There is an exception to this rule if your landing contains an isolated bug fix. Right now there is not a strict definition of this term, but as guidance an isolated bug fix landing is:
- Only one or a couple of bug fixes
- A small change and not impacting a large number of sub-components of that package
- A change in a package that does not have a lot of reverse dependencies, since this makes it much harder to assess the impact of the landing.
If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the Test plans field of the ticket so that QA have a good basis on which to build their additional testing. Here are some tips:
- Never just test that the bug(s) alone are fixed - remember the potential for regressions
- Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the silo is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
Take into consideration the dependencies and reverse dependencies involved - ask yourself, who could I break? Ideally your package's test plan should have some guidance on this (for example see the Dependents/Clients section media-hub's test plan
- If you only provide a link to the test plan, then bear in mind that QA will assume you've run all the tests in that plan and they have passed. If tests then fail for a reason you haven't mentioned then it will cause confusion and delay the landing, so make sure to record all cases where you have not run a test or a test has failed (and why).
Lastly, if you don't use the citrain tool for installing the silo then you need to mention what steps you used to install the silo, since citrain is the assumed way.
Isolated bugfixes for stable - confirmation
Whenever a landing is targeting a stable branch and is marked by the lander as not requiring QA sign-off because of an isolated bugfix, the Landing Team tries to verify if this indeed falls into this category. The Landing Team either by themselves or with the help of the QA team look at the landing description, looking for any rationale for this exception. Merge requests or diffs are also examined in case that's needed.
The best practice is to directly ping the QA team with a question if your change can go through without a full QA sign-off. The QA Team is here to ensure that things landing into stable are as safe as possible.
Interpreting Silo Statuses
CI Train monitors silo statuses at a per-package level, which can lead to some detailed status messages. Fortunately, in many common cases, irrelevant details are omitted for clarity. Here are some examples:
Successfully built or Failed to build
When there is no package list, it means that all packages have this same status. Error states such as Failed to build or Dependency wait will appear if that status appears on any arch, but Successfully built will appear only if it succeeded on all arches.
Failed to build (foo-package/xenial). Successfully built (bar-package/xenial, grill-package/xenial).
- When there are multiple sentences with packages/series in brackets, the message applies to the packages named in the brackets. So in this example foo-package failed to build in xenial while the other two succeeded.
Proposed pocket (foo-package/xenial). Release pocket (foo-package/vivid).
This means that the silo has been published but has not yet made it all the way to Ubuntu archive yet. When publishing a dual silo, the vivid package is copied immediately to the overlay PPA, so it will indicate "Release pocket" (the fact that it's the release pocket of the overlay PPA and not the release pocket of the vivid archive is implied by the fact that it's a dual silo. dual silos never target vivid archive). Once the xenial package finishes migrating through Proposed and into Release, the train will notice this and auto-merge your merges for you.
Needs rebuild due to new commits (foo-package/xenial). Successfully built (foo-package/vivid).
- That's weird! Why doesn't the vivid one reflect that it also needs to be rebuilt to match the xenial one? Well the class that implements the vivid half of dual silos does not have any knowledge of MPs or commits, so when the xenial half notices it has unbuilt commits, the vivid half has no way to also check that. So this is the expected status when a dual silo has new commits. Rest assured that when you rebuild the package, both xenial and vivid packages will be rebuilt, the train cannot build one without the other so they can never get out of sync.
Common Status Messages
Ready to build
- No package is in the PPA, mostly likely because the silo is newly assigned, but also possibly because the packages have been deleted.
PPA/bzr version mismatch
The version number in the PPA does not match the version number in the bzr branch. A number of different situations can trigger this:
- If the upload to the PPA has failed. Launchpad batch-processes PPA uploads so even if the log indicates a successful upload, launchpad can still reject it later. Generally the conditions under which launchpad would reject an upload are predictable so if this happens it is almost always a bug in the train code that made it generate a bad version number for the package that launchpad doesn't like (eg, if it's lower than the existing one in the PPA).
- If the upload to the PPA is just being really slow. At times launchpad can be under heavy load and your package just might take a while to appear in the PPA.
- If the push to the bzr branch fails. This has never happened as far as I know, but if it did happen it would cause this error message.
If you see this message prior to the first time you've built the silo, it means the train is looking at the bzr branch leftover from the last time this particular package was built in this particular PPA, and then not seeing that version in the PPA, and getting confused. This fixes itself the first time you build the PPA.
- This happens if you have merges that are either Rejected or Superceded. Click the "Status" link to see the full log, which indicates which merges need attention.
Needs rebuild due to new commits
- The Train has detected new commits on either the source or target branches of your merges. Click the "Status" link to see the list of where new commits have been detected. Note that this message won't appear if the only new commits are Launchpad Translation auto-commits.
Needs rebuild due to burned version number
This happens when the same source package is built in two different silos on the same day, such that they both have the same version number, and one of those silos has been published. The other silo sees that the destination archive has a matching version number, but the package has differing contents, so the current version number is now "burned". Most likely this message will shortly be followed by the new commits message after the other silo merges, so take this as an early warning that you'll soon need to rebuild (but not yet!).
- This silo has been published and is on it's way to the archive! The silo will automatically merge once all packages are found to be in either the Release or Updates pocket. It is safe to manually merge when packages are in the Proposed pocket, however this is discouraged because you most likely want to do this when something is stuck in Proposed, and force-merging is just ignoring the problem. Much better to fix the problem and let the package migrate normally.
If your package has never been released in ubuntu before, it will find itself in the NEW queue, and if the archive is frozen (or you're doing an SRU), it will be in the UNAPPROVED queue. This is not something that will migrate on it's own and does require manual intervention to prod along, so be sure to ask for help if you see this state and you're not sure about it. WARNING: it is not safe to merge the silo in this state! If you merge, the packages will be lost and your release will never make it to the archive.
Apologies for this cryptic message, this happens when the package has been uploaded to the -proposed archive but the silo destination is set to a PPA instead of the ubuntu archive. "DONE" means that the upload to the ubuntu archive was successful, but it doesn't give the expect "Release pocket" or "Proposed pocket" message because it's looking at the overlay PPA instead, and the package isn't there. This situation can be resolved by copying the package from your silo into the overlay PPA.
Destination version [VERSION] is missing from changelog
The current version number in the destination archive (either ubuntu or overlay PPA depending on how your request is configured) could not be found in the debian/changelog in the local tree. This can either be caused by manual uploads, or by conflicting train silos that have already been published. You'll need to investigate the destination archive and see what version is there. If it's a manual upload, you'll need to sync it back to trunk manually, otherwise if it's another train upload, you'll need to wait for it to merge to trunk normally, then rebuild your silo to incorporate those changes.
Failed to build
- Your package was uploaded to the PPA but did not build successfully! Click the "Status" link to see the full list of what packages/arches failed and links to the failure logs.
- The train now automatically generates diffs when it sees this message, so if you're seeing this message it's a bug! Please report it.
- Absolutely no problems could be (automatically) found with your package build. Congrats! Test it out and publish it!
[source_package_name] [version] is missing from the changelog, which has up to [lesser_version]. Please sync destination version back to trunk.
This happens when an ubuntu core developer makes a manual upload of your package to ubuntu without using ci train. 99% of the time this would be no-change rebuilds during library transitions, or other trivial fixes for packaging issues. Before you can build the silo, it is necessary to sync the manual distro upload back to your trunk, so that they match. The first thing to do is check the launchpad page for your source package (for example here's the page for unity8. On that page look for a version number that ends in "0ubuntu2" or similar (the train only ever generates versions ending in "0ubuntu1"). Click through that page to get to the "available diffs" for that version, download that diff, and then commit that directly to your trunk. Do not make a merge proposal with this diff and include it in the silo, that does not work and is a backwards approach. You want to copy what's in distro into your trunk, silos are for getting branches into distro.
- You typed a package name into PACKAGES_TO_REBUILD that wasn't part of the silo configuration. You should double check your spelling, and if it's correct, then your request is misconfigured.
All CI Train silo status changes are pinged in #ubuntu-ci-eng on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about. As a convenience, when making a new request, Bileto will insert your launchpad nick into the landers field. Unfortunately, Bileto doesn't know your IRC nick though, only launchpad nick, so if those differ, the easiest thing for you to do might be to set an IRC highlight for your launchpad nick. Then when the bot pings the status message in the channel you'll see the highlight. Either that, or just make sure to change your launchpad nick to your IRC nick every time you make a landing request.
Important behavioral changes in the train are occaisionally anounced via the mailing list. Please join the team & subscribe to the mailing list so you stay up to date with important announcements.
By default, the train will generate a debian/changelog entry for you, based on the text you enter into the "Commit Message" field of the merge proposals in the silo. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series UNRELEASED so that the train can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, the train will make a new empty changelog message which isn't what you want.
Installing a PPA onto an Ubuntu Touch device is a bit of a chore because you have to set the device writable, pin the silo PPA higher than the overlay PPA, etc. Fortunately there is a tool to automate this.
$ sudo add-apt-repository ppa:phablet-team/ubuntu/tools $ sudo apt-get install phablet-tools-citrain $ citrain help
How to Coordinate the Publishing of Multiple Large Landings
Sometimes, there will be a few large silos wanting to publish around the same time, eg mir and unity8. When this happens, best practices is to do an image build in between publishings, because this makes it easier to bisect where regressions came from. It's a little bit tricky however, because certain tools report information differently and there's a race condition that you can fail if you're not careful. Best to follow these steps:
1. Publish one of the silos.
2. Wait for it to migrate. Eventually, queuebot will ping that the request is Landed.
3. It's not yet safe to kick the image build, however. You must now use rmadison to confirm that the packages are truly out of the -proposed pocket and in the release pocket.
4. Once rmadison confirms publication has completed, now kick the image build.
5. Wait until the image build has completed, to ensure the build succeeds and doesn't need to be retried or fixed.
6. Now publish the second silo.
Renaming a Source Package
Are you trying to rename a source package and getting weird errors? Don't do that!
Just kidding, but here's what's going on. In jenkins, there's a function that looks up bzr branches and fetches the source package name from the branch. It does this by downloading the debian/changelog, and then throwing away everything except the very first word, which is the source package name. This is obviously hilariously slow and inefficient, so a disk cache was implemented, and the cache has no expiry logic, so you can never, ever rename your source packages, ever. In the event that you do need to rename your source package, you'll need to ask one of the trainguards to manually flush the cache for you, which can be done by deleting /tmp/cu2d_cache/ on the jenkins instance.