Landing changes through the CI Train
Hello lander! If you got here, you're probably looking for information about how to use the CI Train or what to do to get your project landed to Ubuntu Touch. For general information about the usage of the train, make your way to the Landing your change to Ubuntu section below. If you already know the basics of the train and just want to know which of the tools and processes to use for your specific landing, the Which landing method to use? section should help you.
Landing your change to Ubuntu
The train provides two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is managed by the CI Train.
Whenever you want to request a release of your component to the archive:
If this is your first time running a landing, ask the trainguards in #ubuntu-ci-eng to give you the required permissions for operating the CI Train.
If this is the first time this source package has ever been released in Ubuntu, refer to the Inline Packaging guide to bring your packaging up to the train's expectations.
Open the CI Train requests page.
- Sign in with SSO, and you'll be presented with a form to fill out.
- Description of landing: write here an overview of what your landing is about - is it fixing bugs? Adding some features? Try being informative without being over-verbose.
Test plans to run: each project handled by CI Train needs to have a test plan that documents how you, the lander, intend to test the given component(s). It's best to present this in the form of a wiki page or other document that can be referred to outside the train. See #QA for more details.
Merge proposals to land: here you list all the links to the merge proposals you want to land in this landing. Please note, this must be a newline-separated list of MP links, not branch links. More on this later.
- Manual Download URLs: This optional field contains click packages / tarball updates that don't require a PPA for testing.
- Name of assigned silo: Leave this blank, the train will fill it out for you.
- Status: Leave this blank also, the train will fill it out as well.
- Landers: List of people who care about this landing. These names will be pinged on IRC when silo status changes, so use IRC nicks if possible (the field defaults to containing your launchpad nickname, I recommend setting an IRC highlight for your lp nick if it's different from your IRC nick).
- Sync Source: This is a special field used only when you are copying packages from one series to another. Hover your mouse over the field to see some example syntax.
Destination PPA: select or type in the PPA to which the packages should be released instead of the main archive - currently mostly used only for cases when landing to the stable overlay PPA is needed. See Landing to the Stable Overlay PPA. If you are doing a touch-specific landing for vivid, you'll want to specify ci-train-ppa-service/ubuntu/stable-phone-overlay.
- Additional source packages to land: here you list the 'additional' source packages that you would like to upload to the silo PPA besides what is already handled by MPs. Can be used for projects that Ubuntu is not upstream for or which have no bzr trunk.
- Target distribution: select which distribution you want to land into. Currently only ubuntu is supported but if ubuntu-rtm ever makes a comeback you'd select that here.
Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select Dual-Landing.
QA signoff status: the rule is: any stable-phone targets like vivid or 'dual' landings by default require it. See #QA for more information. Set to either 'No QA Needed' or 'QA Required' to start. See QA Signoff Status field for more information.
- Once the request is filled out and you've clicked 'Save', you can assign the silo in jenkins by clicking 'Assign' from the menu that hovers beneath your row when you mouse over it, and then 'Build' on the resulting jenkins job.
Once your silo is assigned, the hover menu will grow three new links, 'Build', 'Publish', and 'Merge&Clean'. Use the 'Build' button to start building packages from the specified merge requests.
- If you have added some 'Additional source packages to land', you need to provide the trainguards or Ubuntu core-dev with the required source package so that they can upload those to your silo. Any core-developer should be able to dput packages to the silo PPAs as well.
After the packages finish building, it is now your responsibility to test the packages from the PPA on a real device. More information about this is in the #QA section. Remember: you are responsible not only to test that the fix works, but also to test that your change didn't break anything else. Use the Test Plan.
- If your packages pass your testing criteria, mark the silo as tested:
- Get all your MP's top-approved. If you don't use the feature in your team's processes, just mark it as top-approved yourself.
- Hover over your landing request and click 'Edit' in the hover menu.
Set the QA Signoff Status dropdown to either 'Publish without QA' or 'Ready for QA' as appropriate (generally speaking if your silo only targets the ubuntu devel release, you don't need QA, but if you target the stable release or 'dual' then you do need QA). See QA Signoff Status field for more information.
- In case your silo needs QA sign-off, wait for the QA Team to finish testing on the silo. This will be indicated in the requests and the dashboard.
The landing team will now publish the silo to the archive if everything is OK packaging-wise. Note: Merge Proposals need to be top-approved latest at this point, otherwise landing team cannot publish the silo.
This is the standard process. When you prepare a list of MPs to land and provide them to CI Train, the infrastructure will merge all of them together and build a source package using bzr bd. This package is then uploaded to the silo PPA that has been assigned to your landing. CI Train also automatically generates the debian/changelog for you using the MPs' commit-messages. Every landing creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time. After all changes land into the archive, during the 'Merge & Clean' step these merges are all finally merged into trunk.
So, the general steps in the landing process from the lander's perspective are:
- Fill in a landing request
- Assign it yourself
- Build packages in silo PPA
- Test packages from the silo PPA on real hardware
- Mark silo as tested
- (in some cases) Wait for silo to be QA signed-off
- Wait for silo to be published
The QA Signoff Status field
The current Bileto implementation has one field to indicate the overall sign-off status of a landing. This means that information of if the silo needs QA sign-off or not is in the same place as the information if the silo has been tested by the lander and is ready to publish. This field is a drop-down list named 'QA Signoff Status'. Possible statuses:
No QA Needed - the silo is not tested yet by the lander, but once it is, it does not require additional QA sign-off.
Publish without QA - the silo that did not require QA sign-off is now tested by the lander and is ready to be published.
QA Required - the silo is not tested yet by the lander and once it is, it will require an additional QA sign-off.
Ready for QA - the silo is now tested by the lander and is ready for QA sign-off. Please note that only now the silo appears on the QA queue.
QA Granted - the silo has passed both the lander testing and QA sign-off and is ready to publish.
QA Failed - the silo failed QA sign-off.
CI Train for manual source uploads
Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released as a silo: either because of the QA sign-off requirement for touch, or simply because they're needed by a landing, mixed-in with normal MP-based requests.
In this case, the "Additional source packages to land" field needs to be used.
- In your request, put the name of the source package in the Additional source packages to land field (space-separated)
- Fill in the rest of the request as normal
- Remember you can mix merge-proposals and source uploads at will
- Currently source uploads are not supported for dual-landings and sync silos
- Silo gets assigned
- Prepare the source packages locally
- Get the package uploaded to the PPA:
- If you're a core-dev: simply use dput to upload the package to the silo PPA
- In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you
- Once the package is uploaded, run the Build job with WATCH_ONLY set
- This will make the train scan the PPA for new uploads and check if they build or built correctly
- Proceed normally with rest
Landing your change to the Stable Overlay PPA
The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. In other words, it's the replacement for the ubuntu-rtm/14.09 series. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.
How this works is: all current devel builds (and soon all stable builds) are built from vivid + the stable overlay PPA. This means we get all the goodness from vivid-updates and vivid-security, while we keep the dynamics of publishing touch packages for our purposes without having to go through the SRU process.
By default, whenever a landing request is set for vivid and the packages affected (or it's obvious from the landing description) are touch-specific, you should set ci-train-ppa-service/ubuntu/stable-phone-overlay as the value for the 'Destination PPA' field. This means that once the package is published it will be copied to the overlay PPA instead of migrating to the vivid archive.
Syncing from and to the Stable Overlay PPA
A "sync" silo should only be used if you want to copy an existing package between wily and the overlay ppa (in either direction). If you have a single package that you want to land in both targets, please use Dual-Landings.
WARNING! Sync silos should only be used for packages that are released through the train and include the train-generated version number. For packages that are either released through train source uploads or standard archive-uploads, please copy the packages manually with version numbers changed. See the rationale for this in citrain/SyncSilos.
Currently syncing can work in both ways - from vivid to wily and the other way around. The recommended way is to first land your change to wily and then syncing it back to vivid-overlay - as per the standard procedure of first developing for devel. This can be done in a few ways. The recommended one is:
- Request two silos - one for vivid-overlay and one for wily
- Put your merges to the wily silo
- For the second request only put the silo number of the first request in the 'sync' field.
- A trainguard will first assign the main silo and then configure the second silo to fetch (sync) source packages from the first one
- Build your main silo
- Test your main silo
- Once you're happy with your main silo, go to the secondary one and press build
- The build job will sync packages from the first silo and rebuild them for the new series
- Test your secondary silo
- If there are issues with the silo on the other platform, you will have to rebuild both silos and re-test again
- Once you are happy with both silos, set both of them as ready
Another possible way, but less safe as one might forget about syncing their work to the other archive, is this:
- Perform a normal landing to wily and complete it the normal way.
- Request a new silo, a sync silo - no merges, only 'ubuntu,wily' in the sync field and fill out the list of additional sources with the source packages you want synced.
- Build and test the silo
- Mark as tested and wait for QA sign-off
All sync possibilities are briefly overviewed on citrain/SyncSilos.
Dual-landing for stable and devel
The current development series is wily. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep wily in sync, the CI Train offers functionality for performing dual landings to both overlay and wily through one silo. This is called dual-landing.
NOTICE! Remember that all dual landings are required to go through the QA sign-off process because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your silo won't be released as a dual landing. What the trainguards can do for you is release the wily version and then request you to sync your work back to the vivid stable-overlay once the gates are open.
- Fill in a standard landing request
- The filled merges need to point to your trunk (devel) branch, as the merges will be built against wily and then duplicated to vivid
- In the Target Series drop-down select "dual"
- Wait for silo assignment
- Once assigned, build the silo
- The build job will create 2 source packages - one for wily, one for vivid - and upload them to the silo PPA
- The primary target is always wily
- Your merges will be built for wily (with the changelog entries generated for wily), but then the sources will be duplicated (with re-written versions) for vivid
- If a package build fails for one series, it needs to be fixed in the merge and packages in all series in the silo need to be rebuilt
- Test the generated packages on real hardware
- Please test both wily and vivid
- Mark the silo as tested
- Wait for silo to be QA signed-off
- Wait for the silo to be published and auto-merged
Once published, the silo merge and clean step will be blocked until all packages migrate to their destinations
Dual-landing and handling symbols
Because of the gcc5 C++ ABI break, symbols file maintenance has become more difficult. The compiler generates differently-mangled names for the old and new C++ ABI, so it is not possible to use a single symbols file for both 4.9 and 5.x ABIs.
Rather than maintaining two symbols files, it is better to maintain a symbols file only for Vivid and to use a shlibs file for Wily. Nothing is lost with this approach because the exact same code is compiled for both distros. Therefore, any difference in symbols files between 4.9 and 5.x is due only to changes in the compiler (name mangling and inlining). Or, to spell it out more clearly: a symbols file for Wily cannot detect any ABI breaks that won't also be detected by the symbols file for Vivid.
A workable way to keep a single source tree for both Vivid and Wily is to generate the symbols file for Vivid, and to generate a shlibs file for Wily during the build. In addition, you need to generate all the other debian files, such as control, *.click-hook, and the various *.install files. The hook you need is in debian/rules:
override_dh_auto_clean: /bin/sh $(CURDIR)/debian/gen-debian-files.sh dh_auto_clean
This calls a shell script that generates the files depending on the distro. It's the usual hackery with sed and the like, but it gets the job done quite effectively.
Check out the script here.
The script reads the current version number from a VERSION file in the debian directory. The version file contains something like 1.0.1. (By definition, we are ABI compatible if the minor and major version number stay the same---your rules may be different.) The script then parses out the three version components and derives an soversion number from that. (Our CMakeLists.txt reads the version info from the same VERSION file so, when the version number changes, we change it once in that file and everything else for the build and the packaging adjusts automagically. (For example, we use this to correctly set the soname when we build a shared library, and substitute the soversion into the .pc file for pkgconfig.)
The script uses sed to munge up the various debian files. For example, control is generated from control.in. The control.in file contains substitution markers that sed looks for. For example:
This then turns into libunity-scopes3 on Vivid, and libunity-scopes1.0 on Wily.
Generating the other files uses similar tricks to get the correct Wily or Vivid version and soname into the right spots.
Eventually, we should ditch symbols files altogether and use abi-compliance-checker instead. While the symbols file check is better than nothing, it does not detect many ABI breaks, such as changing the return type of a function, re-ordering struct members, changing an enumeration, making a non-virtual method virtual (plus many, many others). abi-compliance-checker does a much better job at this and provides a far better assurance that the ABI is indeed still intact.
Which landing method to use?
Seeing how many different scenarios and landing techniques are available, sometimes it might be confusing as to which methods to use to land your change to the right place. Currently there are two main landing areas: devel and stable. Devel is a place for all the risky changes that are not yet ready for mainstream usage while stable is what all the shipped user phones are using.
My project only has stable features
In this case, and probably in most other ones, the best way is to use Dual-landing. Your project should only have one trunk and all your merges should be targeting it.
My project had only stable features but now it will diverge
Up until now you probably used dual-landings, which is good, but now you need to release different things to devel and stable.
Currently this is a bit of a tricky situation. Normally you should now diverge your trunk into two branches: leave the current one as devel and create a stable one for stable. But we need to trick CI Train into something before this can work. Since you used trunk to dual-release, the current top-most version of the branch has a devel (currently wily) version number, while stable uses a smaller versioning number. This can cause issues in the train saying things like: cannot release a version lower than the last one.
To deal with this, after splitting and creating your stable branch, you need to modify the bzr branch by hand. You need to change the latest debian/changelog entry's version number to correspond to the currently released stable version. This can look for example like this - current version in trunk 0.3+15.10.20150601-0ubuntu1, needs to be changed to its stable counterpart, so something like: 0.3+15.04.20150601-0ubuntu1.
Once that's done, land individual changes to stable and/or devel whenever you want using the standard procedures. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project develops rapidly and only selective changes are good for stable
In this case it's best to start off by simply creating two trunks (bzr branches) from the start. Now, use standard landings to land for the devel and stable branches. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project only targets stable, but I have risky features I want to test first without breaking user experience
For such a scenario it's best to use syncs. You only have one trunk basically targeting both devel and stable, but in fact the stable versions will be always slightly behind in the archives. Simply release your risky changes to the devel version (wily currently), test them for as long as you wish (in the meantime you can release new versions with fixes to devel) and once you feel it's ready for stable - request a sync silo from devel to stable.
Note that you can use syncs and dual-landings intertwined. For instance, you can release your project using dual-landing for everything that's safe, but switch to syncs once you feel like the current features are risky. Then, once you stabilize your devel branch, you can use dual-landings once again.
Modifying your existing landing
Sometimes after a landing is prepared, a silo assigned and the packages built, during testing the lander can notice that some changes are missing and need to be added for the feature to be complete. This usually requires additional merges or sources included in the silo. CI Train handles this case, but requires certain actions.
Whenever a merge or source is added to the landing request after assignment, the merge and source is not automatically noticed by CI Train. A silo reconfigure is needed first.
To reconfigure, please 'Edit' your request and follow the Assign link present in your landing line. The train will detect that the assignment has the same requestid as an existing silo and reconfigure it instead of assigning a new one.
Also, an important notice for both landers and trainguards: when removing an entire package from a silo configuration, the packages are not automatically removed from the PPA. You must contact a trainguard (or ubuntu core dev) to remove the package from the silo PPA for you.
Dealing with twin packages (-gles)
Some packages in our archives come with so-called package twins - those are usually packages that build for normally un-buildable architectures. Upstreams use those for packages such as qtubuntu and qtmir which normally do not build for desktop architectures. Therefore the *-gles packages have been created for the use by the emulator.
To keep both the normal and -gles packages in sync, the notion of twin packages has been introduced. If the CI Train detects that a silo has such a package in its landing, it will not allow it normally building before the -gles counterpart gets configured in the silo as well. Currently it's a rather troublesome process, but upstream developers invented a rather easy workflow to make it less problematic.
This scenario overviews what to do when you know your projects require twin package uploads.
- When preparing the request, besides all the standard MRs, also include all the required -gles package names as Additional source packages to land (e.g. qtubuntu = qtubuntu-gles)
- Build the silo normally
- Once the main packages build (e.g. qtubuntu), prepare sync MRs for the -gles branches in the following way:
- Target the -gles specific trunk (e.g. qtubuntu-gles)
In this branch, modify the changelog to have exactly the same version number as the main packages in the PPA (e.g. qtubuntu)
- Also, modify the debian/watch file to point to the silo where the packages were built
- Add the newly created MRs to the silo request, remove the -gles names from Additional sources
- Reconfigure the silo
- Build the -gles packages in the silo
CI Train Operation
Aborting ongoing builds
This is a use case that is very poorly supported by the train. The build job has two major parts, the 'preparing packages' part and the 'building' part (this is reflected in the silo status when you run the build job). During the 'preparing packages' phase, all silo state for the packages you've selected is wiped out, and recreated. If you cancel the build job during this phase, the silo will be left in an inconsistent/broken state, meaning that the packages that are in the PPA will be orphaned, and there will be no way to publish this silo as the necessary branches to merge to trunk will be missing. Don't ever cancel a build in this state, it's a bad idea, and you'll be forced to rebuild again anyway.
If the silo status says 'building', that means that the merges are finished being prepared and the train is simply polling launchpad to determine whether the packages have succeeded in building or not. In this state it is safe to cancel the build, however doing so will not actually stop the packages from continuing to build in the PPA, you're just cancelling jenkins from polling launchpad about the build status. The only real reason to want to do this is if you noticed you made a mistake and need to rebuild right away again immediately (as jenkins prevents two build jobs from running simultaneously).
WARNING! You can not abort the build job before the packages are prepared and uploaded to the PPA! The CI Train pretty much guarantees that after a build job is aborted early its state will be missing critical pieces. This will make the silo completely broken, requiring a reconfigure, reassignment or manual trainguard intervention.
Abandon a landing
In order to free a silo from jenkins and give up on the landing request, click 'Merge & Clean' on your landing request. This opens a Jenkins job where you must select ONLY_FREE_SILO and then click 'Build'. Once that job is complete, the silo will be freed, and the request will be set to 'Abandoned' for you.
If you want to abandon a request which does not have a silo assigned, just click 'Edit' and manually set the status to 'Abandoned' and then click 'Save'.
QA Sign-off Needed
Under certain circumstances your landing may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:
- For all landings to the stable images (vivid + overlay PPA)
There is an exception to this rule if your landing contains an isolated bug fix. Right now there is not a strict definition of this term, but as guidance an isolated bug fix landing is:
- Only one or a couple of bug fixes
- A small change and not impacting a large number of sub-components of that package
- A change in a package that does not have a lot of reverse dependencies, since this makes it much harder to assess the impact of the landing.
If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the Test plans to run column of the landing spreadsheet so that QA have a good basis on which to build their additional testing. Here are some tips:
- Never just test that the bug(s) alone are fixed - remember the potential for regressions
- Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the silo is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
Take into consideration the dependencies and reverse dependencies involved - ask yourself, who could I break? Ideally your package's test plan should have some guidance on this (for example see the Dependents/Clients section media-hub's test plan
- If you only provide a link to the test plan, then bear in mind that QA will assume you've run all the tests in that plan and they have passed. If tests then fail for a reason you haven't mentioned then it will cause confusion and delay the landing, so make sure to record all cases where you have not run a test or a test has failed (and why).
Lastly, if you don't use the citrain tool for installing the silo then you need to mention what steps you used to install the silo, since citrain is the assumed way.
Isolated bugfixes for stable - confirmation
Whenever a landing is targeting a stable branch and is marked by the lander as not requiring QA sign-off because of an isolated bugfix, the Landing Team tries to verify if this indeed falls into this category. The Landing Team either by themselves or with the help of the QA team look at the landing description, looking for any rationale for this exception. Merge requests or diffs are also examined in case that's needed.
The best practice is to directly ping the QA team with a question if your change can go through without a full QA sign-off. The QA Team is here to ensure that things landing into stable are as safe as possible.
All CI Train silo status changes are pinged in #ubuntu-ci-eng on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about. As a convenience, when making a new request, Bileto will insert your launchpad nick into the landers field. Unfortunately, Bileto doesn't know your IRC nick though, only launchpad nick, so if those differ, the easiest thing for you to do might be to set an IRC highlight for your launchpad nick. Then when the bot pings the status message in the channel you'll see the highlight. Either that, or just make sure to change your launchpad nick to your IRC nick every time you make a landing request.
By default, the train will generate a debian/changelog entry for you, based on the text you enter into the "Commit Message" field of the merge proposals in the silo. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series UNRELEASED so that the train can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, the train will make a new empty changelog message which isn't what you want.
Installing a PPA onto an Ubuntu Touch device is a bit of a chore because you have to set the device writable, pin the silo PPA higher than the overlay PPA, etc. Fortunately there is a tool to automate this.
$ sudo add-apt-repository ppa:phablet-team/ubuntu/tools $ sudo apt-get install phablet-tools-citrain $ citrain help
How to Coordinate the Publishing of Multiple Large Landings
Sometimes, there will be a few large silos wanting to publish around the same time, eg mir and unity8. When this happens, best practices is to do an image build in between publishings, because this makes it easier to bisect where regressions came from. It's a little bit tricky however, because certain tools report information differently and there's a race condition that you can fail if you're not careful. Best to follow these steps:
1. Publish one of the silos.
2. Wait for it to migrate. Eventually, queuebot will ping that the request is Landed.
3. It's not yet safe to kick the image build, however. You must now use rmadison to confirm that the packages are truly out of the -proposed pocket and in the release pocket.
4. Once rmadison confirms publication has completed, now kick the image build.
5. Wait until the image build has completed, to ensure the build succeeds and doesn't need to be retried or fixed.
6. Now publish the second silo.
Renaming a Source Package
Are you trying to rename a source package and getting weird errors? Don't do that!
Just kidding, but here's what's going on. In jenkins, there's a function that looks up bzr branches and fetches the source package name from the branch. It does this by downloading the debian/changelog, and then throwing away everything except the very first word, which is the source package name. This is obviously hilariously slow and inefficient, so a disk cache was implemented, and the cache has no expiry logic, so you can never, ever rename your source packages, ever. In the event that you do need to rename your source package, you'll need to ask one of the trainguards to manually flush the cache for you, which can be done by deleting /run/shm/cu2d_cache/ on the jenkins instance.
[source_package_name] [version] is missing from the changelog, which has up to [lesser_version]. Please sync destination version back to trunk.
This happens when an ubuntu core developer makes a manual upload of your package to ubuntu without using ci train. 99% of the time this would be no-change rebuilds during library transitions, or other trivial fixes for packaging issues. Before you can build the silo, it is necessary to sync the manual distro upload back to your trunk, so that they match. The first thing to do is check the launchpad page for your source package (for example here's the page for unity8. On that page look for a version number that ends in "0ubuntu2" or similar (the train only ever generates versions ending in "0ubuntu1"). Click through that page to get to the "available diffs" for that version, download that diff, and then commit that directly to your trunk. Do not make a merge proposal with this diff and include it in the silo, that does not work and is a backwards approach. You want to copy what's in distro into your trunk, silos are for getting branches into distro.
You typed a package name into PACKAGES_TO_REBUILD that wasn't part of the silo configuration. You should double check your spelling, and if it's correct, then your request is misconfigured.