Landing changes through the CI Train
Hello lander! If you got here, you're probably looking for information about how to use the CI Train or what to do to get your project landed to Ubuntu Touch. For general information about the usage of the train, make your way to the Landing your change to Ubuntu section below. If you already know the basics of the train and just want to know which of the tools and processes to use for your specific landing, the Which landing method to use? section should help you.
Landing your change to Ubuntu
The train provides two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is managed by the CI Train.
Whenever you want to request a release of your component to the archive:
If this is your first time running a landing, ask the trainguards in #ubuntu-ci-eng to give you the required permissions for operating the CI Train.
Open the CI Train requests page.
- Sign in with SSO, and you'll be presented with a form to fill out.
- Description of landing: write here an overview of what your landing is about - is it fixing bugs? Adding some features? Try being informative without being over-verbose.
- Lander: your irc nickname, and that of other people that will also drive the landing (space separated)
- Comments (optional): used for any additional comments and offline discussion with the Landing Team.
Test plans to run: each project handled by CI Train needs to have a test plan that documents how you, the lander, intend to test the given component(s). It's best to present this in the form of a wiki page or other document that can be referred to outside the spreadsheet. See #QA for more details.
Merge proposals to land: here you list all the links to the merge proposals you want to land in this landing. Please note, this must be a space/newline separated list of MP links, not branch links. More on this later.
- Additional source packages to land: here you list the 'additional' source packages that you would like to upload to the silo PPA besides what is already handled by MPs. Can be used for projects that Ubuntu is not upstream for or which have no bzr trunk.
Target distribution: select which distribution and series you want to land into. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select Dual-Landing.
Destination PPA: select or type in the PPA to which the packages should be released instead of the main archive - currently mostly used only for cases when landing to the stable overlay PPA is needed. See Landing to the Stable Overlay PPA. In case a touch-specific landing is requested for vivid, the LT will by default target it to the overlay PPA.
QA sign off needed?: usually this should be left blank, but the rule is: any stable-phone targets like ubuntu/vivid+overlay or ubuntu-rtm/14.09 landings by default require it. See #QA for more information.
- Once the request is filled out and you've clicked 'Save', you can assign the silo in jenkins by clicking 'Assign' on your request row, and then 'Build' on the resulting jenkins job.
After you have a silo allocated, it will appear on the CI Train Dashboard.
- Use the 'Build' button above your landing card in the dashboard to start building packages from the specified merge requests.
- If you have added some 'Additional source packages to land', you need to provide the trainguards or Ubuntu core-dev with the required source package so that they can upload those to your silo. Any core-developer should be able to dput packages to the silo PPAs as well.
After the packages finish building, it is now the lander's responsibility to test the packages from the PPA on a real device. More information about this is in the #QA section. Remember: you are responsible not only to test that the fix works, but also to test that your change didn't break anything else. Use the Test Plan.
- If your packages pass your testing criteria, mark the silo as tested:
Get all your MP:s top-approved. If you don't use the feature in your team's processes, just mark it as top-approved yourself.
Find your landing request on the CI Train Requests again (the request id number on the dashboard will link directly there).
- Click 'Edit'
- Set the QA Signoff Status dropdown to either 'Publish without QA' or 'Ready for QA' as appropriate (generally speaking if your silo only targets the ubuntu devel release, you don't need QA, but if you target the stable release then you do need QA).
- In case your silo needs QA sign-off, wait for the QA Team to finish testing on the silo. This will be indicated in the requests and the dashboard.
The landing team will now publish the silo to the archive if everything is OK packaging-wise. Note: Merge Proposals need to be top-approved latest at this point, otherwise landing team cannot publish the silo.
This is the standard process. When you prepare a list of MPs to land and provide them to CI Train, the infrastructure will merge all of them together and build a source package using bzr bd. This package is then uploaded to the silo PPA that has been assigned to your landing. CI Train also automatically generates the debian/changelog for you using the MPs' commit-messages. Every landing creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time. After all changes land into the archive, during the 'Merge & Clean' state these merges are all finally merged into trunk.
So, the general steps in the landing process from the lander's perspective are:
- Fill in a landing request
- Wait for silo assignment
- Build packages in silo PPA
- Test packages from the silo PPA on real hardware
- Mark silo as tested
- (in some cases) Wait for silo to be QA signed-off
- Wait for silo to be published
CI Train for manual source uploads
Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released as a silo: either because of the QA sign-off requirement for touch, or simply because they're needed by a landing, mixed-in with normal MP-based requests.
In this case, the "Additional source packages to land" field needs to be used.
- In your request, put the name of the source package in the Additional source packages to land column (space-separated)
- Fill in the rest of the request as normal
- Remember you can mix merge-proposals and source uploads at will
- Currently source uploads are not supported for dual-landings and sync silos
- Silo gets assigned
- Prepare the source packages locally
- Get the package uploaded to the PPA:
- If you're a core-dev: simply use dput to upload the package to the silo PPA
- In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you
- Once the package is uploaded, run the Build job with WATCH_ONLY set
- This will make the train scan the PPA for new uploads and check if they build or built correctly
WARNING! Without this step the train will not know about your direct source uploads
- Proceed normally with rest
Landing your change to the Stable Overlay PPA
The stable overlay PPA is the place where all the stable development will go after the first vivid-based OTA is released. In other words, it's the replacement for the ubuntu-rtm/14.09 series. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.
How this works is: all current devel builds (and soon all stable builds) are built from vivid + the stable overlay PPA. This means we get all the goodness from vivid-updates and vivid-security, while we keep the dynamics of publishing touch packages for our purposes without having to go through the SRU process.
By default, whenever a landing request is set for vivid and the packages affected (or it's obvious from the landing description) are touch-specific, the landing team will automatically target your silo to the overlay PPA instead of the archive. This means that once the package is published it will be copied to the overlay PPA instead of migrating to the vivid archive.
Syncing from and to the Stable Overlay PPA
A "sync" silo should only be used if you want to copy an existing package between wily and the overlay ppa (in either direction). If you have a single package that you want to land in both targets, please use Dual-Landings.
WARNING! Sync silos should only be used for packages that are released through the train and include the train-generated version number. For packages that are either released through train source uploads or standard archive-uploads, please copy the packages manually with version numbers changed. See the rationale for this in citrain/SyncSilos.
Currently syncing can work in both ways - from vivid to wily and the other way around. The recommended way is to first land your change to wily and then syncing it back to vivid-overlay - as per the standard procedure of first developing for devel. This can be done in a few ways. The recommended one is:
- Request two silos - one for vivid-overlay and one for wily
- Put your merges to the wily silo, but it can also be done the other way around
- For the second request only put 'sync:' in the 'Additional Sources to Land'
- A trainguard will first assign the main silo and then configure the second silo to fetch (sync) source packages from the first one
- e.g. if the first request is given silo 13 the second silo will have the following configuration line: sync:13,wily
- Build your main silo
- Test your main silo
- Once you're happy with your main silo, go to the secondary one and press build
- The build job will sync packages from the first silo and rebuild them for the new series
- Test your secondary silo
- If there are issues with the silo on the other platform, you will have to rebuild both silos and re-test again
- Once you are happy with both silos, set both of them as ready
Another possible way, but less safe as one might forget about syncing their work to the other archive, is this:
- Perform a normal landing to wily (or vivid-overlay, but less recommended)
- Request a new silo, a sync silo - no merges, only 'Additional source' with "sync:distribution,series package1 package2" - this can be either sync:ubuntu,wily or sync:stable-overlay,vivid.
- Build and test the silo
- Mark as tested and wait for QA sign-off (if needed)
All sync possibilities are briefly overviewed on citrain/SyncSilos.
Dual-landing for stable and devel
The current development series is wily. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep wily in sync, the CI Train offers functionality for performing dual landings to both overlay and wily through one silo. This is called dual-landing.
NOTICE! Remember that all dual landings are required to go through the QA sign-off process because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your silo won't be released as a dual landing. What the trainguards can do for you is release the wily version and then request you to sync your work back to the vivid stable-overlay once the gates are open.
- Fill in a standard landing request
- The filled merges need to point to your trunk (devel) branch, as the merges will be built against wily and then duplicated to vivid
- In the Target Distribution drop-down select "ubuntu/dual"
- The Target Distribution field now accepts lists of targets, space separated
- Wait for silo assignment
- Once assigned, build the silo
- The build job will create 2 source packages - one for wily, one for vivid - and upload them to the silo PPA
- The primary target is always wily
- Your merges will be built for wily (with the changelog entries generated for wily), but then the sources will be duplicated (with re-written versions) for vivid
- If a package build fails for one series, it needs to be fixed in the merge and packages in all series in the silo need to be rebuilt
- Test the generated packages on real hardware
- Please test both wily and vivid
- Mark the silo as tested
- (in most cases) Wait for silo to be QA signed-off
- Wait for the silo to be published and auto-merged
- The publish step is an all-or-none operation - either all packages are published or none
Once published, the silo merge and clean step will be blocked until all packages migrate to their destinations
Which landing method to use?
Seeing how many different scenarios and landing techniques are available, sometimes it might be confusing as to which methods to use to land your change to the right place. Currently there are two main landing areas: devel and stable. Devel is a place for all the risky changes that are not yet ready for mainstream usage while stable is what all the shipped user phones are using.
My project only has stable features
In this case, and probably in most other ones, the best way is to use Dual-landing. Your project should only have one trunk and all your merges should be targeting it.
My project had only stable features but now it will diverge
Up until now you probably used dual-landings, which is good, but now you need to release different things to devel and stable.
Currently this is a bit of a tricky situation. Normally you should now diverge your trunk into two branches: leave the current one as devel and create a stable one for stable. But we need to trick CI Train into something before this can work. Since you used trunk to dual-release, the current top-most version of the branch has a devel (currently wily) version number, while stable uses a smaller versioning number. This can cause issues in the train saying things like: cannot release a version lower than the last one.
To deal with this, after splitting and creating your stable branch, you need to modify the bzr branch by hand. You need to change the latest debian/changelog entry's version number to correspond to the currently released stable version. This can look for example like this - current version in trunk 0.3+15.10.20150601-0ubuntu1, needs to be changed to its stable counterpart, so something like: 0.3+15.04.20150601-0ubuntu1.
Once that's done, land individual changes to stable and/or devel whenever you want using the standard procedures. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project develops rapidly and only selective changes are good for stable
In this case it's best to start off by simply creating two trunks (bzr branches) from the start. Now, use standard landings to land for the devel and stable branches. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project only targets stable, but I have risky features I want to test first without breaking user experience
For such a scenario it's best to use syncs. You only have one trunk basically targeting both devel and stable, but in fact the stable versions will be always slightly behind in the archives. Simply release your risky changes to the devel version (wily currently), test them for as long as you wish (in the meantime you can release new versions with fixes to devel) and once you feel it's ready for stable - request a sync silo from devel to stable.
Note that you can use syncs and dual-landings intertwined. For instance, you can release your project using dual-landing for everything that's safe, but switch to syncs once you feel like the current features are risky. Then, once you stabilize your devel branch, you can use dual-landings once again.
Modifying your existing landing
Sometimes after a landing is prepared, a silo assigned and the packages built, during testing the lander can notice that some changes are missing and need to be added for the feature to be complete. This usually requires additional merges or sources included in the silo. CI Train handles this case, but requires certain actions.
Whenever a merge or source is added to the landing in the spreadsheet after assignment, the merge and source is not automatically noticed by CI Train. A silo reconfigure is needed first. A silo can be reconfigured by either the trainguards or the owning lander, where the latter can only reconfigure in certain cases.
A lander can reconfigure a silo on his own when:
- Only merge requests field has been changed
- No additional projects/components are added to the silo besides the ones already in use
- Or, in the case of removing a merge from the list
Example: if a silo has 3 merges for unity8 and the lander decides, after assignment, that he needs to add another unity8 merge to the list - this can be done by the lander. But if the lander decides he/she needs to add a qtmir merge to the mix, then this would involve adding a new component (qtmir) - which is not possible without the Landing Team intervention. In this case the merge can be added to the list, but only the trainguards can reconfigure it for the changes to take effect.
To reconfigure, please edit the list of merges on the CI Train spreadsheet and follow the Reconfigure Silo link present in your landing line.
In all other cases, please ping the trainguards with the details and wait for a reconfigure from his side. This is required in case the new component is already locked in a different silo or, for instance, thought to be too risky in terms of the Landing Team.
Also, an important notice for both landers and trainguards: when a silo is currently reconfigured and a package is removed from the landing, the CI Train will not remove the generated packages from the PPA automatically. Someone needs to do that manually right now after such an reconfigure.
Dealing with twin packages (-gles)
Some packages in our archives come with so-called package twins - those are usually packages that build for normally un-buildable architectures. Upstreams use those for packages such as qtubuntu and qtmir which normally do not build for desktop architectures. Therefore the *-gles packages have been created for the use by the emulator.
To keep both the normal and -gles packages in sync, the notion of twin packages has been introduced. If the CI Train detects that a silo has such a package in its landing, it will not allow it normally building before the -gles counterpart gets configured in the silo as well. Currently it's a rather troublesome process, but upstream developers invented a rather easy workflow to make it less problematic.
This scenario overviews what to do when you know your projects require twin package uploads.
- When preparing the request, besides all the standard MRs, also include all the required -gles package names as Additional source packages to land (e.g. qtubuntu = qtubuntu-gles)
- Build the silo normally
- Once the main packages build (e.g. qtubuntu), prepare sync MRs for the -gles branches in the following way:
- Target the -gles specific trunk (e.g. qtubuntu-gles)
In this branch, modify the changelog to have exactly the same version number as the main packages in the PPA (e.g. qtubuntu)
- Also, modify the debian/watch file to point to the silo where the packages were built
- Add the newly created MRs to the silo request, remove the -gles names from Additional sources
- Reconfigure the silo
- Build the -gles packages in the silo
CI Train Operation
Aborting ongoing builds
The CI Train allows aborting the build job, but only in certain cases. Once the build job is finished preparing packages and goes to the watch-ppa state, you can only now abort the build by pressing the red X button at the top of the jenkins ongoing job view.
WARNING! You can not abort the build job before the packages are prepared and uploaded to the PPA! The CI Train does not guarantee that after a build job is aborted early its state file will be correct. This can lead to the silo being completely broken, requiring a reconfigure, reassignment or manual trainguard intervention.
QA Sign-off Needed
Under certain circumstances your landing may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:
- For all landings to the stable images (vivid + overlay PPA)
- For all landings to the ubuntu-rtm branch
There is an exception to this rule if your landing contains an isolated bug fix. Right now there is not a strict definition of this term, but as guidance an isolated bug fix landing is:
- Only one or a couple of bug fixes
- A small change and not impacting a large number of sub-components of that package
- A change in a package that does not have a lot of reverse dependencies, since this makes it much harder to assess the impact of the landing.
If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the Test plans to run column of the landing spreadsheet so that QA have a good basis on which to build their additional testing. Here are some tips:
- Never just test that the bug(s) alone are fixed - remember the potential for regressions
- Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the silo is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
Take into consideration the dependencies and reverse dependencies involved - ask yourself, who could I break? Ideally your package's test plan should have some guidance on this (for example see the Dependents/Clients section media-hub's test plan
- If you only provide a link to the test plan, then bear in mind that QA will assume you've run all the tests in that plan and they have passed. If tests then fail for a reason you haven't mentioned then it will cause confusion and delay the landing, so make sure to record all cases where you have not run a test or a test has failed (and why).
Lastly, if you don't use the citrain tool for installing the silo then you need to mention what steps you used to install the silo, since citrain is the assumed way.
Isolated bugfixes for stable - confirmation
Whenever a landing is targeting a stable branch and is marked by the lander as not requiring QA sign-off because of an isolated bugfix, the Landing Team tries to verify if this indeed falls into this category. The Landing Team either by themselves or with the help of the QA team look at the landing description, looking for any rationale for this exception. Merge requests or diffs are also examined in case that's needed.
The best practice is to directly ping the QA team with a question if your change can go through without a full QA sign-off. The QA Team is here to ensure that things landing into stable are as safe as possible.