|Deletions are marked like this.||Additions are marked like this.|
|Line 242:||Line 242:|
|This is a use case that is very poorly supported by the train. During the 'preparing packages' phase, all silo state for the packages you've selected is wiped out, and recreated. If you cancel the build job during this phase, the silo will be left in an inconsistent/broken state, meaning that the packages that are in the PPA will be orphaned, and there will be no way to publish this silo as the necessary branches to merge to trunk will be missing. Don't ever cancel a build in this state, it's a bad idea, and you'll be forced to rebuild again anyway.
'''WARNING!''' If you cancel a build job while the silo status is '''Preparing packages''' the silo will be corrupted and you'll have to rebuild everything.
|Cancelling the build job is something that is now supported but there is a catch. You can cancel the job at any point during the source package build, and the train will preserve the previous package builds so that, eg, the train's local state will match what is in the PPA, and everything will be happy. However, at the point that the train pushes branches to launchpad (when you see '''INFO Pushing foo-package to lp:~/foo-package/foo-package-ubuntu-xenial-landing-NNN.'''), that is when the train discards it's previous state and overwrites it with the new state. If you cancel a job after that point, the train will be left in an inconsistent state and it will require a rebuild to fix. However cancelling a job before this point is totally harmless and you shouldn't hesitate to do so if you've made a mistake and want to start over.|
Landing changes through the CI Train
Hello lander! If you got here, you're probably looking for information about how to use the CI Train or what to do to get your project landed to Ubuntu Touch. For general information about the usage of the train, make your way to the Landing your change to Ubuntu section below. If you already know the basics of the train and just want to know which of the tools and processes to use for your specific landing, the Which landing method to use? section should help you.
Landing your change to Ubuntu
The train provides two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is maintained by Canonical.
Whenever you want to request a release of your component to the archive:
If this is your first time running a landing, ask the trainguards in #ubuntu-ci-eng to give you the required permissions for operating the CI Train.
If this is the first time this source package has ever been released in Ubuntu, refer to the Inline Packaging guide to bring your packaging up to the train's expectations.
Open the CI Train requests page.
Sign in with SSO, click Create New Request, and you'll be presented with a form to fill out.
- Description of landing: write here an overview of what your landing is about - is it fixing bugs? Adding some features? Try being informative without being over-verbose.
Test plans to run: each project handled by CI Train needs to have a test plan that documents how you, the lander, intend to test the given component(s). It's best to present this in the form of a wiki page or other document that can be referred to outside the train. See #QA for more details.
Merge proposals to land: here you list all the links to the merge proposals you want to land in this landing. Please note, this must be a newline-separated list of MP links, not branch links. More on this later.
- Manual Download URLs: This optional field contains click packages / tarball updates that don't require a PPA for testing.
- Landers: List of people who care about this landing. These names will be pinged on IRC when silo status changes, so use IRC nicks if possible (the field defaults to containing your launchpad nickname, I recommend setting an IRC highlight for your lp nick if it's different from your IRC nick).
- Sync Source: This is a special field used only when you are copying packages from one series to another. Hover your mouse over the field to see some example syntax.
Destination PPA: select or type in the PPA to which the packages should be released instead of the main archive - currently mostly used only for cases when landing to the stable overlay PPA is needed. See Landing to the Stable Overlay PPA. If you are doing a touch-specific landing for vivid, you'll want to specify ci-train-ppa-service/ubuntu/stable-phone-overlay.
- Source Package Names: here you list the 'additional' source packages that you would like to upload to the silo PPA besides what is already handled by MPs. Can be used for projects that Canonical is not upstream for or which have no bzr trunk.
- Target distribution: select which distribution you want to land into. Currently only ubuntu is supported but if ubuntu-rtm ever makes a comeback you'd select that here.
Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select Dual-Landing.
QA signoff status: the rule is: any stable-phone targets like vivid or 'xenial+vivid' landings by default require it. See #QA for more information. Set to either 'No QA Needed' or 'QA Required' to start. See QA Signoff Status field for more information.
- Once the request is filled out and you've clicked 'Save', you can assign the silo in jenkins by clicking 'Assign' from the menu that hovers beneath your row when you mouse over it, and then 'Build' on the resulting jenkins job.
Once your silo is assigned, the hover menu will grow three new links, 'Build', 'Publish', and 'Merge&Clean'. Use the 'Build' button to start building packages from the specified merge requests.
- If you have added some 'Source Package Names', you need to provide the trainguards or Ubuntu core-dev with the required source package so that they can upload those to your silo.
After the packages finish building, it is now your responsibility to test the packages from the PPA on a real device. More information about this is in the #QA section. Remember: you are responsible not only to test that the fix works, but also to test that your change didn't break anything else. Use the Test Plan.
- If your packages pass your testing criteria, mark the silo as tested:
- Get all your MP's top-approved. If you don't use the feature in your team's processes, just mark it as top-approved yourself.
- Hover over your landing request and click 'Edit' in the hover menu.
Set the QA Signoff Status dropdown to either 'Publish without QA' or 'Ready for QA' as appropriate (generally speaking if your silo only targets the ubuntu devel release, you don't need QA, but if you target the stable release or 'dual' then you do need QA). See QA Signoff Status field for more information.
- In case your silo needs QA sign-off, wait for the QA Team to finish testing on the silo. This will be indicated in the requests and the dashboard.
The landing team will now publish the silo to the archive if everything is OK packaging-wise. Note: Merge Proposals need to be top-approved latest at this point, otherwise landing team cannot publish the silo.
This is the standard process. When you prepare a list of MPs to land and provide them to CI Train, the infrastructure will merge all of them together and build a source package using bzr bd. This package is then uploaded to the silo PPA that has been assigned to your landing. CI Train also automatically generates the debian/changelog for you using the MPs' commit-messages. Every landing creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time. After all changes land into the archive, during the 'Merge & Clean' step these merges are all finally merged into trunk.
So, the general steps in the landing process from the lander's perspective are:
- Fill in a landing request
- Assign it yourself
- Build packages in silo PPA
- Test packages from the silo PPA on real hardware
- Mark silo as tested
- (in some cases) Wait for silo to be QA signed-off
- Publish the silo yourself
The QA Signoff Status field
The current Bileto implementation has one field to indicate the overall sign-off status of a landing. This means that information of if the silo needs QA sign-off or not is in the same place as the information if the silo has been tested by the lander and is ready to publish. This field is a drop-down list named 'QA Signoff Status'. Possible statuses:
No QA Needed - the silo is not tested yet by the lander, but once it is, it does not require additional QA sign-off.
Publish without QA - the silo that did not require QA sign-off is now tested by the lander and is ready to be published.
QA Required - the silo is not tested yet by the lander and once it is, it will require an additional QA sign-off.
Ready for QA - the silo is now tested by the lander and is ready for QA sign-off. Please note that only now the silo appears on the QA queue.
QA Granted - the silo has passed both the lander testing and QA sign-off and is ready to publish.
QA Failed - the silo failed QA sign-off.
CI Train for manual source uploads
Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released as a silo: either because of the QA sign-off requirement for touch, or simply because they're needed by a landing, mixed-in with normal MP-based requests.
In this case, the "Source Package Names" field needs to be used.
- In your request, put the name of the source package in the Source Package Names field (space-separated)
- Fill in the rest of the request as normal
- Remember you can mix merge-proposals and source uploads
- Silo gets assigned
- Prepare the source packages locally
- Get the package uploaded to the PPA:
- If you're a core-dev: simply use dput to upload the package to the silo PPA
- In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you (one option is to dput to your personal PPA and then core-dev or trainguard can copy the package from there).
Proceed normally with building & testing
Landing your change to the Stable Overlay PPA
The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. In other words, it's the replacement for the ubuntu-rtm/14.09 series. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.
How this works is: all current devel builds (and soon all stable builds) are built from vivid + the stable overlay PPA. This means we get all the goodness from vivid-updates and vivid-security, while we keep the dynamics of publishing touch packages for our purposes without having to go through the SRU process.
By default, whenever a landing request is set for vivid and the packages affected (or it's obvious from the landing description) are touch-specific, you should set ci-train-ppa-service/ubuntu/stable-phone-overlay as the value for the 'Destination PPA' field. This means that once the package is published it will be copied to the overlay PPA instead of migrating to the vivid archive.
For a dual silo such as xenial+vivid, do NOT set the destination PPA field. The vivid half of a dual always goes to the stable overlay PPA, so the destination field only controls where xenial goes, and that you do not want in the overlay PPA.
Syncing from and to the Stable Overlay PPA
A "sync" silo should only be used if you want to copy an existing package between xenial and the overlay ppa (in either direction). If you have a single package that you want to land in both targets, please use Dual-Landings.
WARNING! Sync silos should only be used for packages that are released through the train and include the train-generated version number. For packages that are either released through train source uploads or standard archive-uploads, please copy the packages manually with version numbers changed. See the rationale for this in citrain/SyncSilos.
Currently syncing can work in both ways - from vivid to xenial and the other way around. The recommended way is to first land your change to xenial and then syncing it back to vivid-overlay - as per the standard procedure of first developing for devel. This can be done in a few ways. The recommended one is:
- Request two silos - one for vivid-overlay and one for xenial
- Put your merges to the xenial silo
- For the second request only put the silo number of the first request in the 'sync' field.
- A trainguard will first assign the main silo and then configure the second silo to fetch (sync) source packages from the first one
- Build your main silo
- Test your main silo
- Once you're happy with your main silo, go to the secondary one and press build
- The build job will sync packages from the first silo and rebuild them for the new series
- Test your secondary silo
- If there are issues with the silo on the other platform, you will have to rebuild both silos and re-test again
- Once you are happy with both silos, set both of them as ready
Another possible way, but less safe as one might forget about syncing their work to the other archive, is this:
- Perform a normal landing to xenial and complete it the normal way.
- Request a new silo, a sync silo - no merges, only 'ubuntu,xenial' in the sync field and the names of the source packages you want synced in the "sources" field.
- Build and test the silo
- Mark as tested and wait for QA sign-off
All sync possibilities are briefly overviewed on citrain/SyncSilos.
Dual-landing for stable and devel
The current development series is xenial. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep xenial in sync, the CI Train offers functionality for performing dual landings to both overlay and xenial through one silo. This is called dual-landing.
NOTICE! Remember that all dual landings are required to go through the QA sign-off process because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your silo won't be released as a dual landing. What the trainguards can do for you is release the xenial version and then request you to sync your work back to the vivid stable-overlay once the gates are open.
- Fill in a standard landing request
- The filled merges need to point to your trunk (devel) branch, as the merges will be built against xenial and then duplicated to vivid
- In the Target Series drop-down select "xenial+vivid"
- Wait for silo assignment
- Once assigned, build the silo
- The build job will create 2 source packages - one for xenial, one for vivid - and upload them to the silo PPA
- The primary target is always xenial
- Your merges will be built for xenial (with the changelog entries generated for xenial), but then the sources will be duplicated (with re-written versions) for vivid
- If a package build fails for one series, it needs to be fixed in the merge and packages in all series in the silo need to be rebuilt
- Test the generated packages on real hardware
- Please test both xenial and vivid
- Mark the silo as tested
- Wait for silo to be QA signed-off
- Wait for the silo to be published and auto-merged
Once published, the silo merge and clean step will be blocked until all packages migrate to their destinations
Dual-landing and handling symbols
Because of the gcc5 C++ ABI break, symbols file maintenance has become more difficult. The compiler generates differently-mangled names for the old and new C++ ABI, so it is not possible to use a single symbols file for both 4.9 and 5.x ABIs.
Rather than maintaining two symbols files, it is better to maintain a symbols file only for Vivid and to use a shlibs file for xenial. Nothing is lost with this approach because the exact same code is compiled for both distros. Therefore, any difference in symbols files between 4.9 and 5.x is due only to changes in the compiler (name mangling and inlining). Or, to spell it out more clearly: a symbols file for xenial cannot detect any ABI breaks that won't also be detected by the symbols file for Vivid.
A workable way to keep a single source tree for both Vivid and xenial is to generate the symbols file for Vivid, and to generate a shlibs file for xenial during the build. (Also see below--it is possible to ditch the symbols files altogether.) In addition, you need to generate all the other debian files, such as control, *.click-hook, and the various *.install files. The hook you need is in debian/rules:
override_dh_auto_clean: /bin/sh $(CURDIR)/debian/gen-debian-files.sh $(CURDIR)/debian dh_auto_clean
This calls a shell script that generates the files depending on the distro. It's the usual hackery with sed and the like, but it gets the job done quite effectively.
Check out the script here.
The script reads the current version number from a VERSION file in the debian directory. The version file contains something like 1.0.1. (By definition, we are ABI compatible if the minor and major version number stay the same---your rules may be different.) The script then parses out the three version components and derives an soversion number from that. (Our CMakeLists.txt reads the version info from the same VERSION file so, when the version number changes, we change it once in that file and everything else for the build and the packaging adjusts automagically. (For example, we use this to correctly set the soname when we build a shared library, and substitute the soversion into the .pc file for pkgconfig.)
The script uses sed to munge up the various debian files. For example, control is generated from control.in. The control.in file contains substitution markers that sed looks for. For example:
This then turns into libunity-scopes3 on Vivid, and libunity-scopes1.0 on xenial.
Generating the other files uses similar tricks to get the correct xenial or vivid version and soname into the right spots.
For unity-scopes-api, we have ditched symbols files altogether and use abi-compliance-checker instead. While the symbols file check is better than nothing, it does not detect many ABI breaks, such as changing the return type of a function, re-ordering struct members, changing an enumeration, making a non-virtual method virtual (plus many, many others). abi-compliance-checker does a much better job at this and provides a far better assurance that the ABI is indeed still intact.
ABI compliance testing is implemented as normal unit test, so it runs every time we build on Jenkins or in a silo. You can find the relevant scripts here. The test requires a baseline ABI dump for each architecture. One easy way to get at these files is to make a temporary silo that generates the ABI dump and adds it to the binary package. Once the silo is built, you can grab the binary package for each architecture and check it into the source tree for future use, and then throw the silo away. (Remember to adjust the test to use the checked-in ABI dumps to compare against the current build.) Note that you only need to include those headers for the ABI test that are actually published for your package.
ABI compliance checking is a little slow. (For unity-scopes-api, which is large, it adds about 20 minutes to our total build time in Jenkins on Arm.) For most packages, that time will be much less. The added safety and no need to maintain symbols files any longer are well worth the wait though. Note that, when you do decide to break ABI, you will need to re-generate the new baseline ABI dumps and add them to the source tree.
Which landing method to use?
Seeing how many different scenarios and landing techniques are available, sometimes it might be confusing as to which methods to use to land your change to the right place. Currently there are two main landing areas: devel and stable. Devel is a place for all the risky changes that are not yet ready for mainstream usage while stable is what all the shipped user phones are using.
My project only has stable features
In this case, and probably in most other ones, the best way is to use Dual-landing. Your project should only have one trunk and all your merges should be targeting it.
My project had only stable features but now it will diverge
Up until now you probably used dual-landings, which is good, but now you need to release different things to devel and stable.
Currently this is a bit of a tricky situation. Normally you should now diverge your trunk into two branches: leave the current one as devel and create a stable one for stable. But we need to trick CI Train into something before this can work. Since you used trunk to dual-release, the current top-most version of the branch has a devel (currently xenial) version number, while stable uses a smaller versioning number. This can cause issues in the train saying things like: cannot release a version lower than the last one.
To deal with this, after splitting and creating your stable branch, you need to modify the bzr branch by hand. You need to change the latest debian/changelog entry's version number to correspond to the currently released stable version. This can look for example like this - current version in trunk 0.3+15.10.20150601-0ubuntu1, needs to be changed to its stable counterpart, so something like: 0.3+15.04.20150601-0ubuntu1.
Once that's done, land individual changes to stable and/or devel whenever you want using the standard procedures. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project develops rapidly and only selective changes are good for stable
In this case it's best to start off by simply creating two trunks (bzr branches) from the start. Now, use standard landings to land for the devel and stable branches. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project only targets stable, but I have risky features I want to test first without breaking user experience
For such a scenario it's best to use syncs. You only have one trunk basically targeting both devel and stable, but in fact the stable versions will be always slightly behind in the archives. Simply release your risky changes to the devel version (xenial currently), test them for as long as you wish (in the meantime you can release new versions with fixes to devel) and once you feel it's ready for stable - request a sync silo from devel to stable.
Note that you can use syncs and dual-landings intertwined. For instance, you can release your project using dual-landing for everything that's safe, but switch to syncs once you feel like the current features are risky. Then, once you stabilize your devel branch, you can use dual-landings once again.
Modifying your existing landing
For the most part, the train will notice changes to your request without requiring any special action. You can add or change MPs and the train will pick them up automatically on the next build.
Some special steps are required if you want to remove a package from your silo, however:
- You must remove the MPs from the merges field, and remove the package name from the Source Package Names field.
- You must ask a trainguard or core-dev to remove the package from the PPA.
Dealing with twin packages (-gles)
Some packages in our archives come with so-called package twins - those are usually packages that build for normally un-buildable architectures. Upstreams use those for packages such as qtubuntu and qtmir which normally do not build for desktop architectures. Therefore the *-gles packages have been created for the use by the emulator.
To keep both the normal and -gles packages in sync, the notion of twin packages has been introduced. If the CI Train detects that a silo has such a package in its landing, it will not allow it normally building before the -gles counterpart gets configured in the silo as well. Currently it's a rather troublesome process, but upstream developers invented a rather easy workflow to make it less problematic.
This scenario overviews what to do when you know your projects require twin package uploads.
- In addition to your standard MPs, prepare also an MP for the -gles twin. This MP should sync any packaging changes you made in your other MPs, but can also just be null (propose a merge from one copy of a branch to a different copy of the same branch such that the proposal has an empty diff)
- Build the silo normally.
- Train will generate matching version numbers for both of the twins and keep everything in sync with minimal hassle.
CI Train Operation
Aborting ongoing builds
Cancelling the build job is something that is now supported but there is a catch. You can cancel the job at any point during the source package build, and the train will preserve the previous package builds so that, eg, the train's local state will match what is in the PPA, and everything will be happy. However, at the point that the train pushes branches to launchpad (when you see INFO Pushing foo-package to lp:~/foo-package/foo-package-ubuntu-xenial-landing-NNN.), that is when the train discards it's previous state and overwrites it with the new state. If you cancel a job after that point, the train will be left in an inconsistent state and it will require a rebuild to fix. However cancelling a job before this point is totally harmless and you shouldn't hesitate to do so if you've made a mistake and want to start over.
Abandon a landing
In order to free a silo from jenkins and give up on the landing request, click 'Abandon' on your landing request. This opens a Jenkins job where you must click 'Build'. Once that job is complete, the silo assigned to your request (if any) will be emptied, and your request status will be set to 'Abandoned', which hides it from the main list of requests.
QA Sign-off Needed
Under certain circumstances your landing may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:
- For all landings to the stable images (vivid + overlay PPA)
There is an exception to this rule if your landing contains an isolated bug fix. Right now there is not a strict definition of this term, but as guidance an isolated bug fix landing is:
- Only one or a couple of bug fixes
- A small change and not impacting a large number of sub-components of that package
- A change in a package that does not have a lot of reverse dependencies, since this makes it much harder to assess the impact of the landing.
If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the Test plans to run column of the landing spreadsheet so that QA have a good basis on which to build their additional testing. Here are some tips:
- Never just test that the bug(s) alone are fixed - remember the potential for regressions
- Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the silo is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
Take into consideration the dependencies and reverse dependencies involved - ask yourself, who could I break? Ideally your package's test plan should have some guidance on this (for example see the Dependents/Clients section media-hub's test plan
- If you only provide a link to the test plan, then bear in mind that QA will assume you've run all the tests in that plan and they have passed. If tests then fail for a reason you haven't mentioned then it will cause confusion and delay the landing, so make sure to record all cases where you have not run a test or a test has failed (and why).
Lastly, if you don't use the citrain tool for installing the silo then you need to mention what steps you used to install the silo, since citrain is the assumed way.
Isolated bugfixes for stable - confirmation
Whenever a landing is targeting a stable branch and is marked by the lander as not requiring QA sign-off because of an isolated bugfix, the Landing Team tries to verify if this indeed falls into this category. The Landing Team either by themselves or with the help of the QA team look at the landing description, looking for any rationale for this exception. Merge requests or diffs are also examined in case that's needed.
The best practice is to directly ping the QA team with a question if your change can go through without a full QA sign-off. The QA Team is here to ensure that things landing into stable are as safe as possible.
Interpreting Silo Statuses
CI Train monitors silo statuses at a per-package, per-arch level, which can lead to some overwhelmingly detailed status messages. Fortunately, in many common cases, irrelevant details are omitted for clarity. Here are some examples:
Successfully built or Failed to build
- When there is no package list or arch list, it means that all packages on all arches have this same status.
Failed to build (foo-package/xenial). Successfully built (bar-package/xenial, grill-package/xenial).
- When there are multiple sentences with packages in brackets, the message applies to the packages named in the brackets. So in this example foo-package failed to build while the other two succeeded.
Failed to build [amd64 i386] Successfully built [armhf] (foo-package/xenial).
- If a package failed to build on some arches but succeeded on other arches, that will be indicated with the arches in square brackets like this.
Proposed pocket (foo-package/xenial). Release pocket (foo-package/vivid).
This means that the silo has been published but has not yet made it all the way to Ubuntu archive yet. When publishing a dual silo, the vivid package is copied immediately to the overlay PPA, so it will indicate "Release pocket" (the fact that it's the release pocket of the overlay PPA and not the release pocket of the vivid archive is implied by the fact that it's a dual silo. dual silos never target vivid archive). Once the xenial package finishes migrating through Proposed and into Release, the train will notice this and auto-merge your merges for you.
Needs rebuild due to new commits (foo-package/xenial). Successfully built (foo-package/vivid).
- That's weird! Why doesn't the vivid one reflect that it also needs to be rebuilt to match the xenial one? Well the class that implements the vivid half of dual silos does not have any knowledge of MPs or commits, so when the xenial half notices it has unbuilt commits, the vivid half has no way to also check that. So this is the expected status when a dual silo has new commits. Rest assured that when you rebuild the package, both xenial and vivid packages will be rebuilt, the train cannot build one without the other so they can never get out of sync.
Here are some explanations for common status messages:
Ready to build
- No package is in the PPA, mostly likely because the silo is newly assigned, but also possibly because the packages have been deleted.
[VERSION] not in PPA
- The version number in the local source tree does not match the version number in the silo PPA. This is most likely caused by a source package build failure (such as a merge conflict) which prevented the package from being uploaded, but can also be caused by upload failures (in certain conditions launchpad will reject an upload after the jenkins log indicates that the package has been successfully uploaded).
- This happens if you have merges that are either Rejected or Superceded. Click the "Status" link to see which merges are which.
Needs rebuild due to new commits
- The Train has detected new commits on either the source or target branches of your merges. Click the "Status" link to see the list of where new commits have been detected. Note that this message won't appear if the only new commits are Launchpad Translation auto-commits.
Needs rebuild due to burned version number
This happens when the same source package is built in two different silos on the same day, such that they both have the same version number, and one of those silos has been published. The other silo sees that the destination archive has a matching version number, but the package has differing contents, so the current version number is now "burned". Most likely this message will shortly be followed by the new commits message after the other silo merges, so take this as a "pre-warning" that you'll eventually need to rebuild.
- This silo has been published and is on it's way to the archive! The silo will automatically merge once all packages are found to be in either the Release or Updates pocket. It is safe to manually merge when packages are in the Proposed pocket, however this is discouraged because you most likely want to do this when something is stuck in Proposed, and force-merging is just ignoring the problem. Much better to fix the problem and let the package migrate normally.
If your package has never been released in ubuntu before, it will find itself in the NEW queue, and if the archive is frozen (or you're doing an SRU), it will be in the UNAPPROVED queue. This is not something that will migrate on it's own and does require manual intervention to prod along, so be sure to ask for help if you see this state and you're not sure about it. WARNING: it is not safe to merge the silo in this state! If you merge, the packages will be lost and your release will never make it to the archive.
Destination version [VERSION] is missing from changelog
The current version number in the destination archive (either ubuntu or overlay PPA depending on how your request is configured) could not be found in the debian/changelog in the local tree. This can either be caused by manual uploads, or by conflicting train silos that have already been published. You'll need to investigate the destination archive and see what version is there. If it's a manual upload, you'll need to sync it back to trunk manually, otherwise if it's another train upload, you'll need to wait for it to merge to trunk normally, then rebuild your silo to incorporate those changes.
Failed to build
- Your package was uploaded to the PPA but did not build successfully! Click the "Status" link to see the full list of what packages/arches failed and links to the failure logs.
- This status implies that your packages built successfully, but the expected debdiffs could not be found. If you see this status on a package built from merges, this is a bug and you should report it immediately! The train should always automatically generate diffs for MP builds. If the package is a manual source, this simply means that you uploaded the package to the PPA but didn't yet run the build job to generate the diff. Just run that build job!
- Absolutely no problems could be (automatically) found with your package build. Congrats! Test it out and publish it!
All CI Train silo status changes are pinged in #ubuntu-ci-eng on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about. As a convenience, when making a new request, Bileto will insert your launchpad nick into the landers field. Unfortunately, Bileto doesn't know your IRC nick though, only launchpad nick, so if those differ, the easiest thing for you to do might be to set an IRC highlight for your launchpad nick. Then when the bot pings the status message in the channel you'll see the highlight. Either that, or just make sure to change your launchpad nick to your IRC nick every time you make a landing request.
By default, the train will generate a debian/changelog entry for you, based on the text you enter into the "Commit Message" field of the merge proposals in the silo. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series UNRELEASED so that the train can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, the train will make a new empty changelog message which isn't what you want.
Installing a PPA onto an Ubuntu Touch device is a bit of a chore because you have to set the device writable, pin the silo PPA higher than the overlay PPA, etc. Fortunately there is a tool to automate this.
$ sudo add-apt-repository ppa:phablet-team/ubuntu/tools $ sudo apt-get install phablet-tools-citrain $ citrain help
How to Coordinate the Publishing of Multiple Large Landings
Sometimes, there will be a few large silos wanting to publish around the same time, eg mir and unity8. When this happens, best practices is to do an image build in between publishings, because this makes it easier to bisect where regressions came from. It's a little bit tricky however, because certain tools report information differently and there's a race condition that you can fail if you're not careful. Best to follow these steps:
1. Publish one of the silos.
2. Wait for it to migrate. Eventually, queuebot will ping that the request is Landed.
3. It's not yet safe to kick the image build, however. You must now use rmadison to confirm that the packages are truly out of the -proposed pocket and in the release pocket.
4. Once rmadison confirms publication has completed, now kick the image build.
5. Wait until the image build has completed, to ensure the build succeeds and doesn't need to be retried or fixed.
6. Now publish the second silo.
Renaming a Source Package
Are you trying to rename a source package and getting weird errors? Don't do that!
Just kidding, but here's what's going on. In jenkins, there's a function that looks up bzr branches and fetches the source package name from the branch. It does this by downloading the debian/changelog, and then throwing away everything except the very first word, which is the source package name. This is obviously hilariously slow and inefficient, so a disk cache was implemented, and the cache has no expiry logic, so you can never, ever rename your source packages, ever. In the event that you do need to rename your source package, you'll need to ask one of the trainguards to manually flush the cache for you, which can be done by deleting /tmp/cu2d_cache/ on the jenkins instance.
[source_package_name] [version] is missing from the changelog, which has up to [lesser_version]. Please sync destination version back to trunk.
This happens when an ubuntu core developer makes a manual upload of your package to ubuntu without using ci train. 99% of the time this would be no-change rebuilds during library transitions, or other trivial fixes for packaging issues. Before you can build the silo, it is necessary to sync the manual distro upload back to your trunk, so that they match. The first thing to do is check the launchpad page for your source package (for example here's the page for unity8. On that page look for a version number that ends in "0ubuntu2" or similar (the train only ever generates versions ending in "0ubuntu1"). Click through that page to get to the "available diffs" for that version, download that diff, and then commit that directly to your trunk. Do not make a merge proposal with this diff and include it in the silo, that does not work and is a backwards approach. You want to copy what's in distro into your trunk, silos are for getting branches into distro.
You typed a package name into PACKAGES_TO_REBUILD that wasn't part of the silo configuration. You should double check your spelling, and if it's correct, then your request is misconfigured.