Landing changes through the CI Train
Hello lander! If you got here, you're probably looking for information about how to use the CI Train or what to do to get your project landed to Ubuntu Touch. For general information about the usage of the train, make your way to the Landing your change to Ubuntu section below. If you already know the basics of the train and just want to know which of the tools and processes to use for your specific landing, the Which landing method to use? section should help you.
Landing your change to Ubuntu
The train provides two options for landing packages. You can either land your changes into the archive through listing merge proposals that you want to release, or by uploading source packages directly to the ppa. The first approach is recommended in all cases where the source package's upstream branch is maintained by Canonical.
Whenever you want to request a release of your component to the archive:
If this is your first time running a landing, ask the trainguards in #ubuntu-ci-eng to give you the required permissions for operating the CI Train.
If this is the first time this source package has ever been released with the train, refer to the Inline Packaging guide to bring your packaging up to the train's expectations.
Open the CI Train requests page.
Sign in with SSO, click Create New Request, and you'll be presented with a form to fill out.
- Description of landing: write here an overview of what your landing is about - is it fixing bugs? Adding some features? Try being informative without being over-verbose.
Test plans to run: each project handled by CI Train needs to have a test plan that documents how you, the lander, intend to test the given component(s). It's best to present this in the form of a wiki page or other document that can be referred to outside the train. See #QA for more details.
Merge proposals to land: here you list all the links to the merge proposals you want to land in this landing. Please note, this must be a newline-separated list of MP links, not branch links. More on this later.
- Manual Download URLs: This optional field contains click packages / device tarball updates that don't require a PPA for testing.
- Landers: List of people who care about this landing. These names will be pinged on IRC when silo status changes, so use IRC nicks if possible (the field defaults to containing your launchpad nickname, I recommend setting an IRC highlight for your lp nick if it's different from your IRC nick).
- Sync Source: This is a special field used only when you are copying packages from one series to another. Hover your mouse over the field to see some example syntax.
Destination PPA: select or type in the PPA to which the packages should be released instead of the main archive - currently mostly used only for cases when landing to the stable overlay PPA is needed. See Landing to the Stable Overlay PPA. If you are doing a touch-specific landing for vivid, you'll want to specify ci-train-ppa-service/ubuntu/stable-phone-overlay.
- Source Package Names: You can leave this blank and the train will auto-fill the value based on the contents of the PPA.
- Target distribution: select which distribution you want to land into. Currently only ubuntu is supported but if ubuntu-rtm ever makes a comeback you'd select that here.
Target series: Which ubuntu release you want to build for. If you are requesting a touch landing which is developed on one trunk (i.e. everything that lands in devel is to land in stable), please select xenial+vivid.
QA signoff: Values "N/A", "Required", or "Ready" will be automatically set as appropriate for your landing. The rule is: any stable-phone targets like vivid or 'xenial+vivid' landings by default require it. See #QA for more information.
- Once the request is filled out and you've clicked 'Create New Request', you can assign the silo in jenkins by clicking 'Assign' from the menu at the bottom of your new ticket, and then 'Build' on the jenkins job that opens.
Once your silo is assigned, the menu will grow three new links, 'Build', 'Publish', and 'Merge&Clean'. Use the 'Build' button to start building packages from your merge requests.
- If you have manual packages not built from merges, you need a trainguard or Ubuntu core-dev to upload those to your silo.
After the packages finish building, it is now your responsibility to test the packages from the PPA on a real device. More information about this is in the #QA section. Remember: you are responsible not only to test that the fix works, but also to test that your change didn't break anything else. Use the Test Plan.
- If your packages pass your testing criteria, mark the silo as tested:
- Get all your MP's top-approved. If you don't use the feature in your team's processes, just mark it as top-approved yourself.
- Set 'Lander Signoff' to 'Approved'.
- Britney will now automatically inspect your silo and perform various installability checks and autopkgtests. Once britney approves your silo, it will automatically be marked ready for QA, if appropriate.
- If your ticket says "QA Signoff: Ready" you must wait for a QA person to approve your silo.
- Once your silo has been approved by both Britney and QA (or if QA is N/A), you can publish your own silo. Landers are permitted to publish their own silos so long as there are no packaging changes, and no manual sources in the silo. In the event that you do have either packaging changes or manual sources, it is your responsibility to find a core dev to publish the silo for you.
This is the standard process. When you prepare a list of MPs to land and provide them to CI Train, the infrastructure will merge all of them together and build a source package using bzr bd. This package is then uploaded to the silo PPA that has been assigned to your landing. CI Train also automatically generates the debian/changelog for you using the MPs' commit-messages. Every landing creates a new version in the changelog with one entry per merge. The version number is also auto-generated every time. After all changes land into the archive, these merges are automatically merged into trunk.
So, the general steps in the landing process from the lander's perspective are:
- Fill in a landing request
- Assign it yourself
- Build packages in silo PPA
- Test packages from the silo PPA on real hardware
- Mark silo as Lander Signoff: Approved
- Wait for Britney to mark the request Approved.
- (in some cases) Wait for QA to mark the silo Approved
- Publish the silo yourself
- (in some cases) Find a core dev to ACK packaging changes and publish the silo for you
CI Train for manual source uploads
Sometimes not everything can be dealt with using merge requests. Some packages (usually those Canonical is not upstream for) don't have their development happening on LP but they might be needed to be released as a silo: either because of the QA sign-off requirement for touch, or simply because they're needed by a landing, mixed-in with normal MP-based requests.
- Prepare the source packages locally
- Get the package uploaded to the PPA:
- If you're a core-dev: simply use dput to upload the package to the silo PPA
- In other cases, send the source package to a core-dev or trainguard with a request to upload to the PPA for you (one option is to dput to your personal PPA and then core-dev or trainguard can copy the package from there).
Proceed normally with building & testing
Landing your change to the Stable Overlay PPA
The stable overlay PPA is the place where all the stable development is happening for Ubuntu Touch. In other words, it's the replacement for the ubuntu-rtm/14.09 series. The rationale for it is that this time, learning by experience, we want to base our stable branch out off a released series - in this case: vivid.
How this works is: all current devel builds (and soon all stable builds) are built from vivid + the stable overlay PPA. This means we get all the goodness from vivid-updates and vivid-security, while we keep the dynamics of publishing touch packages for our purposes without having to go through the SRU process.
By default, whenever a landing request is set for vivid and the packages affected (or it's obvious from the landing description) are touch-specific, you must set ci-train-ppa-service/ubuntu/stable-phone-overlay as the value for the 'Destination PPA' field. This means that once the package is published it will be copied to the overlay PPA instead of migrating to the vivid archive (which requires an SRU).
For a dual silo such as xenial+vivid, the destination PPA field is ignored. The vivid half of a dual always goes to the stable overlay PPA, and the xenial half never goes there.
Syncing from and to the Stable Overlay PPA
A "sync" silo should only be used if you want to copy an existing package between xenial and the overlay ppa (in either direction). If you have a single package that you want to land in both targets, please use Dual-Landings.
WARNING! Sync silos should only be used for packages that are released through the train and include the train-generated version number. For packages that are either released through train source uploads or standard archive-uploads, please copy the packages manually with version numbers changed. See the rationale for this in citrain/SyncSilos.
Currently syncing can work in both ways - from vivid to xenial and the other way around. The recommended way is to first land your change to xenial and then syncing it back to vivid-overlay - as per the standard procedure of first developing for devel. This can be done in a few ways. The recommended one is:
- Request two silos - one for vivid-overlay and one for xenial
- Put your merges to the xenial silo
- For the second request only put the silo number of the first request in the 'sync' field.
- First assign the main silo and then configure the second silo to sync source packages from the first one
- Build your main silo
- Test your main silo
- Once you're happy with your main silo, go to the secondary one and press build
- The build job will sync packages from the first silo and rebuild them for the new series
- Test your secondary silo
- If there are issues with the silo on the other platform, you will have to rebuild both silos and re-test again
- Once you are happy with both silos, set both of them as Lander Signoff: Approved
Another possible way, but less safe as one might forget about syncing their work to the other archive, is this:
- Perform a normal landing to xenial and complete it the normal way.
- Request a new silo, a sync silo - no merges, only 'ubuntu,xenial' in the sync field and the names of the source packages you want synced in the "sources" field.
- Build and test the silo
- Mark as Approved and wait for QA sign-off
All sync possibilities are briefly overviewed on citrain/SyncSilos.
Dual-landing for stable and devel
The current development series is xenial. As per the current guidelines, all touch development will happen on the vivid overlay PPA. But to keep xenial in sync, the CI Train offers functionality for performing dual landings to both overlay and xenial through one silo. This is called dual-landing.
NOTICE! Remember that all dual landings are required to go through the QA sign-off process because of their stable-overlay nature. This also means that if the stable-overlay gates are closed for a milestone, your silo won't be released as a dual landing. What the trainguards can do for you is release the xenial version and then request you to sync your work back to the vivid stable-overlay once the gates are open.
- Fill in a standard landing request
- The filled merges need to point to your trunk (devel) branch, as the merges will be built against xenial and then duplicated to vivid
- In the Target Series drop-down select "xenial+vivid"
- Assign your own silo
- Build the silo
- The build job will create 2 source packages - one for xenial, one for vivid - and upload them to the silo PPA
- The primary target is always xenial
- Your merges will be built for xenial (with the changelog entries generated for xenial), but then the sources will be duplicated (with re-written versions) for vivid
- If a package build fails for one series, it needs to be fixed in the merge and packages in all series in the silo need to be rebuilt
- Test the generated packages on real hardware
- Please test both xenial and vivid
- Set Lander Signoff: Approved
- Wait for Britney to approve the silo
- Wait for QA to approve the silo
- Publish it yourself once everything is approved.
Dual-landing and handling symbols
Because of the gcc5 C++ ABI break, symbols file maintenance has become more difficult. The compiler generates differently-mangled names for the old and new C++ ABI, so it is not possible to use a single symbols file for both 4.9 and 5.x ABIs.
Rather than maintaining two symbols files, it is better to maintain a symbols file only for Vivid and to use a shlibs file for xenial. Nothing is lost with this approach because the exact same code is compiled for both distros. Therefore, any difference in symbols files between 4.9 and 5.x is due only to changes in the compiler (name mangling and inlining). Or, to spell it out more clearly: a symbols file for xenial cannot detect any ABI breaks that won't also be detected by the symbols file for Vivid.
A workable way to keep a single source tree for both Vivid and xenial is to generate the symbols file for Vivid, and to generate a shlibs file for xenial during the build. (Also see below--it is possible to ditch the symbols files altogether.) In addition, you need to generate all the other debian files, such as control, *.click-hook, and the various *.install files. The hook you need is in debian/rules:
override_dh_auto_clean: /bin/sh $(CURDIR)/debian/gen-debian-files.sh $(CURDIR)/debian dh_auto_clean
This calls a shell script that generates the files depending on the distro. It's the usual hackery with sed and the like, but it gets the job done quite effectively.
The script reads the current version number from a VERSION file in the debian directory. The version file contains something like 1.0.1. (By definition, we are ABI compatible if the minor and major version number stay the same---your rules may be different.) The script then parses out the three version components and derives an soversion number from that. (Our CMakeLists.txt reads the version info from the same VERSION file so, when the version number changes, we change it once in that file and everything else for the build and the packaging adjusts automagically. (For example, we use this to correctly set the soname when we build a shared library, and substitute the soversion into the .pc file for pkgconfig.)
The script uses sed to munge up the various debian files. For example, control is generated from control.in. The control.in file contains substitution markers that sed looks for. For example:
This then turns into libunity-scopes3 on Vivid, and libunity-scopes1.0 on xenial.
Generating the other files uses similar tricks to get the correct xenial or vivid version and soname into the right spots.
For unity-scopes-api, we have ditched symbols files altogether and use abigail instead. While the symbols file check is better than nothing, it does not detect many ABI breaks, such as changing the return type of a function, re-ordering struct members, changing an enumeration, making a non-virtual method virtual (plus many, many others). abigail does a much better job at this and provides a far better assurance that the ABI is indeed still intact.
ABI compliance testing is implemented as normal unit test, so it runs every time we build on Jenkins or in a silo. You can find the relevant scripts here. The test requires a baseline ABI dump for each architecture. One easy way to get at these files is to make a temporary silo that generates the ABI dump and adds it to the binary package. Once the silo is built, you can grab the binary package for each architecture, pull out the dump file, check that into the source tree for future use, and then throw the silo away.
ABI compliance checking is fast enough to tolerate. (For unity-scopes-api, which is large, it adds about 6 minutes to our total build time in Jenkins on Arm.) For most packages, that time will be much less. The added safety and no need to maintain symbols files any longer are well worth the wait though. Note that, when you do decide to break ABI, you will need to re-generate the new baseline ABI dumps and add them to the source tree.
Which landing method to use?
Seeing how many different scenarios and landing techniques are available, sometimes it might be confusing as to which methods to use to land your change to the right place. Currently there are two main landing areas: devel and stable. Devel is a place for all the risky changes that are not yet ready for mainstream usage while stable is what all the shipped user phones are using.
My project only has stable features
In this case, and probably in most other ones, the best way is to use Dual-landing. Your project should only have one trunk and all your merges should be targeting it.
My project had only stable features but now it will diverge
Up until now you probably used dual-landings, which is good, but now you need to release different things to devel and stable.
Currently this is a bit of a tricky situation. Normally you should now diverge your trunk into two branches: leave the current one as devel and create a stable one for stable. But we need to trick CI Train into something before this can work. Since you used trunk to dual-release, the current top-most version of the branch has a devel (currently xenial) version number, while stable uses a smaller versioning number. This can cause issues in the train saying things like: cannot release a version lower than the last one.
To deal with this, after splitting and creating your stable branch, you need to modify the bzr branch by hand. You need to change the latest debian/changelog entry's version number to correspond to the currently released stable version. This can look for example like this - current version in trunk 0.3+15.10.20150601-0ubuntu1, needs to be changed to its stable counterpart, so something like: 0.3+15.04.20150601-0ubuntu1.
Once that's done, land individual changes to stable and/or devel whenever you want using the standard procedures. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project develops rapidly and only selective changes are good for stable
In this case it's best to start off by simply creating two trunks (bzr branches) from the start. Now, use standard landings to land for the devel and stable branches. Please note that anything that lands in stable should also already be available in devel - so possibly any stable landing should be a cherry-pick from devel.
My project only targets stable, but I have risky features I want to test first without breaking user experience
For such a scenario it's best to use syncs. You only have one trunk basically targeting both devel and stable, but in fact the stable versions will be always slightly behind in the archives. Simply release your risky changes to the devel version (xenial currently), test them for as long as you wish (in the meantime you can release new versions with fixes to devel) and once you feel it's ready for stable - request a sync silo from devel to stable.
Note that you can use syncs and dual-landings intertwined. For instance, you can release your project using dual-landing for everything that's safe, but switch to syncs once you feel like the current features are risky. Then, once you stabilize your devel branch, you can use dual-landings once again.
Modifying your existing landing
For the most part, the train will notice changes to your request without requiring any special action. You can add or change MPs and the train will pick them up automatically on the next build.
Some special steps are required if you want to remove a package from your silo, however:
- You must remove the MPs from the merges field, and remove the package name from the Source Package Names field.
- You must ask a trainguard or core-dev to remove the package from the PPA.
Dealing with twin packages (-gles)
Some packages in our archives come with so-called package twins - those are usually packages that build for normally un-buildable architectures. Upstreams use those for packages such as qtubuntu and qtmir which normally do not build for desktop architectures. Therefore the *-gles packages have been created for the use by the emulator.
To keep both the normal and -gles packages in sync, the notion of twin packages has been introduced. If the CI Train detects that a silo has such a package in its landing, it will not allow it normally building before the -gles counterpart gets configured in the silo as well. Currently it's a rather troublesome process, but upstream developers invented a rather easy workflow to make it less problematic.
This scenario overviews what to do when you know your projects require twin package uploads.
- In addition to your standard MPs, prepare also an MP for the -gles twin. This MP should sync any packaging changes you made in your other MPs, but can also just be null (propose a merge from one copy of a branch to a different copy of the same branch such that the proposal has an empty diff)
- Build the silo normally.
- Train will generate matching version numbers for both of the twins and keep everything in sync with minimal hassle.
CI Train Operation
Aborting ongoing builds
WARNING Train build logs now include the message POINT OF NO RETURN after which cancelling the job will leave your silo in an inconsistent state. Fortunately this is towards the end of the build, and it is totally safe to cancel the job prior to the appearance of this message.
Abandon a landing
In order to free a silo from jenkins and give up on the landing request, click 'Abandon' on your landing request. This opens a Jenkins job where you must click 'Build'. Once that job is complete, the silo assigned to your request (if any) will be emptied, and your request status will be set to 'Abandoned', which hides it from the main list of requests.
QA Sign-off Needed
Under certain circumstances your landing may need to go through QA sign off. This is usually during situations when the quality level of the image has regressed severely or must be maintained at a high-level. Right now these are:
- For all landings to the stable images (vivid + overlay PPA)
There is an exception to this rule if your landing contains an isolated bug fix. Right now there is not a strict definition of this term, but as guidance an isolated bug fix landing is:
- Only one or a couple of bug fixes
- A small change and not impacting a large number of sub-components of that package
- A change in a package that does not have a lot of reverse dependencies, since this makes it much harder to assess the impact of the landing.
If QA sign-off is required then you will be expected to provide more detailed information about the testing you have done in the Test plans to run column of the landing spreadsheet so that QA have a good basis on which to build their additional testing. Here are some tips:
- Never just test that the bug(s) alone are fixed - remember the potential for regressions
- Look at what is actually landing and make sure testing is done for all the updated packages. For example if the primary intent of the silo is to fix music player controls, but the fix requires a media-hub update then you should probably run the whole media-hub test plan.
Take into consideration the dependencies and reverse dependencies involved - ask yourself, who could I break? Ideally your package's test plan should have some guidance on this (for example see the Dependents/Clients section media-hub's test plan
- If you only provide a link to the test plan, then bear in mind that QA will assume you've run all the tests in that plan and they have passed. If tests then fail for a reason you haven't mentioned then it will cause confusion and delay the landing, so make sure to record all cases where you have not run a test or a test has failed (and why).
Lastly, if you don't use the citrain tool for installing the silo then you need to mention what steps you used to install the silo, since citrain is the assumed way.
Isolated bugfixes for stable - confirmation
Whenever a landing is targeting a stable branch and is marked by the lander as not requiring QA sign-off because of an isolated bugfix, the Landing Team tries to verify if this indeed falls into this category. The Landing Team either by themselves or with the help of the QA team look at the landing description, looking for any rationale for this exception. Merge requests or diffs are also examined in case that's needed.
The best practice is to directly ping the QA team with a question if your change can go through without a full QA sign-off. The QA Team is here to ensure that things landing into stable are as safe as possible.
Interpreting Silo Statuses
CI Train monitors silo statuses at a per-package level, which can lead to some detailed status messages. Fortunately, in many common cases, irrelevant details are omitted for clarity. Here are some examples:
Successfully built or Failed to build
When there is no package list, it means that all packages have this same status. Error states such as Failed to build or Dependency wait will appear if that status appears on any arch, but Successfully built will appear only if it succeeded on all arches.
Failed to build (foo-package/xenial). Successfully built (bar-package/xenial, grill-package/xenial).
- When there are multiple sentences with packages/series in brackets, the message applies to the packages named in the brackets. So in this example foo-package failed to build in xenial while the other two succeeded.
Proposed pocket (foo-package/xenial). Release pocket (foo-package/vivid).
This means that the silo has been published but has not yet made it all the way to Ubuntu archive yet. When publishing a dual silo, the vivid package is copied immediately to the overlay PPA, so it will indicate "Release pocket" (the fact that it's the release pocket of the overlay PPA and not the release pocket of the vivid archive is implied by the fact that it's a dual silo. dual silos never target vivid archive). Once the xenial package finishes migrating through Proposed and into Release, the train will notice this and auto-merge your merges for you.
Needs rebuild due to new commits (foo-package/xenial). Successfully built (foo-package/vivid).
- That's weird! Why doesn't the vivid one reflect that it also needs to be rebuilt to match the xenial one? Well the class that implements the vivid half of dual silos does not have any knowledge of MPs or commits, so when the xenial half notices it has unbuilt commits, the vivid half has no way to also check that. So this is the expected status when a dual silo has new commits. Rest assured that when you rebuild the package, both xenial and vivid packages will be rebuilt, the train cannot build one without the other so they can never get out of sync.
Here are some explanations for common status messages:
Ready to build
- No package is in the PPA, mostly likely because the silo is newly assigned, but also possibly because the packages have been deleted.
PPA/bzr version mismatch
- The version number in the local source tree does not match the version number in the silo PPA. This is most likely caused by a source package upload failure (in certain conditions launchpad will reject an upload after the jenkins log indicates that the package has been successfully uploaded).
- This happens if you have merges that are either Rejected or Superceded. Click the "Status" link to see the full log, which indicates which merges need attention.
Needs rebuild due to new commits
- The Train has detected new commits on either the source or target branches of your merges. Click the "Status" link to see the list of where new commits have been detected. Note that this message won't appear if the only new commits are Launchpad Translation auto-commits.
Needs rebuild due to burned version number
This happens when the same source package is built in two different silos on the same day, such that they both have the same version number, and one of those silos has been published. The other silo sees that the destination archive has a matching version number, but the package has differing contents, so the current version number is now "burned". Most likely this message will shortly be followed by the new commits message after the other silo merges, so take this as an early warning that you'll soon need to rebuild (but not yet!).
- This silo has been published and is on it's way to the archive! The silo will automatically merge once all packages are found to be in either the Release or Updates pocket. It is safe to manually merge when packages are in the Proposed pocket, however this is discouraged because you most likely want to do this when something is stuck in Proposed, and force-merging is just ignoring the problem. Much better to fix the problem and let the package migrate normally.
If your package has never been released in ubuntu before, it will find itself in the NEW queue, and if the archive is frozen (or you're doing an SRU), it will be in the UNAPPROVED queue. This is not something that will migrate on it's own and does require manual intervention to prod along, so be sure to ask for help if you see this state and you're not sure about it. WARNING: it is not safe to merge the silo in this state! If you merge, the packages will be lost and your release will never make it to the archive.
Destination version [VERSION] is missing from changelog
The current version number in the destination archive (either ubuntu or overlay PPA depending on how your request is configured) could not be found in the debian/changelog in the local tree. This can either be caused by manual uploads, or by conflicting train silos that have already been published. You'll need to investigate the destination archive and see what version is there. If it's a manual upload, you'll need to sync it back to trunk manually, otherwise if it's another train upload, you'll need to wait for it to merge to trunk normally, then rebuild your silo to incorporate those changes.
Failed to build
- Your package was uploaded to the PPA but did not build successfully! Click the "Status" link to see the full list of what packages/arches failed and links to the failure logs.
- This status implies that your packages built successfully, but the expected debdiffs could not be found. If you see this status on a package built from merges, this is a bug and you should report it immediately! The train should always automatically generate diffs for MP builds. If the package is a manual source, this simply means that you uploaded the package to the PPA but didn't yet run the build job to generate the diff. Just run that build job!
- Absolutely no problems could be (automatically) found with your package build. Congrats! Test it out and publish it!
All CI Train silo status changes are pinged in #ubuntu-ci-eng on freenode. Messages will contain the contents of the "landers" field in the landing request, so make sure your IRC nick appears in the "landers" field of any requests that you care about. As a convenience, when making a new request, Bileto will insert your launchpad nick into the landers field. Unfortunately, Bileto doesn't know your IRC nick though, only launchpad nick, so if those differ, the easiest thing for you to do might be to set an IRC highlight for your launchpad nick. Then when the bot pings the status message in the channel you'll see the highlight. Either that, or just make sure to change your launchpad nick to your IRC nick every time you make a landing request.
By default, the train will generate a debian/changelog entry for you, based on the text you enter into the "Commit Message" field of the merge proposals in the silo. For simple changes this is fine, however there are a number of limitations and you may find yourself disappointed with the generated changelog. If so, you are free to create your own debian/changelog entry, just make sure to use series UNRELEASED so that the train can still do it's auto-version-number generation magic. If you make your own debian/changelog and use a series other than UNRELEASED, the train will make a new empty changelog message which isn't what you want.
Installing a PPA onto an Ubuntu Touch device is a bit of a chore because you have to set the device writable, pin the silo PPA higher than the overlay PPA, etc. Fortunately there is a tool to automate this.
$ sudo add-apt-repository ppa:phablet-team/ubuntu/tools $ sudo apt-get install phablet-tools-citrain $ citrain help
How to Coordinate the Publishing of Multiple Large Landings
Sometimes, there will be a few large silos wanting to publish around the same time, eg mir and unity8. When this happens, best practices is to do an image build in between publishings, because this makes it easier to bisect where regressions came from. It's a little bit tricky however, because certain tools report information differently and there's a race condition that you can fail if you're not careful. Best to follow these steps:
1. Publish one of the silos.
2. Wait for it to migrate. Eventually, queuebot will ping that the request is Landed.
3. It's not yet safe to kick the image build, however. You must now use rmadison to confirm that the packages are truly out of the -proposed pocket and in the release pocket.
4. Once rmadison confirms publication has completed, now kick the image build.
5. Wait until the image build has completed, to ensure the build succeeds and doesn't need to be retried or fixed.
6. Now publish the second silo.
Renaming a Source Package
Are you trying to rename a source package and getting weird errors? Don't do that!
Just kidding, but here's what's going on. In jenkins, there's a function that looks up bzr branches and fetches the source package name from the branch. It does this by downloading the debian/changelog, and then throwing away everything except the very first word, which is the source package name. This is obviously hilariously slow and inefficient, so a disk cache was implemented, and the cache has no expiry logic, so you can never, ever rename your source packages, ever. In the event that you do need to rename your source package, you'll need to ask one of the trainguards to manually flush the cache for you, which can be done by deleting /tmp/cu2d_cache/ on the jenkins instance.
[source_package_name] [version] is missing from the changelog, which has up to [lesser_version]. Please sync destination version back to trunk.
This happens when an ubuntu core developer makes a manual upload of your package to ubuntu without using ci train. 99% of the time this would be no-change rebuilds during library transitions, or other trivial fixes for packaging issues. Before you can build the silo, it is necessary to sync the manual distro upload back to your trunk, so that they match. The first thing to do is check the launchpad page for your source package (for example here's the page for unity8. On that page look for a version number that ends in "0ubuntu2" or similar (the train only ever generates versions ending in "0ubuntu1"). Click through that page to get to the "available diffs" for that version, download that diff, and then commit that directly to your trunk. Do not make a merge proposal with this diff and include it in the silo, that does not work and is a backwards approach. You want to copy what's in distro into your trunk, silos are for getting branches into distro.
You typed a package name into PACKAGES_TO_REBUILD that wasn't part of the silo configuration. You should double check your spelling, and if it's correct, then your request is misconfigured.