GitWorkflow

Differences between revisions 59 and 76 (spanning 17 versions)
Revision 59 as of 2017-07-19 14:49:36
Size: 37537
Editor: paelzer
Comment:
Revision 76 as of 2019-07-16 08:25:12
Size: 28737
Editor: racb
Comment: Note range-diff to help with merges
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
The Ubuntu Server Team has devised a git-based workflow for merges. It relies on a set of tools written to assist in the process, but the underlying driver is a basic git-rebase process. This page will attempt to describe the tools used, and the general workflow, with an example or two. Finally, there will be some notes about corner-cases or caveats to the process that might need special-handling.

There are two aspects to the workflow: on the one hand is a the importer logic, which relates to the construction of the repository(ies) in Launchpad for source package, and is documented in brief [[#importer|here]] mostly for transparency and documentation; on the other hand are tools for using the git repositories in Launchpad to do bugfixes and merges, which most likely the section of interest to developers.
The Ubuntu Server Team has devised a git-based workflow for package maintenance. It relies on a set of tools written to assist in the process, but the underlying driver is a basic git-rebase process. This page will attempt to describe the tools used, and the general workflow, with an example or two. Finally, there will be some notes about corner-cases or caveats to the process that might need special-handling.

There are two aspects to the workflow: on the one hand is the importer logic, which relates to the construction of the repository(ies) in Launchpad for source package, and is documented in brief [[#importer|here]] mostly for transparency and documentation; on the other hand are tools for using the git repositories in Launchpad to do bugfixes and merges, which is most likely the section of interest to developers.
Line 9: Line 9:
= Introductory blog posts =
There is a series of blog posts covering the `git ubuntu` tool and this Git Workflow:
 * [[https://insights.ubuntu.com/2017/07/24/developing-ubuntu-using-git/|Developing Ubuntu Using Git]]
 * [[https://naccblog.wordpress.com/2017/08/01/git-ubuntu-clone/|git ubuntu clone]]
Line 11: Line 16:
Throughout this documentation, we will make reference to several git 'insteadof' shortcuts, which can be placed into `.gitconfig` as:
Since this workflow obviously uses `git`, you should have at least this basic config in `~/.gitconfig`:
Line 14: Line 18:
[url "git+ssh://yourusername@git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/"]
    insteadOf = ldusdp:
[url "git+ssh://yourusername@git.launchpad.net/~usd-import-team/ubuntu/+source/"]
    insteadOf = lpusip:
[url "git+ssh://yourusername@git.launchpad.net/~yourusername/ubuntu/+source/"]
    insteadOf = lpmep:
[user]
 email = youremail@example.com
 name = Your Name
Line 22: Line 23:
Important note: lpusip and lpusdp are used somewhat interchangeably on this page, to refer to the 'canonical' (little c) imported git tree. lpusip is used currently as a staging area while we work through bugs in the importer and the process documented here. Once we have accepted this tooling as the "default", we will move to lpusdp and lpusip will probably be decommisioned.
Line 29: Line 30:
 1. Deconstruct current Ubuntu delta into logical commits.  1. Split current Ubuntu delta into logical commits.
Line 48: Line 49:
 1. Deconstruct current Ubuntu delta into logical commits.  1. Split current Ubuntu delta into logical commits.
Line 51: Line 52:
   1. You can pass --bug to `git ubuntu merge` and it will prefix all of the following names with `lp<bug>`, e.g. 'lp<bug>/old/ubuntu'. This allows you to have multiple merges in the same local repository without having to pass -f and overriding tags.
   1.
This will create a local 'debian/sid' branch that tracks 'importer/debian/sid'.
   1. This will create 3 tags (you might need to pass '-f' if the tags already exist).
   * This will create a local 'debian/sid' branch that tracks 'importer/debian/sid'.
   * This will create 3 tags (you might need to pass '-f' if the tags already exist).
Line 57: Line 57:
   1. You may want to verify the tags (with `git show <tag>`) are pointing to the expected commits/versions.
    1. especially if you are re-merging something formerly done with this process you will need "-f" on `git ubuntu merge` otherwise the tags will still point to the last merge revisions.
   1. This will replay the git-merged delta into a sequence of linear non-merge commits.
    1. In the importer algorithm, explained later, each imported version is possibly a git merge. If you were to directly rebase 'old/ubuntu' onto 'debian/sid', the merges would be replayed, which will typically fail.
    1. Instead, this will replay the current ubuntu delta back onto 'old/debian', but dropping the merge commits as artifacts of the `git ubuntu import` algorithm.
    1. The end result, which will be the checked-out if successful, is a sequence of `git-cherry-pick`s of published Ubuntu versions after the 'old/debian' version that are not merges (but are otherwise identical to the sequence of commits in 'old/ubuntu'.
    1. In this process, there might be some empty commits cherry-picked over. This is the result of, e.g., copy-forwards between series (Y -> Z) or pockets (proposed -> release). To maximize the information retained, these empty commits are retained in the reconstruct, but will be dropped in the deconstruct phase.
     1.1. This might be dropped from a future release.
   1. If development occurred in the git tree previously, and that work was uploaded and tagged (e.g., upload/<version>), it's possible that some of the following steps can be skipped. This reflects the fact that the imported version is logically already deconstructed in the git tree.
   1. This commit is tagged as 'reconstruct/<ubuntu version>'.
    1. Please note that our convention is to use '{reconstruct,deconstruct,logical}/<version>' (with standard [[http://dep.debian.net/deps/dep14/|dep14]] substitutions). This allows us to store multiple tags associated with multiple versions for archival purposes.
   * You may want to verify the tags (with `git show <tag>`) are pointing to the expected commits/versions.
    * especially if you are re-merging something formerly done with this process you will need "-f" on `git ubuntu merge` otherwise the tags will still point to the last merge revisions.
   * This will replay the git-merged delta into a sequence of linear non-merge commits.
    * In the importer algorithm, explained later, each imported version is possibly a git merge. If you were to directly rebase 'old/ubuntu' onto 'debian/sid', the merges would be replayed, which will typically fail.
    * Instead, this will replay the current ubuntu delta back onto 'old/debian', but dropping the merge commits as artifacts of the `git ubuntu import` algorithm.
    * The end result, which will be the checked-out if successful, is a sequence of `git-cherry-pick`s of published Ubuntu versions after the 'old/debian' version that are not merges (but are otherwise identical to the sequence of commits in 'old/ubuntu'.
    * In this process, there might be some empty commits cherry-picked over. This is the result of, e.g., copy-forwards between series (Y -> Z) or pockets (proposed -> release). To maximize the information retained, these empty commits are retained in the reconstruct, but will be dropped in the split phase.
     * This might be dropped from a future release.
   * If development occurred in the git tree previously, and that work was uploaded and tagged (e.g., upload/<version>), it's possible that some of the following steps can be skipped. This reflects the fact that the imported version is logically already split in the git tree.
   * This commit is tagged as 'reconstruct/<ubuntu version>'.
    * Please note that our convention is to use '{reconstruct,split,logical}/<version>' (with standard [[http://dep.debian.net/deps/dep14/|dep14]] substitutions). This allows us to store multiple tags associated with multiple versions for archival purposes.
   * You can pass --bug to `git ubuntu merge` and it will prefix all of the following names with `lp<bug>`, e.g. 'lp<bug>/old/ubuntu'. This allows you to have multiple merges in the same local repository without having to pass -f and overriding tags.
    * If you do pass --bug, all the tags referred to below will be under lp<bug>, please adjust your commands accordingly.
Line 89: Line 91:
  1. `git ubuntu tag --deconstruct`   1. `git ubuntu tag --split`
Line 92: Line 94:
   1. Interactively rebase the deconstructed Ubuntu delta and drop 'changelog' and 'update-metadata' changes, by simply dropping those lines from the rebase todo file.    1. Interactively rebase the split Ubuntu delta and drop 'changelog' and 'update-metadata' changes, by simply dropping those lines from the rebase todo file.
Line 101: Line 103:
  1. `git rebase new/debian`
    1. Replay the logical delta onto the latest Debian release.
  1. `git rebase --onto new/debian old/debian HEAD`
    1. Replay the logical delta (changes between old/debian and HEAD) onto the latest Debian release.
Line 109: Line 111:
 1. Run `git ubuntu merge finish ubuntu/devel`. This effectively wraps all of the following steps, which can be run manually instead, if desired.  1. Run `git ubuntu merge finish ubuntu/devel`. This effectively wraps all of the following steps. From git those can be run manually instead, if desired - in the snap not all subcommands might be exposed.
  1. Verify we started a merge from either new/debian or lp#/new/debian as ancestor of HEAD
Line 111: Line 114:
   1. `git merge-changelogs old/debian old/ubuntu new/debian`    1. `dpkg-mergechangelogs old/debian old/ubuntu new/debian`
Line 124: Line 127:
 1. While doing the former the Changelog is constructed out of the commit messages you had in your branch. Please check debian/changelog if they are sufficient for an upload.
Line 130: Line 134:
 1. Push and review.  1. Make Ready for Push and review.
Line 137: Line 141:
  1. `git push <launchpad user> merge old/debian new/debian old/ubuntu reconstruct/<ubuntu version> \`
   `deconstruct/<ubuntu version> logical/<ubuntu version>`
   1. The remote name here, <launchpad user>, is an artifact of `git ubuntu clone`. If you used git directly, you might push to lpmep:<source package name>, e.g.
  1. Go to the Launchpad page of your just uploaded merge branch and propose for merging into the USD git project: `~<yourusername>/ubuntu/+source/<yourpackage>/+git/<yourpackage>/+ref/merge`
   1. usually you just search for the package you want to upload and will find the related usd importer tree. For example ``~usd-import-team/ubuntu/+source/<yourpackage>`
   1. As target use `debian/sid` or whatever you used to rebase on
   1. Discussions, new revisions and so on will be handled on that merges launchpad page. It is typical to file merges as Launchpad bugs as well, so ensure that the appropriate bug is mentioned in debian/changelog (e.g., 'Merge with Debian unstable (LP: #...). Remaining changes:') and Launchpad will automatically refer to your MR in the bug.
 1. Upload (work in progress!)
  1. not sure if eventually the tooling will do so, but whoever uploads after the review is done should tag and push an uploaded tag. That is used later on to provide the split history instead of the squashed import to the next user of this process. `git ubuntu tag`
  1. `git ubuntu merge commit ubuntu/<latest series>`
  1. `git push lpusdp:<source package name>
   1. Presumes you have write rights to the
USD git repository.
 1. Push and review
  1. Pushing via `git ubuntu` is still WIP but was nicely outlined including the following review process which is specific to the Server Team and therefore in their Knowledge Base as "[[https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#Merge_Proposals_and_Reviewing|Merge Proposals and Reviewing]]"
  1. You can also push an MP via git and Launchpad by
     
`git push <launchpad user> merge old/debian new/debian old/ubuntu reconstruct/<ubuntu version> \`
   `split/<ubuntu version> logical/<ubuntu version>`
   1. The remote name here, <launchpad user>, is an artifact of `git ubuntu clone`. If you used git directly, you might push to git+ssh://yourusername@git.launchpad.net/~yourusername/ubuntu/+source/<source package name>
   1. Then go to the Launchpad page of your just uploaded merge branch and propose for merging into the USD git project: `~<yourusername>/ubuntu/+source/<yourpackage>/+git/<yourpackage>/+ref/merge`
    1. usually you just search for the package you want to upload and will find the related usd importer tree. For example ``~usd-import-team/ubuntu/+source/<yourpackage>`
    1. As target use `debian/sid` or whatever you used to rebase on
  1. Discussions, new revisions and so on will be handled on that merges launchpad page. It is typical to file merges as Launchpad bugs as well, so ensure that the appropriate bug is mentioned in debian/changelog (e.g., 'Merge with Debian unstable (LP: #...). Remaining changes:') and Launchpad will automatically refer to your MR in the bug.
 1. Upload (Eventually that step won't be needed as it will be picked up from approved Merge Proposals)
  1. tag the propsed merge as the upload that you'll sponsor (or be uploaded by other sponsors)
  `$ git ubuntu tag --upload`
  
1. Upload this tag to USDI, the remotes are all set up already
  `$ git push pkg upload/<version>`

  `pkg` is the remote provided by `git ubuntu` already and maps to `ssh://<LPuser>@git.launchpad.net/~usd-import-team/ubuntu/+source/<packagename>` (Presumes you have write rights to the USDI git repository.)
  1. After this when the importer picks up the new version it will map the rich history provided by the upload tag to the new import it does.
Line 184: Line 194:
Available from [[https://code.launchpad.net/~usd-import-team/usd-importer/+git/usd-importer|Launchpad git repository]].

It is also available as as snap: `sudo snap install git-ubuntu --classic` and then it should be available as a `git-ubuntu`.

NOTE: we are still working on issues related to argcomplete with the recent rename from `usd` -> `git-ubuntu`.

Top-level command to all the Ubuntu git repository-manipulating tools. To see all available subcommands, type:
Line 185: Line 202:
usage: git-ubuntu [-h] [-P PARENTFILE] [-L PULLFILE]
                  build|build-source|clone|import|import-local|import-ppa|merge|queue|tag
                  ...

Ubuntu git development tool

positional arguments:
  build|build-source|clone|import|import-local|import-ppa|merge|queue|tag
                        
                        build - Build a git-ubuntu-cloned tree with dpkg-buildpackage
                        build-source - Build a source package and changes file
                        clone - Clone a source package git repository to a directory
                        import - Update a launchpad git tree based upon the state of the Ubuntu and Debian archives
                        import-local - Import a DSC file locally
                        import-ppa - Update a launchpad git tree based upon the state of a PPA archive
                        merge - Given a git-ubuntu import'd tree, assist with an Ubuntu merge
                        queue - Interact with the unapproved and new queues
                        tag - Given a git-ubuntu import'd tree, tag a commit respecting DEP14

optional arguments:
  -h, --help show this help message and exit
  -P PARENTFILE, --parentfile PARENTFILE
                        File specifying parent-child relationship overrides as:
                        <pkgname> <child version> <publish parent version> <changelog parent version>
                        with one override per line.
                        default=/home/nacc/work/usd-importer/gitubuntu/parent_overrides.txt
  -L PULLFILE, --pullfile PULLFILE
                        File specifying source files to use instead of Launchpad to work around source file shadowing:
                        <pkgname> <version> <URL of DSC file> <URLs of orig files>
                        with one package per line. '-' for the DSC file indicates to use the
                        one from Launchpad.
                        default=/home/nacc/work/usd-importer/gitubuntu/pull_overrides.txt
git ubuntu
Line 218: Line 204:

Available from [[https://code.launchpad.net/~usd-import-team/usd-importer/+git/usd-importer|Launchpad git repository]].

It is also available as as snap: `sudo snap install git-ubuntu --classic` and then it should be available as a `git-ubuntu`.

NOTE: we are still working on issues related to argcomplete with the recent rename from `usd` -> `git-ubuntu`.

Top-level command to all the Ubuntu git repository-manipulating tools.
Each subcommand has its own help page. For example:
{{{
git ubuntu clone --help
}}}
Line 228: Line 211:
{{{
usage: git-ubuntu import [-h] [-v] [--retries RETRIES]
                         [--retry-backoffs RETRY_BACKOFFS]
                         [--proto [git|http|https]] [-o LP_OWNER] [-l LP_USER]
                         [--dl-cache DL_CACHE] [--no-fetch] [--no-push]
                         [--no-clean] [-d DIRECTORY] [--active-series-only]
                         package

Update a launchpad git tree based upon the state of the Ubuntu and Debian
archives

positional arguments:
  package Which package to update in the git tree

optional arguments:
  -h, --help show this help message and exit
  -v, --verbose Increase verbosity (default: False)
  --retries RETRIES Number of times to attempt to retry downloading from
                        Launchpad. (default: 3)
  --retry-backoffs RETRY_BACKOFFS
                        Comma-separated list of backoffs in seconds to use
                        between each retry attempt. Default is exponential
                        backoff.
  --proto [git|http|https]
                        Specify protocol to use for fetch. Default: git
  -o LP_OWNER, --lp-owner LP_OWNER
                        Which launchpad user's tree to update (default: usd-
                        import-team)
  -l LP_USER, --lp-user LP_USER
                        Specify the Launchpad user to use (default: nacc)
  --dl-cache DL_CACHE Cache directory for downloads. Default is to put
                        downloads in <directory>/.git/git-ubuntu-cache
  --no-fetch Do not fetch from the remote (DANGEROUS; only useful
                        for debugging); implies --no-push (default: False)
  --no-push Do not push to the remote (default: False)
  --no-clean Do not clean the temporary directory (default: False)
  -d DIRECTORY, --directory DIRECTORY
                        Use git repository at specified location rather than a
                        temporary directory (implies --no-clean). Will create
                        the local repository if needed.
  --active-series-only Do an import of only active Ubuntu series history,
                        implies --no-push. (default: False)
}}}

The primary problem we are trying to solve is how to make merges
consistent across the server team, and, ideally easy to review for
sponsors/uploaders.

Robie Basak and others came up with a git-based
workflow (documented broadly [[https://git.launchpad.net/usd-importer/tree/README.workflow|here]] and hopefully clarified on this page). This workflow 'just' uses git (the

The primary problem we are trying to solve is how to make merges consistent across the server team, and, ideally easy to review for sponsors/uploaders.

Robie Basak and others came up with a git-based workflow (documented broadly [[https://git.launchpad.net/usd-importer/tree/README.workflow|here]] and hopefully clarified on this page). This workflow 'just' uses git (the
Line 320: Line 257:
While the intention is to use the lpusip repositories as the remotes for merge requests, it is also possible to run the importer fully locally and not push to a remote. While the intention is to use the `~usd-import-team/ubuntu/+source` repositories as the remotes for merge requests, it is also possible to run the importer fully locally and not push to a remote.
Line 324: Line 261:
This will run the import algorithm locally, which can take some time, leaving the repository in `/path/to/repository`, but not pushing to any remotes. By the nature of the importing algorithm, the SHAs of the commits in your local tree will match the commits of the lpusip trees. '''Note that the -d path should not exist and it must be provided as an absolute path.''' This will run the import algorithm locally, which can take some time, leaving the repository in `/path/to/repository`, but not pushing to any remotes. By the nature of the importing algorithm, the SHAs of the commits in your local tree will match the commits of the `~usd-import-team/ubuntu` trees. '''Note that the -d path should not exist and it must be provided as an absolute path.'''
Line 338: Line 275:
{{{
usage: git-ubuntu clone [-h] [-v] [--retries RETRIES]
                        [--retry-backoffs RETRY_BACKOFFS]
                        [--proto [git|http|https]] [--no-launchpad-user]
                        [-l LAUNCHPAD_USER]
                        package [directory]

Clone a source package git repository to a directory

positional arguments:
  package Name of source package to clone
  directory Local directory to clone to. If not specified, a directory with the same name as PACKAGE will be used

optional arguments:
  -h, --help show this help message and exit
  -v, --verbose Increase verbosity
  --retries RETRIES Number of times to attempt to retry downloading from Launchpad.
  --retry-backoffs RETRY_BACKOFFS
                        Comma-separated list of backoffs in seconds to use between each retry attempt. Default is exponential backoff.
  --proto [git|http|https]
                        Specify protocol to use for fetch. Default: git
  --no-launchpad-user Do not prompt for launchpad user
  -l LAUNCHPAD_USER, --launchpad-user LAUNCHPAD_USER
                        Specify launchpad user

Example:
   * clone to open-iscsi/
     git-ubuntu clone open-iscsi
   * clone to ubuntu.git
     git-ubuntu clone open-iscsi ubuntu.git
   * use https rather than git protocol for read-only remotes:
     git-ubuntu clone --https open-iscsi
}}}

A subcommand which does not rely on the lpusip insteadof definition (and creates them when not found). Instead it configures a local repository with appropriate remotes and tracking branches.
Clone a source package git repository to a directory with appropriate remotes and tracking branches.

Examples:
   * clone to open-iscsi/: `git-ubuntu clone open-iscsi`
   * clone to ubuntu.git: `git-ubuntu clone open-iscsi ubuntu.git`
   * use https rather than git protocol for read-only remotes: `git-ubuntu clone --https open-iscsi`
Line 376: Line 285:
{{{
usage: git-ubuntu merge [-h] [-v] [--bug BUG] [--release RELEASE]
                        [-d DIRECTORY] [-f] [--tag-only]
                        start|finish commitish [onto]

Given a git-ubuntu merge import'd tree, assist with an Ubuntu merge

positional arguments:
  start|finish start - Breakup Ubuntu delta into individual commits
                        finish - Construct d/changelog suitable for uploading. `git-ubuntu merge merge start` must have been run first.
  commitish A reference to a commit whose corresponding version is to be merged.
  onto A reference to a commit whose corresponding version to prepare to merge with. If not specified, debian/sid is used. If <onto> does not already
                        exist as a local branch, it will be created and track lpusip/<onto>.

optional arguments:
  -h, --help show this help message and exit
  -v, --verbose Increase verbosity
  --bug BUG Launchpad bug to close with this merge.
  --release RELEASE Ubuntu release to target with merge. If not specified, the current development release from Launchpad is used.
  -d DIRECTORY, --directory DIRECTORY
                        Use git repository at specified location.
  -f, --force Overwrite existing tags and branches, where applicable.
  --tag-only When rebasing a previous merge, only tag old/ubuntu, old/debian and new/debian.
Given a git-ubuntu merge import'd tree, assist with an Ubuntu merge.
Line 403: Line 290:
}}}

A
subcommand which takes a `git ubuntu clone`'d tree and a

This subcommand takes a `git ubuntu clone`'d tree and a
Line 420: Line 306:
deconstruct/<version> because it's clever. split/<version> because it's clever.
Line 424: Line 310:
end-commit as deconstruct/<version>. end-commit as split/<version>.
Line 431: Line 317:
{{{
usage: git-ubuntu tag [-h] [-v]
                      [--logical | --deconstruct | --reconstruct | --upload | --format FORMAT]
                      [--bug BUG] [-d DIRECTORY] [-f]
                      [commitish]
Line 439: Line 319:
positional arguments:
  commitish Commitish to tag

optional arguments:
  -h, --help show this help message and exit
  -v, --verbose Increase verbosity
  --logical Equivalent to --format='logical/%(version)s', with version extracted from the commit tagged as old/ubuntu
  --deconstruct Equivalent to --format='deconstruct/%(version)s'
  --reconstruct Equivalent to --format='reconstruct/%(version)s'
  --upload Equivalent to --format='upload/%(version)s'
  --format FORMAT Use FORMAT when tagging commit
  --bug BUG Launchpad bug for the tags (primarily used in the merge workflow)
  -d DIRECTORY, --directory DIRECTORY
                        Use git repository at specified location.
  -f, --force Overwrite existing tags and branches, where applicable.
Line 456: Line 320:
}}}

A subcommand which
tags the given commitish with the specified format, following the DEP14 requirements. The base version is read from debian/changelog at that commitish, except for logical tags, which reads from 'old/ubuntu'.

This subcommand tags the given commitish with the specified format, following the DEP14 requirements. The base version is read from debian/changelog at that commitish, except for logical tags, which reads from 'old/ubuntu'.
Line 462: Line 325:
== git-dsc-commit ==

No longer used in our workflow.

{{{
usage: git-dsc-commit [--tree-only] DSCFILE

Untars the version specified by the DSCFILE and commits it to a git repository.

optional arguments:
  --tree-only Only call git-write-tree on the result.
                        git-commit-tree will need to be manually
                        invoked to commit the changes to the
                        repository.
}}}

Available from [[https://code.launchpad.net/~usd-import-team/usd-importer/+git/usd-importer|Launchpad git repository]].

== git-reconstruct-changelog ==

No longer used in our workflow.

{{{
usage: git-reconstruct-changelog COMMITISH

Replays git commit messages starting after COMMITISH into debian/changelog.
}}}

Available from [[https://code.launchpad.net/~usd-import-team/usd-importer/+git/usd-importer|Launchpad git repository]].

== git-merge-changelogs ==

No longer used in our workflow.

{{{
usage: git-merge-changelogs BASE_COMMITISH NEWA_COMMITISH NEWB_COMMITISH

Runs dpkg-mergechangelogs on the debian/changelogs from the passed commitishs.
}}}

Available from [[https://code.launchpad.net/~usd-import-team/usd-importer/+git/usd-importer|Launchpad git repository]].

== xgit ==

No longer used in our workflow.

{{{
usage: xgit

Either 'enters' or 'exits' an xgit-configured git repository. Such a repository
as a git/ and gitwd/ directory structure, where git/ is the GIT_DIR and gitwd/
is the GIT_WORK_TREE. This allows for easier git-dsc-commit invocation and
clarity on what is in the working tree and what is not.
}}}

Available from [[https://code.launchpad.net/~usd-import-team/usd-importer/+git/usd-importer|Launchpad git repository]].
Line 536: Line 343:
 2. Account for every change in the previous logical commit.  2. Account for every change in the previous logical commit. You can use `git range-diff old/debian..old/ubuntu new/debian..<merge branch>` to help with this (requires Git >= 2.19).

The Ubuntu Server Team has devised a git-based workflow for package maintenance. It relies on a set of tools written to assist in the process, but the underlying driver is a basic git-rebase process. This page will attempt to describe the tools used, and the general workflow, with an example or two. Finally, there will be some notes about corner-cases or caveats to the process that might need special-handling.

There are two aspects to the workflow: on the one hand is the importer logic, which relates to the construction of the repository(ies) in Launchpad for source package, and is documented in brief here mostly for transparency and documentation; on the other hand are tools for using the git repositories in Launchpad to do bugfixes and merges, which is most likely the section of interest to developers.

NB: this page is a work in progress and no guarantees are asserted as to its correctness while it is being fleshed out.

Introductory blog posts

There is a series of blog posts covering the git ubuntu tool and this Git Workflow:

Git configuration

Since this workflow obviously uses git, you should have at least this basic config in ~/.gitconfig:

[user]
        email = youremail@example.com
        name = Your Name

Git workflow for merging

Algorithm outline

  1. Obtain git tree.
  2. Split current Ubuntu delta into logical commits.
  3. Rebase delta onto latest Debian release.
  4. Build/test.
  5. Reconstruct debian/changelog for new version.
  6. Update metadata for new version.
  7. Upload.

Detailed workflow

This guide assumes we are attempting to merge the version of <source package name> in the latest Ubuntu series with the latest Debian unstable version. If that is not the case, you might need to pass different commitishes to git ubuntu merge and/or specify the onto. See below for details.

  1. Obtain git tree.
    1. See if there is already an imported tree at https://code.launchpad.net/~usd-import-team/+git.

      1. If there is, check if it is current. This usually means checking rmadison and ensuring that both 'debian/sid' and 'ubuntu/devel' are consistent with it. This eventually should not be necessary as we will run the importer in a cronjob.

        1. If it is, proceed to the next step.
        2. If it is not, send an e-mail to the Ubuntu Server list asking for it to be updated or import it locally (see the git ubuntu import section). You should eventually get a reply with a reference to the imported tree.

      2. If there is not, send an e-mail to the Ubuntu Server list asking for it to be imported or import it locally (see the git ubuntu import section). You should eventually get a reply with a reference to the imported tree.

    2. git ubuntu clone <source package name>

      1. Feel free to use git directly with other git configuration flags, etc. as desired, but this guide will only cover the use of the git ubuntu clone tool. Similar results can be achieve throughout this guide with appropriate git subcommands, but YMMV.

  2. Split current Ubuntu delta into logical commits.
    1. cd into the just created dir cd <source package name>

    2. git ubuntu merge start ubuntu/devel

      • This will create a local 'debian/sid' branch that tracks 'importer/debian/sid'.
      • This will create 3 tags (you might need to pass '-f' if the tags already exist).
        1. 'old/ubuntu': will point to the same commit as ubuntu/devel tip and represents the current state of Ubuntu for this source package. In other words, this tag points to a commit corresponding to the latest Ubuntu version and containing the complete Ubuntu delta.
        2. 'old/debian': will point to the commit that is a common ancestor of both ubuntu/devel tip and debian/sid tip, in other words the commit corresponding to the version last merged with in Debian.
        3. 'new/debian': will point to debian/sid tip, in other words the commit corresponding to the version to be merged with and the latest version in Debian unstable.
      • You may want to verify the tags (with git show <tag>) are pointing to the expected commits/versions.

        • especially if you are re-merging something formerly done with this process you will need "-f" on git ubuntu merge otherwise the tags will still point to the last merge revisions.

      • This will replay the git-merged delta into a sequence of linear non-merge commits.
        • In the importer algorithm, explained later, each imported version is possibly a git merge. If you were to directly rebase 'old/ubuntu' onto 'debian/sid', the merges would be replayed, which will typically fail.
        • Instead, this will replay the current ubuntu delta back onto 'old/debian', but dropping the merge commits as artifacts of the git ubuntu import algorithm.

        • The end result, which will be the checked-out if successful, is a sequence of git-cherry-picks of published Ubuntu versions after the 'old/debian' version that are not merges (but are otherwise identical to the sequence of commits in 'old/ubuntu'.

        • In this process, there might be some empty commits cherry-picked over. This is the result of, e.g., copy-forwards between series (Y -> Z) or pockets (proposed -> release). To maximize the information retained, these empty commits are retained in the reconstruct, but will be dropped in the split phase.

          • This might be dropped from a future release.
      • If development occurred in the git tree previously, and that work was uploaded and tagged (e.g., upload/<version>), it's possible that some of the following steps can be skipped. This reflects the fact that the imported version is logically already split in the git tree.

      • This commit is tagged as 'reconstruct/<ubuntu version>'.

        • Please note that our convention is to use '{reconstruct,split,logical}/<version>' (with standard dep14 substitutions). This allows us to store multiple tags associated with multiple versions for archival purposes.

      • You can pass --bug to git ubuntu merge and it will prefix all of the following names with lp<bug>, e.g. 'lp<bug>/old/ubuntu'. This allows you to have multiple merges in the same local repository without having to pass -f and overriding tags.

        • If you do pass --bug, all the tags referred to below will be under lp<bug>, please adjust your commands accordingly.

    3. git rebase -i old/debian

      1. Interactively rebase the reconstructed Ubuntu delta and edit ('e' in the rebase options) each cherry-picked commit into its components.
        1. While we will try to cover as many cases as possible in this document (when final), the man git-rebase "INTERACTIVE MODE" section is probably still worth reviewing.

        2. In the editor provided by git-rebase, there are three cases to consider:

          1. For commits of imports to the git tree (typically with messages like 'Import version <version> to <distribution>/<series>[-<pocket>]'), modify the 'p'/'pick' line to 'e'/'edit'.

          2. For commits that are already logical changes (typically this is the result of a prior merge using this process, or active development in the ubuntu-server-dev git tree), the line can be left unmodified ('p'/'pick').
          3. For copies of the same version between pockets/releases, the line will be commented out (reflecting an empty commit) and can be left commented out. This is mentioned above as an artifact of our rebasing process.
        3. As the git-rebase process stops at each commit, run git reset HEAD^, which will unstage each commit.

        4. See what was previously in the commit at this point with git-diff. In particular, we are interested in what is indicated by 'debian/changelog' as being contained in this published Ubuntu version.

        5. For each logical change listed in git diff -- debian/changelog:

          1. Run git add -i (interactive add) to stage a commit for the change.

          2. Run git commit to commit the staged changes, using the corresponding 'debian/changelog' entry as the commit message.

          3. Sometimes, there will be changes in a published release that are not mentioned in 'debian/changelog'. It is recommended to commit each of these as a new commit after those that are listed in 'debian/changelog', with an appropriate commit message, appending '[previously undocumented]'.
        6. The remaining changes should only be to 'debian/changelog' (the actual changelog entries) and 'debian/control' (VCS/maintainer modifications). This is verifiable by examining git diff -p and git diff.

          1. Commit all the 'debian/changelog' entries for this step with commit message 'changelog'.
          2. Commit any debian/control metadata (VCS, Maintainer) for this step with commit message 'update-metadata'.
        7. The result at the end of each rebase step is no modifications are lost/gained relative to the old commit (which no longer applies in the current state).
          1. At this point, git diff should report no unstaged changes. Run git rebase --continue and repeat the previous steps for the next published Ubuntu version.

      2. The end result of the above loop will be a sequence of smaller commits, one per 'debian/changelog' entry (with potentially additional commits for previously undocumented changes).
        1. The final commit will represent a broken-out history (viewable with git-log) for the latest Ubuntu version and no contentful differences to that Ubuntu version.

        2. This can be verified with git diff -p old/ubuntu.

    4. git ubuntu tag --split

      1. Tag the final commit for easy reference and review.
    5. git rebase -i old/debian

      1. Interactively rebase the split Ubuntu delta and drop 'changelog' and 'update-metadata' changes, by simply dropping those lines from the rebase todo file.
      2. An additional goal in this step is to consolidate the delta, e.g. sometimes a change is added in one Ubuntu release and then removed in a subsequent Ubuntu release. The changes, in this case, should simply be dropped.
      3. These changes will be easily reproducible from this process and will lead to merge conflicts with the final rebase.
      4. The final commit will represent a consolidated broken-out history (viewable with git-log) for the latest Ubuntu version, without changes to 'debian/changelog' or to the metadata in 'debian/control'.

        1. This can be verified with git diff -p old/ubuntu which should indicate the only changes are in 'debian/changelog' and 'debian/control'.

    6. git ubuntu tag --logical

      1. Tag this commit for easy reference and review.
      2. This command uses 'old/ubuntu' as a reference for the version to use in the tag-name, so it must exist LP: #165113.

  3. Rebase delta onto latest Debian release.
    1. git rebase --onto new/debian old/debian HEAD

      1. Replay the logical delta (changes between old/debian and HEAD) onto the latest Debian release.
      2. It is possible individual logical changes no longer apply. A few common reasons and solutions are:
        1. The context for the patch has changed, so a manual edit of the commit is necessary.
        2. The change has been integrated in the new Debian version. In this case, there is no logical change in the new delta. However, for the purposes of reconstructing the changelog later, it is important to document what parts of the old delta have been dropped and why. It is recommended, therefore, to use git status --ignored to verify the current rebase progress before issuing git rebase --continue and if a logical commit becomes empty, to use git commit --allow-empty to specify that a given change is now 'Dropped' and ideally why, e.g., 'Fixed in Debian <version>'.

  4. Build/test.
    1. Explain sbuild / point to it.
    2. you can benefit from the importer being able to restore orig tarballs, for a merge all you need to do to create tarball, sign and upload with new tarball would be git ubuntu build-source -v --sign --for-merge

  5. Run git ubuntu merge finish ubuntu/devel. This effectively wraps all of the following steps. From git those can be run manually instead, if desired - in the snap not all subcommands might be exposed.

    1. Verify we started a merge from either new/debian or lp#/new/debian as ancestor of HEAD
    2. Reconstruct debian/changelog for new version.
      1. dpkg-mergechangelogs old/debian old/ubuntu new/debian

        1. This rebuilds the changelog for published versions from old/debian to new/debian, inserting Ubuntu releases as appropriate.
      2. git commit -m 'merge-changelogs' debian/changelog

      3. git reconstruct-changelog new/debian

        1. This replays the commit messages for all commits since new/debian on the current branch into debian/changelog in a new entry.
        2. Note that the new entry is 'UNRELEASED' and should be updated, and it's possible you will need to reorganized the replayed entries.
          • 1.1. Using git ubuntu merge finish will target the current development release or --release if specified.

      4. git commit -m 'reconstruct-changelog' debian/changelog

        1. It is appropriate to either commit the reconstruct-changelog result and then make changes, followed by a second commit with the final changelog; or to not commit reconstruct-changelog and then make changes, followed by a commit with the final changelog. In either case, the result of this step should be a changelog entry which fully covers all the delta, and explains any dropped or added changes.
    3. Update metadata for new version.
      1. update-maintainer

        1. This updates the maintainer entry in the control file appropriately for Ubuntu source packages.
      2. git commit -m 'update-maintainer' debian/control

  6. While doing the former the Changelog is constructed out of the commit messages you had in your branch. Please check debian/changelog if they are sufficient for an upload.
  7. Consider some Cleanup and check open Bugs
    1. Go to https://bugs.launchpad.net/ubuntu/+source/<source package name>. There check if any open bugs are either:

      1. closed already but not updated yet => close them

      2. likely closed, but need verification => notify reporter

      3. would be reasonable to add on top of the merge now that the package is touched anyway => implement

    2. Check all you have listed as "Remaining changes" in your changelog and consider pushing as much as possible to Debian.
  8. Make Ready for Push and review.
    1. git checkout -b merge

      1. if you happened to have a merge branch for this source already, recommended branch names to avoid conflicts are 'merge-<release>' and if multiple merges happen in one release 'merge-<release>-<version>'

      2. The MR will be based off this branch, feel free to name it something else.
      3. You will see those commits again next time this is merged. Please see the Workflow commentary for details on squashing changes (optional on the merge, but required on creating logical).

    2. git ubuntu lint

      1. You will get an automated check on your merge proposal anyway, but you could do so and clean up before uploading by calling the linter which checks for common issues. This feature is still new, please report bugs if you think a check is a false positive.
  9. Push and review
    1. Pushing via git ubuntu is still WIP but was nicely outlined including the following review process which is specific to the Server Team and therefore in their Knowledge Base as "Merge Proposals and Reviewing"

    2. You can also push an MP via git and Launchpad by
      • git push <launchpad user> merge old/debian new/debian old/ubuntu reconstruct/<ubuntu version> \

      • split/<ubuntu version> logical/<ubuntu version>

      • The remote name here, <launchpad user>, is an artifact of git ubuntu clone. If you used git directly, you might push to git+ssh://yourusername@git.launchpad.net/~yourusername/ubuntu/+source/<source package name>

      • Then go to the Launchpad page of your just uploaded merge branch and propose for merging into the USD git project: ~<yourusername>/ubuntu/+source/<yourpackage>/+git/<yourpackage>/+ref/merge

        1. usually you just search for the package you want to upload and will find the related usd importer tree. For example ~usd-import-team/ubuntu/+source/<yourpackage>`

        2. As target use debian/sid or whatever you used to rebase on

        3. Discussions, new revisions and so on will be handled on that merges launchpad page. It is typical to file merges as Launchpad bugs as well, so ensure that the appropriate bug is mentioned in debian/changelog (e.g., 'Merge with Debian unstable (LP: #...). Remaining changes:') and Launchpad will automatically refer to your MR in the bug.
  10. Upload (Eventually that step won't be needed as it will be picked up from approved Merge Proposals)
    1. tag the propsed merge as the upload that you'll sponsor (or be uploaded by other sponsors)

      $ git ubuntu tag --upload

    2. Upload this tag to USDI, the remotes are all set up already

      $ git push pkg upload/<version>

      pkg is the remote provided by git ubuntu already and maps to ssh://<LPuser>@git.launchpad.net/~usd-import-team/ubuntu/+source/<packagename> (Presumes you have write rights to the USDI git repository.)

    3. After this when the importer picks up the new version it will map the rich history provided by the upload tag to the new import it does.

Workflow commentary

During the review of a merge, sometimes changes are needed and these usually will get added on to the end rather than being squashed in. We think this approach is easier. We don't object to the merge being redone, and everything squashed in, if desired or if it is decided it is worth it in a particular case, but in the general case we don't think it is necessary.

But that means that the merge branch will often not be squashed. And on a re-merge, there may be extra uploads that were done after the merge upload.

So we think it makes sense to always do the squashing before doing the logical tag at the start of a merge. The previous merge should help significantly with this, but will not be sufficient in the general case.

This would also have the advantage that the logical tag has a strict definition: a logical tag must always consist of squashed commits. In other words, it a logical tag should never have commits that can be squashed. Of course, what can be squashed is subjective, so different developers won't necessarily produce identical logical commits.

The reason why one might suddenly be able to squash more than on logical is that due to Debian or Upstream changing there might be new opportunities to do so. But it turned out to be most useful to not squash (everything) on the merge branch.

After the merge or next time on reworking logical things can be re-squashed/separated, also it might be worth to rewrite commit messages at that time to make up a much better reconstructed changelog eventually.

It can in general be quite useful to store any sort of knowledge that one handling a merge or complex bug had along. But rather often the extra insight is not suitable for the main commit message. So if there is anything reasonable to save we recommend to use git notes to add those along the commits of the merge branch.

Tools: their usage, and where to get them

git ubuntu

Available from Launchpad git repository.

It is also available as as snap: sudo snap install git-ubuntu --classic and then it should be available as a git-ubuntu.

NOTE: we are still working on issues related to argcomplete with the recent rename from usd -> git-ubuntu.

Top-level command to all the Ubuntu git repository-manipulating tools. To see all available subcommands, type:

git ubuntu

Each subcommand has its own help page. For example:

git ubuntu clone --help

git ubuntu import

The primary problem we are trying to solve is how to make merges consistent across the server team, and, ideally easy to review for sponsors/uploaders.

Robie Basak and others came up with a git-based workflow (documented broadly here and hopefully clarified on this page). This workflow 'just' uses git (the only addition to 'stock' git are a few commands (git-dsc-commit, git-merge-changelogs, git-reconstruct-changelog) from Launchpad git repository.)) to effectively rebase the ubuntu tip onto the latest debian tip. That presumes you spent the time yourself to create a repository with commits representing the current states of debian unstable and the ubuntu development release.

The goal for the importer is to have canonical (little c) location for such commits to live, per-package. Everyone can refer to commits and tags in that tree, and they will have well-defined semantics and share SHA1s.

A secondary problem was obtaining the 'complete' history for a given source package. This would allow a user to git-blame a give file and get useful data. So we extended the algorithm (that Robie designed) to be more flexible. After hitting many corner-cases during implementation, we scrapped our complicated algorithm that produced clean trees for a clean algorithm that produces complicated trees.

Feel free to skip the next section if you're not interested in the implementation.

Algorithmic discussion

In essence, the algorithm looks at Launchpad's publishing history for versions it hasn't seen before (which if an empty or no local repository is specified and you aren't cloning an existing repository will be all of them). For each such version, it uses pull-lp-source/pull-debian-source equivalents and git-dsc-commit to import them into the git repository. Technically it uses a lower-level command then a proper commit (git write-tree and `git commit-tree`), so that we can get the imported tree, examine it and find its parents and then commit it. The parents for an imported tree are at most 2:

  1. The last version imported into the same series/pocket, with some knowledge of how to establish a new series/pocket.
  2. The last debian/changelog entry (using debian/changelog from the just-imported tree) that was successfully imported.

We call these the 'publishing parent' and 'changelog parent' respectively. Now, if we can't find either of these, we will orphan the import and if we only find one, we'll use what we have. But the resulting tree looks like many git-merges, even where there are not ubuntu-merges. This is correct by the algorithm but can be confusing to the original git-rebase workflow, because rebase will try to replay the git-merges and that doesn't work. More on this later.

Local usage

While the intention is to use the ~usd-import-team/ubuntu/+source repositories as the remotes for merge requests, it is also possible to run the importer fully locally and not push to a remote.

git ubuntu import --no-push <srcpkgname> -d /path/to/repository/

This will run the import algorithm locally, which can take some time, leaving the repository in /path/to/repository, but not pushing to any remotes. By the nature of the importing algorithm, the SHAs of the commits in your local tree will match the commits of the ~usd-import-team/ubuntu trees. Note that the -d path should not exist and it must be provided as an absolute path.

Now that you have a local import, it's probably best to perform the merge steps in a separate clone, e.g.

$ git clone -n /path/to/repository/git mergewd

Now you can do all the merge steps described above in mergewd. That way if you mess up and need to start again, your import directory is still pristine.

Also, because you'll likely need the orig.tar.gz for the new (Debian) version, you will find that in /path/to/repository. Copy that or symlink it to the parent directory of mergewd so debuild or whatever you use can find it.

git ubuntu clone

Clone a source package git repository to a directory with appropriate remotes and tracking branches.

Examples:

  • clone to open-iscsi/: git-ubuntu clone open-iscsi

  • clone to ubuntu.git: git-ubuntu clone open-iscsi ubuntu.git

  • use https rather than git protocol for read-only remotes: git-ubuntu clone --https open-iscsi

git ubuntu merge

Given a git-ubuntu merge import'd tree, assist with an Ubuntu merge.

Creates old/ubuntu, old/debian and new/debian tags for the specified Ubuntu merge. Breaks the specified Ubuntu merge into a sequence of non-git-merge-commits suitable for rebasing. Working tree must be clean.

This subcommand takes a git ubuntu clone'd tree and a commitish and attempts to give you a reconstructed sequence of linear commits that represent the same state as the commitish. It does this by using merge-base to figure out the common ancestor (it assumes onto is debian/sid but it also accepts a parameter) and then playing back the git history for commits between the merge-base and commitish. It then cherry-picks those tags from oldest to newest against the merge-base and does a quick sanity check that the resulting commit does not differ from the original commitish.

It tags that as reconstruct/<version> which may conflict with the git-based workflow some have been using previously. In that workflow, reconstruct/<version> is the broken-down sequence of changes including changelog and metadata, where each commit is a single change from the changelog. I have taken to now calling that pointer split/<version> because it's clever.

So you would break down reconstruct/<version> into its constituent changes (per debian/changelog for each version) and tag the resulting end-commit as split/<version>.

That can then be broken up by our workflow into a logical/<version> tag and then rebased onto debian/sid (or the specified onto).

git ubuntu tag

Given a git-ubuntu tag import'd tree, tag a commit respecting DEP14

By default, an upload tag is created. Working tree must be clean.

This subcommand tags the given commitish with the specified format, following the DEP14 requirements. The base version is read from debian/changelog at that commitish, except for logical tags, which reads from 'old/ubuntu'.

If no format is specified, an upload tag is created, suitable for pushing to the importer's repository.

Example(s)

Already imported trees live, currently, at: https://code.launchpad.net/~usd-import-team/+git.

Currently, because once we feel confident in the utility and correctness of the importer, we'll move them to https://code.launchpad.net/~ubuntu-server-dev/+git.

Caveats

All of this tooling is not meant to replace or reproduce any prior tools or works. And it is definitely not meant to replace a basic understanding of what a merge is and why they are necessary. The tooling is meant to make merges easier, but they are still not foolproof and require care and thought.

Reviewing Merges

Some notes to checking for correctness. The goal is to come up with a step-by-step process that, if done correctly, guarantees that the merge is correct.

  1. Mechnical checks: see wip/review
  2. Account for every change in the previous logical commit. You can use git range-diff old/debian..old/ubuntu new/debian..<merge branch> to help with this (requires Git >= 2.19).

    1. Is transferred exactly, transferred modified, or dropped.
    2. Is noted correctly in changelog (twice if modified).
    3. The changelog note still accurately reflects the changes.
  3. Account for any new commits.
    1. Check that the changelog matches the commits.
    2. All commits in the new delta should now be accounted for.
  4. Check new delta:
    1. Anything dropped was appropriately dropped.
    2. All previous delta still kept is still correct and appropriate to keep.
    3. Anything new or changed is appropriate to add or change.

UbuntuDevelopment/Merging/GitWorkflow (last edited 2023-01-17 15:54:05 by racb)