GitWorkflow

Differences between revisions 2 and 3
Revision 2 as of 2016-06-03 23:41:22
Size: 9922
Editor: nacc
Comment:
Revision 3 as of 2016-06-07 00:54:01
Size: 14944
Editor: nacc
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
NB: this page is a work in progress and no guarantees are asserted as to its correctness while it is fleshed out. NB: this page is a work in progress and no guarantees are asserted as to its correctness while it is being fleshed out.
Line 7: Line 7:
= Git configuration =

Throughout this documentation, we will make reference to several git 'insteadof' shortcuts, which can be placed into `.gitconfig` as:
{{{
[url <url>]
     insteadof = <shortcut>
}}}. Retain the specified quotes.

 1. "lpusdp:"
  1. "git+ssh://<lpuser>@git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/"
 1. "lpusip:"
  1. "git+ssh://<lpuser>@git.launchpad.net/~usd-import-team/ubuntu/+source/"
 1. "lpmep:"
  1. "git+ssh://<lpuser>@git.launchpad.net/~<lpuser>/ubuntu/+source/"

Note that eventually, lpusip may go away, as once the tooling is stable and accepted, we will not need to have a staging area for imports, potentially. Functionally, lpusdp and lpusip are the same, and ideally tree contents would never differ between the two.
Line 11: Line 28:
 1. Clone usd-import'd tree.
  1. `xgit`
 1. `usd-merge tag -f ubuntu/<development release>`
  1. Verify tags
 1. `usd-merge reconstruct old/ubuntu`
  1. Verify full history is present
 1. `git rebase -i old/debian`
  1. Break up git commits into one-commit-per-changelog entry.
  1. `git tag deconstruct/<version>
 1. `git rebase -i old/debian`
  1. Drop changelog and metadata modifications.
  1. `git tag logical/<version>
 1. `git rebase -i new/debian`
  1. Fold, fixup, drop Ubuntu delta.
 1. Obtain git tree.
 1. Deconstruct current Ubuntu delta into logical commits.
 1. Rebase delta onto latest Debian release.
 1. Build/test.
 1. Reconstruct debian/changelog for new version.
 1. Update metadata for new version.
 1. Upload.

== Detailed workflow ==

This guide assumes we are attempting to merge the version of <source package name> in the latest Ubuntu series with the latest Debian unstable version. If that is not the case, you might need to pass different commitishes to usd-merge and/or specify the onto. See below for details.

 1. Obtain git tree.
  1. See if there is already an imported tree at https://code.launchpad.net/~usd-import-team/+git.
   1. If there is, check if it is current.
    1. If it is, proceed to the next step.
    1. If it is not, send an e-mail to [[mailto:ubuntu-server@lists.ubuntu.com|the Ubuntu Server list] asking for it to be updated.
   1. If there is not, send an e-mail to [[mailto:ubuntu-server@lists.ubuntu.com|the Ubuntu Server list] asking for it to be imported.
  1. `git clone lpusip:<source package name>`
   1. Feel free to use other git configuration flags, etc. as desired.
  1. `git remote add lpme lpmep:<source package name>`
   1. `git fetch lpme`
    1. This may not fetch any objects if the remote doesn't exist yet, but that's ok.
 1. Deconstruct current Ubuntu delta into logical commits.
  1. `usd-merge tag ubuntu/<latest series>`
   1. This will create 3 tags (you might need to pass '-f' if the tags already exist).
    1. old/ubuntu: will point to the same commit as ubuntu/<latest series> tip, represents the current state of Ubuntu for this source package, in other words the commit corresponding to the latest Ubuntu version and containing the complete Ubuntu delta.
    1. old/debian: will point to the commit that is a common ancestor of both ubuntu/<latest series> tip and debian/sid tip, in other words the commit corresponding to the version last merged/synced from Debian.
    1. new/debian: will point to debian/sid tip, in other words the commit corresponding to the version to be merged with and the latest version in Debian unstable.
   1. You may want to verify the tags (with `git show <tag>`) are pointing to the expected commits/versions.
  1. `usd-merge reconstruct old/ubuntu`
   1. This will replay the current ubuntu delta back onto old/debian, but dropping the git-merge-commits that are artifacts of the usd-import algorithm.
   1. The end result, which will be the currently checked-out commit if successful, is a sequence of cherry-picks of published Ubuntu versions after the old/debian version.
   1. This commit is tagged as reconstruct/<ubuntu version>.
  1. `git rebase -i old/debian`
   1. Interactively rebase the reconstructed Ubuntu delta and edit ('e' in the rebase options) each cherry-picked commit into its components.
   1. This will result in a sequence of smaller commits, ideally one per debian/changelog entry.
   1. Commit all the debian/changelog entries for one rebase step with commit message 'changelog'.
   1. Commit any debian/control metadata (VCS, Maintainer) for one rebase step with commit message 'update-metadata'.
   1. The result at the end of each rebase step is a modifications are lost/gained relative to the old commit (which is no longer applies in the current status).
   1. Repeat this for each Ubuntu version.
   1. The end-result will be a commit with a broken-out history for the latest Ubuntu version and no contentful differences to that Ubuntu version.
    1. This can be verified with `git diff -p old/ubuntu`.
  1. `git tag deconstruct/<ubuntu version>`
   1. Tag this commit for easy reference and review.
  1. `git rebase -i old/debian`
   1. Interactively rebase the deconstructed Ubuntu delta and drop 'changelog' and 'update-metadata' changes.
   1. These changes will be easily reproducible from this process and will lead to merge conflicts with the final rebase.
   1. The end-result will be a commit with a broken-out history for the latest Ubuntu version, but no changes to debian/changelog or to the metadata in debian/control.
    1. This can be verified with `git diff -p old/ubuntu`.
  1. `git tag logical/<ubuntu version>`
   1. Tag this commit for each reference and review.
 1. Rebase delta onto latest Debian release.
  1. `git rebase -i new/debian`
    1. Replay the logical delta onto the latest Debian release.
 1. Build/test.
 1. Reconstruct debian/changelog for new version.
  1. `git merge-changelogs old/debian old/ubuntu new/debian`
Line 26: Line 87:
  1. `git merge-changelogs old/debian old/ubuntu new/debian`  1. Update metadata for new version.
Line 28: Line 89:
 1. `usd-merge commit`
 1. `git push`
 1. Request merge (or push directly to the usd-import repository)

== Detailed workflow ==
 1. Tag, push and review.
  1. `git checkout -b merge`
  1. `git push lpme merge`
  1. Go to Launchpad page for USD git project and request a merge review.
 1. Upload.
  1. `usd-merge commit ubuntu/<latest series>`
  1. `git push lpusdp:<source package name>
   1. Presumes you have write rights to the USD git repository.

The Ubuntu Server Team has devised a git-based workflow for merges. It relies on a set of tools written to assist in the process, but the underlying driver is a basic git-rebase process. This page will attempt to describe the tools used, and the general workflow, with an example or two. Finally, there will be some notes about corner-cases or caveats to the process that might need special-handling.

NB: this page is a work in progress and no guarantees are asserted as to its correctness while it is being fleshed out.

Git configuration

Throughout this documentation, we will make reference to several git 'insteadof' shortcuts, which can be placed into .gitconfig as:

[url <url>]
     insteadof = <shortcut>

. Retain the specified quotes.

  1. "lpusdp:"
    1. "git+ssh://<lpuser>@git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/"

  2. "lpusip:"
    1. "git+ssh://<lpuser>@git.launchpad.net/~usd-import-team/ubuntu/+source/"

  3. "lpmep:"
    1. "git+ssh://<lpuser>@git.launchpad.net/~<lpuser>/ubuntu/+source/"

Note that eventually, lpusip may go away, as once the tooling is stable and accepted, we will not need to have a staging area for imports, potentially. Functionally, lpusdp and lpusip are the same, and ideally tree contents would never differ between the two.

Git workflow for merging

Algorithm outline

  1. Obtain git tree.
  2. Deconstruct current Ubuntu delta into logical commits.
  3. Rebase delta onto latest Debian release.
  4. Build/test.
  5. Reconstruct debian/changelog for new version.
  6. Update metadata for new version.
  7. Upload.

Detailed workflow

This guide assumes we are attempting to merge the version of <source package name> in the latest Ubuntu series with the latest Debian unstable version. If that is not the case, you might need to pass different commitishes to usd-merge and/or specify the onto. See below for details.

  1. Obtain git tree.
    1. See if there is already an imported tree at https://code.launchpad.net/~usd-import-team/+git.

      1. If there is, check if it is current.
        1. If it is, proceed to the next step.
        2. If it is not, send an e-mail to [[mailto:ubuntu-server@lists.ubuntu.com|the Ubuntu Server list] asking for it to be updated.

      2. If there is not, send an e-mail to [[mailto:ubuntu-server@lists.ubuntu.com|the Ubuntu Server list] asking for it to be imported.

    2. git clone lpusip:<source package name>

      1. Feel free to use other git configuration flags, etc. as desired.
    3. git remote add lpme lpmep:<source package name>

      1. git fetch lpme

        1. This may not fetch any objects if the remote doesn't exist yet, but that's ok.
  2. Deconstruct current Ubuntu delta into logical commits.
    1. usd-merge tag ubuntu/<latest series>

      1. This will create 3 tags (you might need to pass '-f' if the tags already exist).
        1. old/ubuntu: will point to the same commit as ubuntu/<latest series> tip, represents the current state of Ubuntu for this source package, in other words the commit corresponding to the latest Ubuntu version and containing the complete Ubuntu delta.

        2. old/debian: will point to the commit that is a common ancestor of both ubuntu/<latest series> tip and debian/sid tip, in other words the commit corresponding to the version last merged/synced from Debian.

        3. new/debian: will point to debian/sid tip, in other words the commit corresponding to the version to be merged with and the latest version in Debian unstable.
      2. You may want to verify the tags (with git show <tag>) are pointing to the expected commits/versions.

    2. usd-merge reconstruct old/ubuntu

      1. This will replay the current ubuntu delta back onto old/debian, but dropping the git-merge-commits that are artifacts of the usd-import algorithm.
      2. The end result, which will be the currently checked-out commit if successful, is a sequence of cherry-picks of published Ubuntu versions after the old/debian version.
      3. This commit is tagged as reconstruct/<ubuntu version>.

    3. git rebase -i old/debian

      1. Interactively rebase the reconstructed Ubuntu delta and edit ('e' in the rebase options) each cherry-picked commit into its components.
      2. This will result in a sequence of smaller commits, ideally one per debian/changelog entry.
      3. Commit all the debian/changelog entries for one rebase step with commit message 'changelog'.
      4. Commit any debian/control metadata (VCS, Maintainer) for one rebase step with commit message 'update-metadata'.
      5. The result at the end of each rebase step is a modifications are lost/gained relative to the old commit (which is no longer applies in the current status).
      6. Repeat this for each Ubuntu version.
      7. The end-result will be a commit with a broken-out history for the latest Ubuntu version and no contentful differences to that Ubuntu version.
        1. This can be verified with git diff -p old/ubuntu.

    4. git tag deconstruct/<ubuntu version>

      1. Tag this commit for easy reference and review.
    5. git rebase -i old/debian

      1. Interactively rebase the deconstructed Ubuntu delta and drop 'changelog' and 'update-metadata' changes.
      2. These changes will be easily reproducible from this process and will lead to merge conflicts with the final rebase.
      3. The end-result will be a commit with a broken-out history for the latest Ubuntu version, but no changes to debian/changelog or to the metadata in debian/control.
        1. This can be verified with git diff -p old/ubuntu.

    6. git tag logical/<ubuntu version>

      1. Tag this commit for each reference and review.
  3. Rebase delta onto latest Debian release.
    1. git rebase -i new/debian

      1. Replay the logical delta onto the latest Debian release.
  4. Build/test.
  5. Reconstruct debian/changelog for new version.
    1. git merge-changelogs old/debian old/ubuntu new/debian

    2. git reconstruct-changelog new/debian

  6. Update metadata for new version.
    1. update-maintainer

  7. Tag, push and review.
    1. git checkout -b merge

    2. git push lpme merge

    3. Go to Launchpad page for USD git project and request a merge review.
  8. Upload.
    1. usd-merge commit ubuntu/<latest series>

    2. `git push lpusdp:<source package name>

      1. Presumes you have write rights to the USD git repository.

Tools: their usage, and where to get them

usd-import

usage: usd-import [-h] [-o LP_OWNER] [-l LP_USER] [--no-push] [--force-push]
                  [--no-clean] [-v] [-d DIRECTORY] [-P PARENTFILE]
                  package

Update a launchpad git tree based upon the state of the Ubuntu and Debian
archives

positional arguments:
  package               Which package to update in the git tree

optional arguments:
  -h, --help            show this help message and exit
  -o LP_OWNER, --lp-owner LP_OWNER
                        Which launchpad user's tree to update (default:
                        ubuntu-server-dev)
  -l LP_USER, --lp-user LP_USER
                        Specify the Launchpad user to use (default: <current username>)
  --no-push             Do not push to the remote (default: False)
  --force-push          Forcibly push all tags and branches from the import
                        (default: False)
  --no-clean            Do not clean the temporary directory (default: False)
  -v, --verbose         Increase verbosity (default: False)
  -d DIRECTORY, --directory DIRECTORY
                        Use git repository at specified location rather than a
                        temporary directory (implies --no-clean). Will create
                        the local repository if needed. (default: None)
  -P PARENTFILE, --parentfile PARENTFILE
                        File specifying parent-child relationship overrides
                        as: <pkgname> <child version> <parent1 version>
                        <parent2 version> with one override per line.
                        (default: None)

Available from Launchpad git repository.

The primary problem we are trying to solve is how to make merges consistent across the server team, and, ideally easy to review for sponsors/uploaders.

Robie Basak and others came up with a git-based workflow (documented roughly on Github and hopefully clarified here). This workflow 'just' uses git (the only addition to 'stock' git are a few commands (git-dsc-commit, git-merge-changelogs, git-reconstruct-changelog) from git://github.com/basak/ubuntu-git-tools.git)) to effectively rebase the ubuntu tip onto the latest debian tip. That presumes you spent the time yourself to create a repository with commits representing the current states of debian unstable and the ubuntu development release.

The goal for the importer is to have canonical (little c) location for such commits to live, per-package. Everyone can refer to commits and tags in that tree, and they will have well-defined semantics and share SHA1s.

A secondary problem was obtaining the 'complete' history for a given source package. This would allow a user to git-blame a give file and get useful data. So we extended the algorithm (that Robie designed) to be more flexible. After hitting many corner-cases during implementation, we scrapped our complicated algorithm that produced clean trees for a clean algorithm that produces complicated trees.

Feel free to skip the next section if you're not interested in the implementation.

Algorithmic discussion

In essence, the algorithm looks at Launchpad's publishing history for versions it hasn't seen before (which if an empty or no local repository is specified and you aren't cloning an existing repository will be all of them). For each such version, it uses pull-lp-source/pull-debian-source equivalents and git-dsc-commit to import them into the git repository. Technically it uses a lower-level command then a proper commit (git write-tree and `git commit-tree`), so that we can get the imported tree, examine it and find its parents and then commit it. The parents for an imported tree are at most 2:

  1. The last version imported into the same series/pocket, with some knowledge of how to establish a new series/pocket.
  2. The last debian/changelog entry (using debian/changelog from the just-imported tree) that was successfully imported.

We call these the 'publishing parent' and 'changelog parent' respectively. Now, if we can't find either of these, we will orphan the import and if we only find one, we'll use what we have. But the resulting tree looks like many git-merges, even where there are not ubuntu-merges. This is correct by the algorithm but can be confusing to the original git-rebase workflow, because rebase will try to replay the git-merges and that doesn't work. More on this later.

git-dsc-commit

usage: git-dsc-commit [--tree-only] DSCFILE

Untars the version specified by the DSCFILE and commits it to a git repository.

optional arguments:
  --tree-only           Only call git-write-tree on the result.
                        git-commit-tree will need to be manually
                        invoked to commit the changes to the
                        repository.

git-reconstruct-changelog

usage: git-reconstruct-changelog COMMITISH

Replays git commit messages starting after COMMITISH into debian/changelog.

git-merge-changelogs

usage: git-merge-changelogs BASE_COMMITISH NEWA_COMMITISH NEWB_COMMITISH

Runs dpkg-mergechangelogs on the debian/changelogs from the passed commitishs.

xgit

usage: xgit

Either 'enters' or 'exits' an xgit-configured git repository. Such a repository
as a git/ and gitwd/ directory structure, where git/ is the GIT_DIR and gitwd/
is the GIT_WORK_TREE. This allows for easier git-dsc-commit invocation and
clarity on what is in the working tree and what is not.

Available from Github repository.

usd-merge

usage: usd-merge [tag [-f]|reconstruct|commit] COMMITISH [ONTOISH]

Given a usd-import'd tree, attempts to reconstruct the current sequence of
commits from `git-merge-base ONTOISH COMMITISH`, where ONTOISH defaults to
debian/sid (the last imported Debian unstable version). It attempts to match
debian/changelog from COMMITISH, and verifies that the resulting commit is
content-identical to COMMITISH. It then tags the resulting commit as
reconstruct/<latest version from COMMITISH debian/changelog>.

** work in progress, the tag and commit tools are yet-to-be-written.

A simple helper script entitled 'usd-import-reconstruct-merge' [to be renamed to usd-merge] which can take a usd-import'd tree and a commitish and attempts to give you a reconstructed sequence of linear commits that represent the same state as the commitish. It does this by using merge-base to figure out the common ancestor (it assumes onto is debian/sid but it also accepts a parameter) and then playing back the debian/changelog looking for imported tags. It then cherry-picks those tags from oldest to newest against the merge-base and does a quick sanity check that the resulting commit does not differ from the original one.

It tags that as reconstruct/<version> which sort of conflicts with the git-based workflow some have been using previously. In that workflow, reconstruct/<version> is the broken-down sequence of changes including changelog and metadata, where each commit is a single change from the changelog. I have taken to now calling that pointer deconstruct/<version> because it's clever.

So you would break down reconstruct/<version> into its constituent changes (per debian/changelog for each version) and tag the resulting end-commit as deconstruct/<version>.

That can then be broken up by our workflow into a logical/<version> tag and then rebased onto debian/sid (or the specified onto).

Available from Launchpad git repository.

Example(s)

Already imported trees live, currently, at: https://code.launchpad.net/~usd-import-team/+git.

Currently, because once we feel confident in the utility and correctness of the importer, we'll move them to https://code.launchpad.net/~ubuntu-server-dev/+git.

Caveats

All of this tooling is not meant to replace or reproduce any prior tools or works. And it is definitely not meant to replace a basic understanding of what a merge is and why they are necessary. The tooling is meant to make merges easier, but they are still not foolproof and require care and thought.

UbuntuDevelopment/Merging/GitWorkflow (last edited 2021-05-24 12:21:43 by racb)