git

Revision 55 as of 2018-10-03 13:42:16

Clear message

The DesktopTeam uses git to manage its packages. Here's how to do some common things. You need git-buildpackage installed for most of this.

Suggested configuration is available in the end.

General concepts

Branches and repo layout

We will use a layout with 3 remote repositories:

$ git remote show
origin          # will point to launchpad (default for pull and push without any argument)
salsa           # will point to debian salsa repository, to cherry-pick easily fixes from debian and rebase
gnome           # will point to upstream repository, to cherry-pick easily upstream fixes

Note: we named in the page the remotes salsa and gnome, to not be mislead with local branch names (which are for instance debian/branchname and upstream/branchname) if any. Consequently, instead of having debian/master, a local branch tracking debian/debian/master remote, we end up with debian/master tracking salsa/debian/master, which is easier to understand.

As those remote name are local to your system, you can rename them as you see

Launchpad remote branches

Default

Let's look at the various branches in the origin (launchpad) remote repository:

$ git remote show origin 
* remote origin
[…]
  HEAD branch: ubuntu/master
  Local branches configured for 'git pull':
    pristine-tar    merges with remote pristine-tar
    ubuntu/master   merges with remote ubuntu/master
    upstream/latest merges with remote upstream/latest
  Local refs configured for 'git push':
    pristine-tar    pushes to pristine-tar    (up to date)
    ubuntu/master   pushes to ubuntu/master   (up to date)
    upstream/latest pushes to upstream/latest (up to date)
  • ubuntu/master branch is the content of the latest development release. Work ready to be uploaded or uploaded in dev release mostly happens here. It's the default branch.

  • pristine-tar is an internal gbp-buildpackage branch, used to reconstruct release tarballs (using upstream/latest). You are not intended to interact with it directly.

  • upstream/latest is another internal gbp-buildpackage branch. It's a merge between the upstream git branch corresponding to the latest release from the upstream repository and extra content comming from the tarball. You are not intended to interact with it directly.

We find back those 3 branches locally:

  • ubuntu/master -> pull and push from origin (launchpad) remote repository, on ubuntu/master branch (origin/ubuntu/master)

  • pristine-tar -> pull and push from origin (launchpad) remote repository, on pristine-tar branch (origin/pristine-tar)

  • upstream/latest -> pull from upstream remote repository, on the master branch (upstream/master), and push to origin (launchpad) remote repository, on upstream/latest branch (origin/upstream/master).

Basically, you will only interact with ubuntu/master under that configuration, and let gbp handling the 2 other branches. When you gbp pull and gbp push, the 3 branches will be kept up to date if no conflict occurs. This is easier than checkout every branch before pushing them.

Maintenance branches

Once we have maintenance branches on a project (ensure you already followed #Create_a_maintenance_branch), additional remote branches are available:

$ git remote show origin 
* remote origin
  […]
  Remote branches:
    pristine-tar    tracked
    ubuntu/bionic   tracked
    ubuntu/master   tracked
    upstream/latest tracked
    upstream/3.28.x tracked
  • ubuntu/bionic branch is the content for the corresponding release, once a particular release it out. It's the previous ubuntu/master branch when bionic was in development.

  • upstream/3.28.x: this branch (automatically managed by gbp buildpackage) corresponds to the upstream/latest branch, but tracking a particular upstream series, similar to the ubuntu/bionic branch. This branch is only necessary if you imported a new tarball (like 3.29 series) in 'upstream/latest and want to release an unimported yet 3.28.3 upstream release in bionic for instance.

Each one should have a corresponding local branch when working on bionic.

Debian remote branches

In addition, we'll have at least another branch tracking salsa debian remote repository. Their master branch is named debian/master:

  • debian/master -> pull (no push, unless you are a debian developer) from salsa remote repository, on debian/master branch.

Upstream remote branches

By default, we don't have upstream branches checked out locally. We added the gnome repo and have access to its content, which will be fetched when importing new upstream tarball thanks to the version tag, and inject the history in upstream/latest (or upstream/version, as explained before).

$ git remote show gnome
* remote gnome
[…]
  HEAD branch: master
[…]
    gnome-3-28                                  tracked
[…]
  Local branch configured for 'git pull':
    upstream/latest merges with remote master

You can ofc checkout any of those branch locally if you want We have upstream/master tracking the gnome repo on master branch. We can track any additional branches as needed for cherry-picking fixes.

Let's checkout locally gnome/master branch and make it the current checkout:

$ git checkout -b gnome-master gnome/master
Branch 'gnome-master' set up to track remote branch 'master' from 'gnome'.
Switched to a new branch 'gnome-master'

We asked to create a new local branch named gnome-master (first argument), tracking the remote named gnome, on the branch master (second argument).

Local branches

To sum up all this, the minimal setup it the following ([] are tracked branch):

$ git branch -vv
  debian/master   c02b29966 [salsa/debian/master] Revert "Don't recommend libnss-myhostname since it doesn't seem needed any more (LP: #1766575)"
  gnome-master    d814d9db0 [gnome/master] Update Chinese (Taiwan) translation
  pristine-tar    e59ab7422 [origin/pristine-tar] pristine-tar data for gnome-control-center_3.29.1.orig.tar.xz
* ubuntu/master   63337a4cd [origin/ubuntu/master] releasing package gnome-control-center version 1:3.29.1-0ubuntu5
  upstream/latest 0bd90ca34 [origin/upstream/latest] region: Show scrollbars if needed

With an ubuntu maintenance branch, and so, linked gbp buildpackage tarball branch, and upstream "master" branch checked out, with a local branch, we have in addition:

  gnome-master    d814d9db0 [gnome/master] Update Chinese (Taiwan) translation
  ubuntu/bionic   ba45348d3 [origin/ubuntu/bionic] debian/gbp.conf: use gbp-buildpackage for ubuntu.
  upstream/3.28.x 83b100dc9 [origin/upstream/3.28.x] New upstream version 3.28.3

Local changes

With the suggested configuration (see it at the end of the page), any non committed changes (file modifications, additions or removal) will halt the build, as they won't be included as a safety reminder.

You can either:

  • ignore them, appending --git-ignore-new. The resulting package and build won't include those uncommitted changes. (../build-area/<your-package> will match the last commit).

  • force using the current directory with your local modifications instead of build-area, and include (despite the "ignore-new" option) any local modifications to the package. For this, append --git-ignore-new --git-export-dir=""

Check out a branch for the first time

1. Launchpad remote

If the Vcs-Git field is correct, you can

$ debcheckout --git-track=* gnome-control-center
declared git repository at https://git.launchpad.net/~ubuntu-desktop/ubuntu/+source/gnome-control-center
git clone https://git.launchpad.net/~ubuntu-desktop/ubuntu/+source/gnome-control-center gnome-control-center ...
[…}

Otherwise, you can check out any git-buildpackage repository manually

$ gbp clone lp:~ubuntu-desktop/ubuntu/+source/gnome-control-center
gbp:info: Cloning from 'lp:~ubuntu-desktop/ubuntu/+source/gnome-control-center'
$ cd gnome-control-center

We want to push tags automatically, as gbp relies on them (this is a local configuration):

$ git config push.followTags true

2. Add upstream and debian remote

$ git remote add -f gnome git@gitlab.gnome.org:GNOME/gnome-control-center.git
$ git remote add -f salsa https://salsa.debian.org/gnome-team/gnome-control-center.git

You should now have 3 branches correctly setup and tracking the correct branch with expected configuration.

$ git branch -vv
  pristine-tar    e59ab7422 [origin/pristine-tar] pristine-tar data for gnome-control-center_3.28.1.orig.tar.xz
* ubuntu/master   ba45348d3 [origin/ubuntu/master] debian/gbp.conf: use gbp-buildpackage for ubuntu.
  upstream/latest 83b100dc9 [origin/upstream/latest] New upstream version 3.28.1

And 3 remotes, we only use one right now.

$ git remote
gnome
origin
salsa

3. Maintenance branches

If we want to track maintenance branches as well:

$ git remote show origin 
* remote origin
  […]
  HEAD branch: ubuntu/master
  Remote branches:
    pristine-tar    tracked
    ubuntu/bionic   tracked
    ubuntu/master   tracked
    upstream/latest tracked

Let's track ubuntu/bionic:

$ git checkout ubuntu/bionic 
Branch 'ubuntu/bionic' set up to track remote branch 'ubuntu/bionic' from 'origin'.
Switched to a new branch 'ubuntu/bionic'

Warning /!\ this short syntax is only avilable because we create a local ubuntu/bionic tracking on the origin repo the matching named ubuntu/bionic. It doesn't work with any other repository name or when name doesn't match (and need an explicit git branch -u local-name remote/branch-name && git checkout local-name).

Let's check that:

$ git branch -vv
  pristine-tar    e59ab7422 [origin/pristine-tar] pristine-tar data for gnome-control-center_3.28.1.orig.tar.xz
* ubuntu/bionic   cf931abe4 [origin/ubuntu/bionic] debian/patches/privacy-panel-whoopsie.patch
  ubuntu/master   dce53a3d2 [origin/ubuntu/master] Change gbp option order to be more meaningful
  upstream/latest 83b100dc9 [origin/upstream/latest] New upstream version 3.28.1

Day to day operation

Building a binary or source package

  • Binary package:

$ gbp buildpackage
[…]
  • Source package:

$ gbp buildpackage -S
[…]

With the proposed configuration, the artefacts will all end up in ../build-area (including tarball, which is reconstructed from the pristine-tar + upstream/latest branch) and build directory is cleaned up.

Useful tips

Some potential cases:

  • Don't purge the **build-area** directory after building: gbp buildpackage --git-no-purge

  • Build current branch with local uncommitted modifications: gbp buildpackage --git-ignore-new --git-export-dir=""

Refreshing branches

  • Use gbp pull to refresh current ubuntu/<branch> and corresponding upstream/<release> (only the ones referenced in your current checkout debian/gbp.conf) as well as pristine-tar branch.

on ubuntu/master:

$ gbp pull
gbp:info: Fetching from default remote for each branch
gbp:info: Branch 'upstream/latest' is already up to date.
gbp:info: Branch 'ubuntu/master' is already up to date.
gbp:info: Updating 'pristine-tar': 2bed57f20816..e59ab7422a13

-> the branch pristine-tar has been updated as refenced in debian/gbp.conf.

If you have a separate ubuntu/bionic branch, with its own upstream/3.28.x branch:

$ git checkout ubuntu/bionic
$ gbp pull
gbp:info: Fetching from default remote for each branch
gbp:info: Branch 'pristine-tar' is already up to date.
gbp:info: Branch 'upstream/3.28.x' is already up to date.
gbp:info: Updating 'ubuntu/bionic': fe031c0214ba..59adb2d88983

gbp pull, contrary to git push, is a way for you to avoid checkouting each branch and git pull them one by one (as you need to have the branch as a current checkout for potential conflicts when pulling).

Pushing your git changes to launchpad

Contrary for pulling, We are using git push for this. All branches tracking the origin repo will be pushed that way (no need to be on the correct checkout, contrary to gbp pull). Relevant tags as well (if you follow our advised configuration) will be pushed.

Merge a new upstream version

Master branch

On the branch you want to have version imported (like ubuntu/master here).

As a first part, you need to create a patch/queue/ubuntu/master branch with gbp pq that is useful in case you need to refresh patches later. If you have it already, just rebase it.

# Create a patch/queue branch that will be useful for refresh patches later
$ gbp pq import # if you already have it, just use `gbp pq rebase instead`

Using upstream tarball

You need to follow this path in the case debian has not imported this version yet, although since we don't want to have duplicated upstream/x.y.z tags, in this case you should push the pristine-tar and upstream/<current> branches to salsa

$ git fetch gnome
$ git checkout ubuntu/master
$ gbp import-orig ../gnome-control-center-3.28.2.tar.xz # can be as well --uscan to let it download and import
What is the upstream version? [3.28.2] 
gbp:info: Importing '../gnome-control-center-3.28.2.tar.xz' to branch 'upstream/latest'...
gbp:info: Source package is gnome-control-center
gbp:info: Upstream version is 3.28.2
gbp:info: Replacing upstream source on 'ubuntu/bionic'
gbp:info: Successfully imported version 3.28.2 of ../gnome-control-center-3.28.2.tar.xz
$ git push

This will push all needed branches (ubuntu/master for instance and pristine-tarball + upstream/3.28.x or upstream/latest if this upload is a new upstream release).

As said, now you should propose or ask someone to sync the affected upstream branch to salsa.debian.org (making sure you push the new tag too).

Note: If you get when importing:

$ gbp import-orig ../gnome-control-center-3.29.2.tar.xz
What is the upstream version? [3.29.2] 
gbp:info: Importing '../gnome-control-center-3.29.2.tar.xz' to branch 'upstream/latest'...
gbp:info: Source package is gnome-control-center
gbp:info: Upstream version is 3.29.2
gbp:error: Import of ../gnome-control-center-3.29.2.tar.xz failed: revision '…' not found

There are 3 cases:

  • You didn't run git fetch gnome and so, didn't get latest commits and tags in your repository metadata.

  • Upstream has changed the tagging pattern (and so, debian/gbp.conf needs to be updated to match it)

  • Upstream is using an inconsistent release pattern (and so, debian/gbp.conf can't have a regexp for the version). In that case, please specify manually --upstream-vcs-tag=exact_version_tag to gbp import-orig. You can find the exact tag via git describe --tags --abbrev=0 gnome/master.

Another error you could get is:

$ gbp import-orig ../gnome-control-center-3.29.2.tar.xz
What is the upstream version? [3.29.2] 
gbp:error: Upstream tag 'upstream/3.29.2' already exists

In this case, debian has already this tarball so you should follow the next chapter ("Using debian's upstream branch tags").

Using debian's upstream branch tags

If debian has already this version, you don't have to import the actual tarball but just merge with the latest upstream code, and so:

git merge upstream/3.28.2 -m "Update upstream source from tag 'upstream/3.28.2'"

Maintenance branch

There could be some additional steps involved, if master does have a newer version than the maintenance branch, and you are the first to deal with that case for that maintenance release.

Basically if debian/gbp.conf has upstream-branch=upstream/latest and the ubuntu/master branch has a newer upstream version than the one you are importing (which was never imported), please read this. Otherwise, just follow the steps in the previous case, after eventually creating a maintenance branch (see corresponding section) and switching to it.

For instance, let's say: - ubuntu/master is on 3.29.2 - ubuntu/bionic is on 3.28.1 and we want to update to 3.28.2 - we have pristine-tar having 3.28.1 and 3.29.2 imported - consequently the upstream/latest branch has upstream commits and tags for 3.28.1, 3.29.2 and corresponding gbp tags upstream/3.28.1 and upstream/3.29.2.

Ensure you are on your ubuntu/bionic branch.

1. Create a new upstream/<series> branch

Note that you need to check latest version of upstream/latest in your maintenance branch (upstream/3.28.1 in our case).

$ git branch upstream/3.28.x upstream/3.28.1

2. Update debian/gbp.conf to reference correct upstream gbp branch

# edit debian/gbp.conf and replace with '''upstream-branch=upstream/3.28.x'''
$ git commit -a

Using upstream tarball

3a. Then if this revision is not in salsa upstream branch yet

$ git fetch gnome
$ gbp import-orig ../gnome-control-center-3.28.2.tar.xz

3b. Remember pushing your changes, including the new branch that you want to track

$ git push
$ git push -u upstream/3.28.x

Using debian's upstream branch tags

3a. Then if this revision is already in salsa upstrema branch

git merge upstream/3.28.2 -m "Update upstream source from tag 'upstream/3.28.2'"

4. Remember to push the upstream and pristine-tar branches to salsa

You can push to your guest account and prepare a pull-request or directly on the gnome branch if you have the right.

Refreshing patches

When merging with an upstream release, the patches might need to be refreshed. In case your patch/queue branch is in sync with previous release you can just use:

gbp pq rebase

# Then use git rebase tools to refresh them

This is described better, for the case of using gbp import-orig in gbp documentation. This might won't work if you're instead merging with an upstream/x.z.y tag or if you didn't create the patch/queue branch first, so let's just write here one that should work in both cases, so you don't have to worry about too much:

# on corresponding branch…

# Remove the patch-queue branch, if any... Not needed if you don't have one
gbp pq drop

# This is a brute-force approach that will try to find the first point where the patches merge,
# so you can tune it increasing until when you won't find the previous release commit
gbp pq import --force --time-machine=30

# This will re-apply the debian/patches/series on top of new upstream code stopping for manual
# action if is needed.
gbp pq rebase

# If a patch doesn't apply cleanly, manual intervention is needed. In such case, just use
# classic `git add`, `git rm` to resolve the conflicts, and eventually just proceed with
# `git rebase --continue`

# Regenerate debian/patches data, ready to be committed and switch back to your ubuntu/ branch
gbp pq export --no-patch-numbers

Once back on ubuntu/master or ubuntu/<series> branch, you need to commit the changes too.

Release a package

If this is a sponsored upload, the sponsor performs these steps.

Generate changelog for native packages

# Add debian/changelog entries based on git log
gbp dch

# You might need to manually edit debian/changelog
# to properly group changes per people

Finalise changelog

# on corresponding branch…
$ dch -r ""
$ git commit debian/changelog -m "Finalise changelog"
$ gbp buildpackage --git-tag-only
$ gbp buildpackage -S
$ dput ../build-area/….changes
$ git push

# Make sure you also pushed the tag(s) we care about:
$ git tag | grep -E '^ubuntu/|^debian/|^upstream/' | xargs -r git push

This will push all tracked branches to launchpad if you made any change: ubuntu/master, ubuntu/old-series, pristine-tar, upstream/lastest. Tags will also be pushed.

Working on patches

1. Ensure you are on the correct ubuntu/ branch (or your local experimental-feature branch).

$ git checkout ubuntu/master # for instance

2. Turn each patches in debian/patches as commits.

$ gbp pq rebase
gbp:info: Switching to 'patch-queue/ubuntu/master'
Current branch patch-queue/ubuntu/master is up to date.

-> Now, we have every patch referenced in debian/patches/series applied as separate commits on top of ubuntu/master, in a patch-queue/ubuntu/master branch, and we switched to that current checkout.

For instance

$ git missing ubuntu/master.. # git log would also show the commits
* 11a3a8cc6 - (HEAD -> patch-queue/ubuntu/master) [PATCH] night-ligth-dialog: Avoid dereferencing invalid p
ointer (5 minutes ago)
* da06252e1 - [PATCH 4/4] thunderbolt: move to the 'Devices' page (5 minutes ago)
* 2c9f5bcbb - [PATCH 3/4] thunderbolt: new panel for device management (5 minutes ago)
* 660e9e633 - [PATCH 2/4] shell: Icon name helper returns symbolic name (5 minutes ago)
* a78ae89dd - [PATCH 1/4] shell: Don't set per-panel icon (5 minutes ago)
[…]

3. Hack…

We can git add/git commit. Any new commit will be a separate patch applied at the end of debian/patches/series. Commit description will be then converted to the patch description.

Remember previous tip of gbp buildpackage --git-ignore-new --git-export-dir="" to build a package with current content without having to switch branch.

4. optional: Reorder patches

If we don't want this patch to be the last one, we can use an interactive git rebase:

$ git rebase -i ubuntu/master

That way, we only see the patches commit, can reorder them, amend or stash them. Removing one will also remove the patch from debian/patches/series

5. Reapply all changes to the original branch:

$ gbp pq export --no-patch-numbers
gbp:info: On 'patch-queue/ubuntu/master', switching to 'ubuntu/master'
gbp:info: Generating patches from git (ubuntu/master..patch-queue/ubuntu/master)

6. Modify changelog, git add, git commit

Note that gbp pq will unfuzz a lot of patches everytime you use it, creating some noise. Please add those changes separated from your new or modified patch hack.

And either push the change to your branch or to the main branches with a simple git push!

Pick some upstream commits

1. Refresh upstream repo if needed:

Remember that we don't have a local checkout, only a remote that we need to refresh

$ git fetch gnome

2. Find the commit(s) you want to cherry-pick

Look at log history with, for instance:

$ git log -p gnome/master

3. Cherry-pick one commit as a patch (here on ubuntu/master branch):

$ git checkout ubuntu/master
$ gbp pq rebase
$ git log -p upstream/latest
# get a hash commit you want to cherry-pick
$ git cherry-pick -x <hash>
$ gbp pq export --no-patch-numbers
# update changelog
$ git add debian/patches/* debian/changelog
$ git commit

See previous section for more info on gbp pq.

Import an out of VCS ubuntu upload

On the branch you want to have that ubuntu upload, after downloading the source tarball:

$ gbp import-dsc ../gnome-control-center_3.28.1-0ubuntu5.dsc
gbp:info: Version '1:3.28.1-0ubuntu5' imported under '…/gnome-control-center'
$ git push

This will push all needed branches (ubuntu/master for instance and pristine-tarball + upstream/latest if this upload is a new upstream release).

Out of VCS upload while having changes in tree

Warning /!\ importing take latest tarball and will eventually revert any changes that were sitting in the VCS. You have 2 strategies:

  • rewriting the history (but everyone who pulled the repo will have conflicts with refreshing) via a rebase. This is thus discouraged.
  • applying the latest source package, and reintroducing the changes (cherry-picking the commits). Note that as your commits are already in tree (but reverted), git cherry-pick <hash> won't work. You should rather:

$ git show <hash> | git apply
$ git commit -C <hash>

Create a maintenance branch

1. Find correct version

Fin latest version in common between the development release and that maintenance branch. Here, we'll use the version tag ubuntu/1%3.28.1-0ubuntu4

2. create a branch from start starting point, and push it to launchpad

$ git branch ubuntu/bionic ubuntu/1%3.28.1-0ubuntu4
$ git checkout ubuntu/bionic 
Switched to branch 'ubuntu/bionic'

3. Adjust parameters in debian/gbp.conf and debian/control.

# Change `debian-branch` to '''ubuntu/bionic''' in debian/gbp.conf
# Change Vcs-Browser to the branch's page on git.launchpad.net
# Add `-b '''ubuntu/bionic'''` at the end of Vcs-Git
$ git commit -a

4. Push your changes to launchpad and track that branch

$ git push -u origin ubuntu/bionic 
Total 0 (delta 0), reused 0 (delta 0)
To git+ssh://git.launchpad.net/~ubuntu-desktop/ubuntu/+source/gnome-control-center
 * [new branch]          ubuntu/bionic -> ubuntu/bionic
Branch 'ubuntu/bionic' set up to track remote branch 'ubuntu/bionic' from 'origin'.

Merge with debian

TODO

Convert a package to git for the first time from Debian

This section has been moved to DesktopTeam/git/NewPackageImport.

Configurations

~/.gitconfig

To easily access to launchpad git repositories with git clone lp:~name-or-team/repo, you should add to your .gitconfig file the following paragraph:

[url "git+ssh://<your_user_name>@git.launchpad.net/"]
        insteadof = lp:

See https://help.launchpad.net/Code/Git for more information on git configuration with launchpad.

~/.gbp.conf

git-buildpackage uses this file for its default configuration; some tweaks are recommended.

[buildpackage]
# use this for exporting first your directory in a ../build-area:
export-dir = ../build-area/
# automatically GPG sign tags with associated keyid
sign-tags = True
keyid = 0xGPGKEYID

See gbp.conf(5) and man of gbp-buildpackage for more options.

Annex

Work to be done:

  • https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888313

  • Talk with upstream about preventing importing older tarball in upstream/latest and handling creation of upstream/maintenance-branch
  • Ensure that if no upstream tag is in debian/gbp.conf and not found in repo, erroring out when importing a tarballs.