Overview

The knowledge base contains development, test, and operational resources specific to the Ubuntu Server team. If you have any questions, don't hesitate to contact other ServerTeam members.

Bug Triage

Goal: To successfully review every bug filed against Ubuntu Server related packages

A review involves analyzing a bug to determine if the bug is valid and if sufficient information was provided. If the bug is both valid and provided with sufficient information, the bug is marked as triaged and will be worked to closure by a member of the server team. Otherwise, the bug will be responded to and marked as 'Incomplete' for more details, 'Invalid' for not a real bug, or 'Won't Fix'. In certain (rare) cases bugs might stay in new/confirmed which reflects we need to look into it again more deeply to triage in a better way.

This is a list of the various queues of interest to the server team. They are a good place to go if you are looking for a good "what to do next" bug.

• Top 20 Bugs ("server-next")

• Bite-sized Bugs

• Recently modified Bugs

• Full Backlog

Daily Bug Triaging

Bug triage is completed for all bugs last updated on a particular day. An assigned member of the server team will look at all bugs that were updated on the previous day. For example, the member with responsibility on Friday will review all the bugs updated on Thursday. For the weekend, the member with responsibility on Monday will review all the bugs updated on Friday, Saturday, and Sunday.

This process is expected to take less than 90 minutes per day. This is not meant to be a full root cause analysis (RCA) investigative time, instead only determining if further attention is warranted and sufficient information has been provided.

If a bug is considered to be a benefit to the majority of users and needing further attention by the Ubuntu Server Team the triager subcribes ~ubuntu-server to it. If possible a task is scheduled e.g. on https://trello.com/b/U9HhWyT0/daily-ubuntu-server for the team or by the triager subscribing/assigning himself to look into it later if possible. Please do note that the classic consideration of a bug being "actionable" is flagged place in the launchpad bug status. We might in some cases e.g. only want to track the bug and therefore subscribe ~ubuntu-server. But On the other hand a triager has to be clear that a totally un-actionable bug is likely to appear as expiring 180 days later, so if possible drive the bugs further via clarifications, questions, more bug tasks, subscribing subject matter experts and so on.

Current members of Ubuntu Server bug triage: https://launchpad.net/~ubuntu-server-active-triagers/+members

Direct team subscriptions

We subscribe ~ubuntu-server directly to a bug to track our community bug backlog when the bug meets the following criteria. When the bug no longer meets these criteria, we unsubscribe it:

1. Anything that, if the bug turns out to valid, is something that would be under the ~ubuntu-server remit to fix (common use cases but not obscure ones - but nothing stops an individual volunteering to work on an obscure use case, of course).
2. By definition, if it's something that we wouldn't fix and request volunteers even if we had time, then it doesn't warrant a subscription.
3. This subscription is for the Ubuntu Server triage community and is not for tracking of internal Canonical customer requests. Whether a Canonical customer has made a request in relation to a particular bug makes no difference and provides no additional priority under this process. A Canonical customer bug may still be subscribed if it qualifies under these criteria.
4. If the bug is assigned to someone on our team, leave it subscribed. No need to subscribe, and feel free to unsubscribe.

server-next

Since the backlog is bigger than what can be achieved in a short time, there is the extra classification via the tag server-next. That tag is set by the Triager (or anyone else working on doing the Root-Cause-Analysis or a Fix) to reflect that this is an issue that shall be tackled by the Teams resources "next".

Another reason to add server-next in some cases is to preserve high quality contributions of the community. An example might be a report that the user already bisected and created a patch for - in those cases the benefit diminishes by bit rot way too fast, so handling that next helps to retain the work the reporters did. And vice versa it might encourage one or the other to provide more high quality bugs.

The goal is to have this list around ~20 bugs most of the time, if dropping below we can refill with candidates from the ~ubuntu-server subscribed bugs. But if it grows significantly out of this range it is non-realistic to expect those issues to be handled in time, we should communicate so to the reporters.

The rules of the server-next tag are as follows:

1. Must not tag unless bug is actionable. Doesn't mean it must have a patch, only that a developer has enough information to work on the bug, even if it means more debugging.
2. Tag only if one of these two things are true:
   1. Delays will discourage this excellent community contribution.
   2. If you believe it affects a major use case for Ubuntu server users. In this case you should also set the bug Importance.
3. The set of all bugs tagged server-next must be kept small. If it grows stuff the lowest priority bugs tagged server-next must be removed until the list isn’t too big.
4. This tag is for the Ubuntu Server triage community and is not for tracking of internal Canonical customer requests. Whether a Canonical customer has made a request in relation to a particular bug makes no difference and provides no additional priority under this process. A Canonical customer bug may still be tagged if it qualifies under these criteria.
5. If the bug is assigned to or otherwise owned by someone on our team, there is no need to tag it.
6. Remove the tag when the bug is assigned to or otherwise owned by someone on our team.

Other metadata

• Use the Ubuntu-wide bug milestones to add a bug to a list to consider fixing for a particular point in the development release, or a particular SRU point release.

• Use the standard Ubuntu-wide bitesize tag to mark a bug as needing only a small and simple amount of work to fix.

Daily Bug Expiration

There are two levels of expiration. The tooling will help to report these to the Triager. Eventually we want those lists of expiring bugs to be zero, and if they are not at least clear them on the day they expire.

Note: especially initially these lists are rather huge and working through all of them takes various subject matter expertise, any help getting those re-triaged is highly appreciated.

• Server-next expiration - default after 60 days

  • If we considered a bug actionable and added it to server-next, but then no update happened in 60 days that usually means something went wrong. Often bugs are blocked on external constraints. This needs to be evaluated as a case-by-case decision. Most common cases are, that it turns out:

  • that the bug is not solvable/reasonable the way it was planned -> re-triage, maybe drop server-next

  • that it is actually fixed or otherwise progressed without update -> update bug

  • that we failed to give it the required focus -> bring up in IRC and the https://trello.com/b/U9HhWyT0/daily-ubuntu-server, search for Developers who could assist, post excuse and explanation on the bug

• Server subscription expiration - default after 180 days

  • If nobody touched a bug for 180 days (~= 1 release cycle) it is reasonable to check for changed conditions. Quite often e.g. a patch one was waiting on is available now. In other cases a newer release fixed it already. Essentially anything that is listed here needs to be fully re-triaged to ensure the list is reflecting the current status. It also can after this time be used as a metric how many more people chimed in got dupped on the bug (importance/#affected). Most common cases are, that it turns out:

  • that recent releases upstream or even already in Ubuntu have the fix -> re-triage, consider tagging server-next for SRU

  • that the bug should have been supported by the community but nothing happened -> re-triage importance, consider dropping ~ubuntu-server subscription

  • that a bug that was formerly considered a real case is not qualifying anymore (e.g. alternative solutions

    • have taken hold as "the" way to do it) -> re-triage importance, consider dropping ~ubuntu-server subscription

Overall for all of these we have to be honest to the bug reporter, try to understand why an issue was not worked on and explain it if possible. Also if we drop server-next or the ~ubuntu-server subscrription for any of the reasons above always add a explanatory comment. If reporters disagree with our re-triage they will report on the bug and it will show up in the daily triage duty the next day to be reconsidered with that point of view taken into consideration.

Triage Task Assignment

Assignment of daily bug triage is done via a recurring Trello board card. Mostly each of us has a fix day per week, and we share Monday (which are all reports Fri-Sun) on a rotation.

Tooling

To assist with the triaging there is https://github.com/powersj/ubuntu-server-triage/. By default it will list the bugs of the current days Triager, but it can be controlled via commandline arguments. That way one can easily do tasks like "last wednesdays tirage duty" if one was away a few days.

Furthermore it will list the expiring bugs that should be re-triaged.

Additional Resources

Helpful Guides and Definitions:

• Bug Triage Workflow

• Bug Triage Tags and Subscriptions

• Definitions for bug importance levels

• Definitions for bug status settings

• Server specific triage responses

• Additional predefined response templates

Merge Proposals and Reviewing

We are transitioning to a git-based workflow for handling package changes. This also allows us to use Launchpad Merge Proposals and request Reviews before publishing a new package version.

Requesting a review

There are three review teams within canonical-server:

• canonical-server-core-reviewers: members are Ubuntu Core Developers and can upload any package

• canonical-server-motu-reviewers: members are Ubuntu MOTU developers and can upload to Universe

• canonical-server-packageset-reviewers: members are Ubuntu Server Developers and can upload Server related packages

To find out which team should review your branch, use the ubuntu-upload-permission tool from the ubuntu-dev-tools package and select the most specific team:

$ ubuntu-upload-permission --list-uploaders <package>

A second review slot should always be requested for the canonical-server team, as we want the merge proposal to always show up in the active reviews page for the canonical-server team regardless if a review slot was taken or not. Launchpad bug #1708172 was filed about it, but for now we will use this workaround.

All this translates to the following git ubuntu command:

$ git ubuntu submit --reviewer canonical-server --reviewer <reviewer team> --target-branch <target>

In this command, <target> depends on which Ubuntu release you are targeting:

• ubuntu/devel: for the current development version of Ubuntu

• ubuntu/<name>-devel: for the <name> Ubuntu release. For example, if preparing an SRU for Xenial, the target reference path would be ubuntu/xenial-devel.

Alternatively, you can go to https://launchpad.net/people/+me/+git, select your repository, a branch within, and then "propose for merging". The details to be filled out are:

• target repository: that will be lp:~usd-import-team/ubuntu/+source/<sourcepackage>

• target reference path: the same as <target> in the git ubuntu submit command above

• reviewer: canonical-server-core-reviewers, canonical-server-motu-reviewers or canonical-server-packageset-reviewers. This depends on the package, see previous discussion about using the ubuntu-upload-permission tool to find out.

After proposing the merge, request a second reviewer slot for canonical-server and you are set.

Eventually you will get review comments. These can of a few types:

• just a comment. Can also be a non-formal review from someone outside the server team.
• approval (also known as a +1)
• needs information: the reviewer didn't understand something, or wants clarification
• needs fixing: the reviewer didn't agree with something and is requesting that it be changed
• disapprove: the approach to the problem is incorrect (also known as a -1)
• abstain (also known as a +0)
• resubmit: you should resubmit the proposal, possibly because the package has changed while this review was up

If you need to work on something in your merge proposal, you should set its status back to "work in progress" to signal others that you are addressing the issue. This will have the effect of removing the proposal from the queue and then people can concentrate on reviewing other branches while you work on yours. If you have Server Trello board rights, you should also move the review card back to the "Doing" list. Once you are done, place it again in the "Review Requests" list and change the status of your MP to "needs review".

Reviewing a Merge Proposal

First of all, never claim the canonical-server review slot. That slot should remain unclaimed as its sole purpose is to facilitate viewing all merge proposals in a single page. You should only claim the review slot of one of the canonical-server reviewers teams. You are welcome to add a review comment anytime, however.

If you have Server Trello board rights, once the review is claimed, move the corresponding Trello card in the server board from the "Review Requests" list to the "Review-Doing" one.

Developer & Packaging Resources

We are focusing on server related packages in main and universe. Developers can use the Triaged Ubuntu Server bugs list to prioritize their work.

For packaging information, head to the MOTUs, the Master Of The Universe.

• There is the PackagingGuide.

• PackagingGuide/Lists/DocumentationResources and MOTU/School have information related to packaging.

• UbuntuDevelopers explains how to become an official packager.

• ubuntu-motu mailing list and #ubuntu-motu on irc.freenode.org are good places to ask for help.

For Ubuntu release specific resources, the Ubuntu Team wiki is the central location where Ubuntu developers exchange ideas and track their progress.

• UbuntuDevelopment gives an overview of the development processes.

• The ubuntu-devel mailing list and #ubuntu-devel on irc.freenode.net are the places where ubuntu developers can be found.

IRC

The server team utilizes IRC to offer support for server-related questions. The team sits on freenode in the #ubuntu-server channel.

The ubottu IRC bot makes it easy to share an extensive set of factoids to others in an IRC channel. E.g. typing !ask | noobie will cause ubottu to tell noobie that folks should just go ahead and ask their questions. Ubottu can also conveniently show the channel information on bugs and packages. See ubottu for more details.

Testing

For an overview of our test areas and opportunities please see the server testing section of the overall Ubuntu testing project.

Documentation

This area is involved with updating and creating new content for the Ubuntu Server Guide and the community help website. We are working with the DocumentationTeam and focus on server related topics.

Team policy

Membership

The Membership policy is described in Membership.

Reporting

The ServerTeam has a section in the monthly report. We try to get status reports on a weekly basis on the day preceding the IRC meeting. The ReportingPage is used to gather the outcome of the tasks done by the ServerTeam members during the week.

The monthly report is a subpage under ServerTeam/ReportingPage. It's a summary from the Meeting minutes and the "a Month in the archive" post.

The subpage is automatically included in the monthly team report with a macro as defined in the ServerTeam wiki page.

IRC Meeting

We hold IRC meeting regularly to report about current tasks and define new ones. The Meeting page presents the Agenda for the next meeting.

irclogs are available on http://irclogs.ubuntu.com/.

Publishing the Minutes

Once the meeting is over, minutes are prepared to summarize the outcome of the meeting.

1. Go to MeetingLogs/Server, edit the page with a link to the logs

2. Update ServerTeam/Header to announce the next meeting date.

3. Update the Agenda for the next meeting at ServerTeam/Meeting by removing completed ACTIONS and adding new action items

4. Update the list of chairs by moving yourself to the back and send an email to the person who will chair next week

5. Publish the minutes to the ubuntu-devel <ubuntu-devel@lists.ubuntu.com> and ubuntu-server <ubuntu-server@lists.ubuntu.com> mailing lists. Use the subject "Server team meeting minutes: YYYY-MM-DD"

CategoryServerTeam