Coverity

Differences between revisions 4 and 9 (spanning 5 versions)
Revision 4 as of 2012-02-08 21:42:04
Size: 5132
Editor: c-98-212-2-38
Comment:
Revision 9 as of 2015-07-08 16:02:36
Size: 7660
Editor: 1
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
The product strategy team in side of Canonical has licensed [[http://coverity.com/|Coverity]] to help increase the quality of various projects that we are working on. Each commit to the trunk of the projects that fall under the Unity umbrella will be scanned by the [[http://coverity.com/products/static-analysis.html|Coverity Static Analyzer]] to determine defects in the codebase. These defects can range from NULL pointer dereferencing to more complex memory issues. The product strategy team within Canonical has licensed [[http://coverity.com/|Coverity]] to help increase the quality of its products. Each commit to the trunk of the projects that fall under the Unity umbrella will be scanned by the [[http://coverity.com/products/static-analysis.html|Coverity Static Analyzer]] to determine defects in the codebase. These defects can range from NULL pointer dereferencing to more complex memory issues.
Line 5: Line 5:
Coverity is a proprietary tool that we unfortunately can't distribute to the community, but we have [[https://launchpad.net/coverity-launchpad-sync-tool|built tools]] to publish the information that Coverity provides in [[http://launchpad.net|Launchpad]] so that everyone in the greater Unity community can see the results of the runs. We expect both Canonical developers and community developers' primary interaction with Coverity to be through the Launchpad bugs, but Canonical developers do have access to the server itself. If someone in the community does need more information that isn't exported on a bug, please contact a Canonical developer and we'd be happy to get that for you. If you believe that a piece of information should consistently be exported and you know Python, patches to the [[https://code.launchpad.net/~coverity-launchpad-sync-tool-team/coverity-launchpad-sync-tool/trunk|sync script]] are welcome as well. Coverity is a proprietary tool that we unfortunately can't distribute to the community, but we have [[https://launchpad.net/coverity-launchpad-sync-tool|built tools]] to publish the information that Coverity provides in [[http://launchpad.net|Launchpad]] so that everyone in the greater Unity community can benefit from its analysis. We expect both Canonical developers and community developers' primary interaction with Coverity to be through the Launchpad bugs, but Canonical developers do have access to the server itself. If someone in the community does need more information that isn't exported on a bug, please contact a Canonical developer and we'd be happy to get that for you. If you believe that a piece of information should consistently be exported and you know Python, patches to the [[https://code.launchpad.net/~coverity-launchpad-sync-tool-team/coverity-launchpad-sync-tool/trunk|sync script]] are welcome as well.

Coverity scans projects during two distinct phases in the Ubuntu developer workflow:

 * Merge proposals generate a "Needs Fixing" review if a significant defect is discovered.
 * Defects discovered in the trunk of a project generate Launchpad bugs.

== Merge proposal reviews ==

When a merge is proposed to a project, a Coverity scan is administered by Jenkins, the Product Strategy continuous integration service. If a defect of a pre-set "impact" or higher is found, Jenkins registers a "Needs fixing" review and offers a pretty-printed report of the source file with the defect highlighted.
Line 9: Line 18:
Coverity tracks a specific issue through a variety of transformations that it could go through. This includes through difference series of the code base, or as the code moves throughout the project. It also merges defects from discrete series into a single bug. When a commit occurs to the trunk of a project, a Coverity scan is administered by Jenkins. If a defect is found, Jenkins enters a bug in Launchpad against the project. The Launchpad bug is given attributes as provided by the Coverity scanner, to assist the developer in fixing and to aid in categorization, etc.

Note that Coverity tracks a defects through a variety of transformations: through difference series of the code base, or as the code moves throughout the project. It also merges defects from discrete series into a single bug.
Line 16: Line 27:
 * '''coverity-''checker_name''''' -- there are a variety of different checks that the analyzer does, and each of them is named by the type of error that it finds. These tags mark the type of error that Coverity has found and can be used for filtering bugs.  For more information on '''checkers''' see the [[ CoverityCheckerDictionary | dictionary of checkers ]].  * '''coverity-''checker_name''''' -- there are a variety of different checks that the analyzer does, and each of them is named by the type of error that it finds. These tags mark the type of error that Coverity has found and can be used for filtering bugs.
Line 20: Line 31:
We use the description field to give quick information about the issue that Coverity found and some structured data to be able to track back to the Coverity tracker.  One of the fields there is the CID, which is the Coverity Unique ID in the Coverity tracker. This is irrelevant except for tracking back to the Coverity database, which could sometimes be required. We use the description field to give quick information about the issue that Coverity found and some structured data to be able to track back to the Coverity tracker.

 * '''
CID''': the Coverity Unique ID in the Coverity tracker. This is irrelevant except for tracking back to the Coverity database, which could sometimes be required.
 * '''Checker''': the ''''checker'''' Coverity has used to find this defect.
 * '''CWE Definition''': a link to the defect's [[ http://cwe.mitre.org/index.html | CWE/MITRE definition ]].
 * '''File''': the source file in which the defect was found.
 * '''Function''': the function in which the defect was found. Note that not all defects are found in a proper function.
 * '''Code Snippet''': an excerpt from the source-file interleaved with Coverity's defect-report.
Line 28: Line 46:
The priority is the bug is set initially by the script using a set of metrics based on the checker as provided by Coverity. Mostly this relates to the severity of what happens when the bug is hit. The Coverity sync script does not change the severity if it is modified by developers, it is expected they may reprioritize based on area of code or whether this bug is important to the development team.  More information can be found in the [[ CoverityCheckerDictionary | dictionary of checkers and their severities ]]. The priority is the bug is set initially by the script using a set of metrics based on the checker as provided by Coverity. Mostly this relates to the severity of what happens when the bug is hit. The Coverity sync script does not change the severity if it is modified by developers, it is expected they may reprioritize based on area of code or whether this bug is important to the development team.
Line 37: Line 55:

== Workflow ==

Developers and managers will use Launchpad bugs to track Coverity defects. Coverity scans will run as part of a Continuous Integration process, and defects will be submitted to Launchpad from Coverity by a '''sync script''' (described below). We envision a few different paths for a Coverity defect through the Ubuntu developer process--in detail:

 * New -> Confirmed -> In Progress -> Fix Committed -> Fix Released
   The vanilla workflow: Coverity discovers a defect and reports it as a bug in Launchpad. A developer fixes the bug as part of his/her normal workflow, ultimately merging his/her branch into trunk. When Coverity fails to find this defect in a new scan, the bug is marked "Fix Committed". When the fixed code is released, a release manager marks the bug as "Fix Released".
 * New -> Confirmed -> In Progress -> New
   Coverity discovers a defect and reports it as a bug in Launchpad. A developer fixes the bug as part of his/her normal workflow, ultimately merging his/her branch into trunk. Coverity finds that the defect still exists, and re-opens the Launchpad bug by re-setting its status to "New", and the triage/fixing process begins again.
 * New -> Invalid/Wontfix
   Coverity discovers a defect and reports it as a bug in Launchpad. A developer believes the bug to be "Invalid" or marks it "Wontfix". Coverity's successive discovery of the same defect won't result in a new bug report.

Introduction

The product strategy team within Canonical has licensed Coverity to help increase the quality of its products. Each commit to the trunk of the projects that fall under the Unity umbrella will be scanned by the Coverity Static Analyzer to determine defects in the codebase. These defects can range from NULL pointer dereferencing to more complex memory issues.

Coverity is a proprietary tool that we unfortunately can't distribute to the community, but we have built tools to publish the information that Coverity provides in Launchpad so that everyone in the greater Unity community can benefit from its analysis. We expect both Canonical developers and community developers' primary interaction with Coverity to be through the Launchpad bugs, but Canonical developers do have access to the server itself. If someone in the community does need more information that isn't exported on a bug, please contact a Canonical developer and we'd be happy to get that for you. If you believe that a piece of information should consistently be exported and you know Python, patches to the sync script are welcome as well.

Coverity scans projects during two distinct phases in the Ubuntu developer workflow:

  • Merge proposals generate a "Needs Fixing" review if a significant defect is discovered.
  • Defects discovered in the trunk of a project generate Launchpad bugs.

Merge proposal reviews

When a merge is proposed to a project, a Coverity scan is administered by Jenkins, the Product Strategy continuous integration service. If a defect of a pre-set "impact" or higher is found, Jenkins registers a "Needs fixing" review and offers a pretty-printed report of the source file with the defect highlighted.

Bugs in Launchpad

When a commit occurs to the trunk of a project, a Coverity scan is administered by Jenkins. If a defect is found, Jenkins enters a bug in Launchpad against the project. The Launchpad bug is given attributes as provided by the Coverity scanner, to assist the developer in fixing and to aid in categorization, etc.

Note that Coverity tracks a defects through a variety of transformations: through difference series of the code base, or as the code moves throughout the project. It also merges defects from discrete series into a single bug.

Tags

We're using a variety of bug tags to give extra information about the bug and how it relates to Coverity.

  • coverity -- this tag marks the bug as being found by the Coverity static analyzer

  • coverity-checker_name -- there are a variety of different checks that the analyzer does, and each of them is named by the type of error that it finds. These tags mark the type of error that Coverity has found and can be used for filtering bugs.

Description

We use the description field to give quick information about the issue that Coverity found and some structured data to be able to track back to the Coverity tracker.

  • CID: the Coverity Unique ID in the Coverity tracker. This is irrelevant except for tracking back to the Coverity database, which could sometimes be required.

  • Checker: the 'checker' Coverity has used to find this defect.

  • CWE Definition: a link to the defect's CWE/MITRE definition.

  • File: the source file in which the defect was found.

  • Function: the function in which the defect was found. Note that not all defects are found in a proper function.

  • Code Snippet: an excerpt from the source-file interleaved with Coverity's defect-report.

Status

The Coverity sync tool will create new bugs with status of "New". It is expected that the developers on the project will adjust the status to track the bug all the way through to the "In Progress" state. When there is a commit to the trunk of the project and Coverity is able to verify the fix it will set the state to "Fix Committed". It is expected that the release manager for the project would then move the status of the bug to "Fix Released" when the bug is released by the project. If Coverity detects that a bug has reemerged in the codebase it will change it from "Fixed *" back to "New."

Severity

The priority is the bug is set initially by the script using a set of metrics based on the checker as provided by Coverity. Mostly this relates to the severity of what happens when the bug is hit. The Coverity sync script does not change the severity if it is modified by developers, it is expected they may reprioritize based on area of code or whether this bug is important to the development team.

Attachments

On each bug there are also several attachments which provide more information about the error that Coverity has found. This can include things like extra information on which conditionals are being used and where there are perhaps other errors that may contribute. These are largely free form in their description of the error and designed to provide more information to the developer. Individual developers may prefer to use this information differently.

Tasks and Series

Coverity will look to understand which series of the project that it is parsing and can link the various defects across those series. It will maintain the status on the series target for the bug.

Workflow

Developers and managers will use Launchpad bugs to track Coverity defects. Coverity scans will run as part of a Continuous Integration process, and defects will be submitted to Launchpad from Coverity by a sync script (described below). We envision a few different paths for a Coverity defect through the Ubuntu developer process--in detail:

  • New -> Confirmed -> In Progress -> Fix Committed -> Fix Released

    • The vanilla workflow: Coverity discovers a defect and reports it as a bug in Launchpad. A developer fixes the bug as part of his/her normal workflow, ultimately merging his/her branch into trunk. When Coverity fails to find this defect in a new scan, the bug is marked "Fix Committed". When the fixed code is released, a release manager marks the bug as "Fix Released".
  • New -> Confirmed -> In Progress -> New

    • Coverity discovers a defect and reports it as a bug in Launchpad. A developer fixes the bug as part of his/her normal workflow, ultimately merging his/her branch into trunk. Coverity finds that the defect still exists, and re-opens the Launchpad bug by re-setting its status to "New", and the triage/fixing process begins again.
  • New -> Invalid/Wontfix

    • Coverity discovers a defect and reports it as a bug in Launchpad. A developer believes the bug to be "Invalid" or marks it "Wontfix". Coverity's successive discovery of the same defect won't result in a new bug report.

Sync Script

There is launchpad project for the Launchpad Coverity Sync Script where the source can be viewed, branched and reviewed by anyone. Unfortunately, we can't provide a test setup or a way for you to test builds without buying a Coverity license, but the code is available and we're happy to help test changes.

How it works

The script goes through a few different conditions as explained in this graphic:

bug work flow

CanonicalProductStrategy/Coverity (last edited 2015-07-08 16:02:36 by 1)