ServerArchitecture

Differences between revisions 1 and 18 (spanning 17 versions)
Revision 1 as of 2012-03-05 16:48:10
Size: 5544
Editor: ev
Comment:
Revision 18 as of 2015-03-03 17:15:20
Size: 12276
Editor: brian-murray
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
{{attachment:crashdatabaseflow.png|Crash Database Flow|width=100%}}
Line 7: Line 8:
 * OOPS CF
||uuid.uuid1() || Date || Architecture || DistroRelease || ... ||
|| || 'Wed Feb 29 22:56:12 2012' || 'amd64' || 'Ubuntu 12.04' || ... ||
'''Awaiting``Retrace CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:x86_64:[vdso]+70c:...' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||
Line 11: Line 12:
 * Stacktrace CF
|| '/bin/bash:11:x86_64:[vdso]+70c:...' || Date || Stacktrace || ... ||
|| || 'Wed Feb 29 22:56:12 2012' || #0 0x0f in foo () at /foo.c:1 ... || ... ||
'''Bucket CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:read:main' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||
Line 15: Line 16:
 * Indexes CF
|| 'crash_signature_for_stacktrace_address_signature' || '/bin/bash:11:x86_64:[vdso]+70c:...' || ... ||
|| || '/bin/bash:11:read:main...' || ... ||
|| 'retracing' || '/bin/bash:11:x86_64:[vdso]+70c:...' || ... ||
|| || '' || ... ||
'''Bucket``Metadata CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:read:main' || 'First``Seen' || 'Last``Seen' || 'Source' || '~Ubuntu 15.04:``First``Seen' || '~Ubuntu 15.04:``Last``Seen' ||
 ||<rowbgcolor="#ffffff"> '' || '4.3-11ubuntu2' || '4.3-11ubuntu2' || 'bash' || '4.3-11ubuntu2' || '4.3-11ubuntu2' ||
Line 21: Line 20:
 * AwaitingRetrace CF
|| '/bin/bash:11:x86_64:[vdso]+70c:...' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
|| || '' || ... ||
'''Bucket``Retrace``Failure``Reason CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'failed:/bin/bash:11:read:main' || 'Missing``Debug``Symbols' || 'Outdated``Packages' || 'Reason' || 'missing_ddeb_count' || 'oops' || 'outdated_pkg_count' ||
 ||<rowbgcolor="#ffffff"> || 'package names' || 'package names' || 'No crash signature after retracing.' || '23' || '7aefa002-4a7046a3-a02d-644b9d847e0d' ' || '8' ||
Line 25: Line 24:
 * Buckets CF
|| '/bin/bash:11:read:main' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
|| || '' || ... ||
'''Bucket``Version``Systems``2 CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:read:main' || 'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||
Line 29: Line 28:
 * DayBucketCounts CF
|| 20120213 || 'Buckets' || 'DistroVersion:Ubuntu 12.04' || 'Architecutre:i386' || ... ||
|| || 30 || 15 || 3 || ... ||
'''Bucket``Versions CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">|| || ... ||

'''Bucket``Versions``Count CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:read:main' || 'Ubuntu 15.04:4.3-11ubuntu2'|| ... ||
 ||<rowbgcolor="#ffffff"> || '8' || ... ||

'''Bucket``Versions``Day CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">20150130 || '/bin/bash:11:read:main:Ubuntu 15.04:4.3-11ubuntu2' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Bucket``Versions``Full CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:read:main:Ubuntu 15.04:4.3-11ubuntu2' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Bug``To``Crash``Signatures CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> '112358' || '/bin/bash:11:read:main' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Could``Not``Bucket CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">20150121 || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Counters CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> oopses:Crash:ubuntu-touch/devel-proposed:apport:2.14.7-0ubuntu8.1:armhf|| 20150121 || ... ||
 ||<rowbgcolor="#ffffff"> || 8 || ... ||
 ||<rowbgcolor="#ffffff"> oopses:Crash:Ubuntu 12.04:gcalctool || 20150121 || ... ||
 ||<rowbgcolor="#ffffff"> || 3 || ... ||

See daisy/utils.py's get_fields_for_bucket_counters() for all the possible counters.

'''Day``Buckets CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> '20150213:/bin/bash/:11:read:main' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Day``Bucket``Counts CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">20120213 || 'Buckets' || 'DistroVersion:Ubuntu 12.04' || 'Architecture:i386' || ... ||
 ||<rowbgcolor="#ffffff"> || 30 || 15 || 3 || ... ||

'''Day``OOPS CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">20140213 || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || || ... ||

'''Day``Users CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> Crash:Ubuntu 15.04:bash:4.3-11ubuntu2:20140213 || 'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd' || ... ||
 ||<rowbgcolor="#ffffff"> || || ... ||

'''Errors``By``Release CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> Ubuntu 15.04:20140213 00:00:00+0000 || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || || ... ||

'''First``Error CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> Ubuntu 15.04 || 'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd' || ... ||
 ||<rowbgcolor="#ffffff"> || || ... ||

'''Hashes CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> bucket_b || 62616561333664356464393435623765373266363365316138623065626539376163353030626662 || ... ||
 ||<rowbgcolor="#ffffff"> || '/bin/bash:11:read:main' || ... ||

'''Indexes CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> 'crash_signature_for_stacktrace_address_signature' || '/bin/bash:11:x86_64:[vdso]+70c:...' || '/bin/bash:11:read:main...' || ... ||
 ||<rowbgcolor="#ffffff"> || || || ... ||
 ||<rowbgcolor="#ffffff"> 'retracing' || '/bin/bash:11:x86_64:[vdso]+70c:...' || ... || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... || ... ||

'''OOPS CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">uuid.uuid1() || Date || Architecture || Distro``Release || ... ||
 ||<rowbgcolor="#ffffff"> || 'Wed Feb 29 22:56:12 2012' || 'amd64' || 'Ubuntu 12.04' || ... ||

'''Retrace``Stats CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> 20140213 || 'Ubuntu 15:04:failed' || 'Ubuntu 15:04:success' || ... ||
 ||<rowbgcolor="#ffffff"> || '4' || '16' || ... ||


'''Source``Version``Buckets CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">(bash, 4.3-11ubuntu2) || '/bin/bash:11:read:main' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Stacktrace CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">'/bin/bash:11:x86_64:[vdso]+70c:...' || Date || Stacktrace || ... ||
 ||<rowbgcolor="#ffffff"> || 'Wed Feb 29 22:56:12 2012' || #0 0x0f in foo () at /foo.c:1 ... || ... ||

'''System``Images CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> channel || 'ubuntu-touch/devel-proposed' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> device_name || 'mako' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

Row keys include: channel, device_name, rootfs_build, device_image.

'''System``OOPS``Hashes CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">|| || ... ||

'''Systems``For``Errors``By``Release CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> Ubuntu 15.04:2014-02-13 00:00:00+0000 || 'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Unique``Systems``For``Errors``By``Release CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff">|| || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||

'''Unique``Users``90``Days CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> Ubuntu 15.04 || 20140213 || 20140214 || ... ||
 ||<rowbgcolor="#ffffff"> || 5 || 10 || ... ||

'''User``Binary``Packages CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> foundations-bugs || apt || ubuntu-release-upgrader || ... ||
 ||<rowbgcolor="#ffffff"> || '' || '' || ... ||

Used to speed up lookups for packages to which a user is subscribed for users subscribed to a large number of packages.

'''User``OOPS CF'''
 ||<tablebgcolor="#cccccc" rowbgcolor="#ffffff"> 'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd' || '7aefa002-4a7046a3-a02d-644b9d847e0d' || ... ||
 ||<rowbgcolor="#ffffff"> || '' || ... ||
Line 43: Line 153:
 2. If it is a binary, a lookup is done with the stacktrace address signature as a row key to the Stacktrace ColumnFamily. If this signature has already been retraced with a previous crash, the Stacktrace CF will contain the fully retraced report. We then add the OOPS ID to the bucket queue.  2. If it is a binary, a lookup is done with the stacktrace address signature as a column name to the crash_signature_for_stacktrace_address_signature row in the Indexes Column``Family. If this signature has already been retraced with a previous crash, the Stacktrace CF will contain the fully retraced report. We then add the OOPS ID to the bucket queue.
Line 45: Line 155:
 3. If the above lookup returns NULL, it is assumed that we don't have a retraced report for this signature, and a full core dump is requested. If the Indexes CF 'retracing' column key has the signature, then we know that we have this core dump already and are just waiting for it to be retraced. We add the OOPS ID to the AwaitingRetrace CF in this latter case rather than asking for the core dump.  3. If the above lookup returns NULL, it is assumed that we don't have a retraced report for this signature, and a full core dump is requested. If the Indexes CF 'retracing' column key has the signature, then we know that we have this core dump already and are just waiting for it to be retraced. We add the OOPS ID to the Awaiting``Retrace CF in both cases, as we may not get a core dump from this user.
Line 47: Line 157:
 4. When the reporting daemon (whoopsie) sends the full core dump on behalf of the user, it's pushed onto the respective architecture queue, then eventually processed. It does this by first pulling the rest of the crash data back down from the OOPS ColumnFamily, then running the now fully formed report through apport-retrace. The fully retraced report is then pushed into the Stacktrace ColumnFamily, row keyed on the stacktrace address signature.  4. When the reporting daemon (whoopsie) sends the full core dump on behalf of the user, it's pushed onto the respective architecture queue, then eventually processed. It does this by first pulling the rest of the crash data back down from the OOPS Column``Family, then running the now fully formed report through apport-retrace. The fully retraced report is then pushed into the Stacktrace Column``Family, row keyed on the stacktrace address signature.
Line 53: Line 163:
 Finally, the OOPS ID (uuid1) for this crash is pushed onto the bucket queue. All the OOPS IDs with the same stacktrace address signature are also pulled from the AwaitingRetrace CF and pushed onto the bucket queue.  Finally, the OOPS ID (uuid1) for this crash is pushed onto the bucket queue. All the OOPS IDs with the same stacktrace address signature are also pulled from the Awaiting``Retrace CF and pushed onto the bucket queue.
Line 55: Line 165:
 5. If retracing fails, then the crash address signature is removed from the retrace row key in the Indexes CF, indicating that we need another core file to process. This is one example of where we can use ConsistencyLevel.ONE. We don't care if the code accepting crashes is made aware of needing to accept another core file straight away.  5. If retracing fails, then the crash address signature is removed from the retrace row key in the Indexes CF, indicating that we need another core file to process. This is one example of where we can use Consistency``Level.ONE. We don't care if the code accepting crashes is made aware of needing to accept another core file straight away.
Line 57: Line 167:
 6. Any crashes that come through while we have the same signature in the Indexes CF's retrace row will be written to the AwaitingRetrace CF, with the stacktrace address signature as the row key, and the OOPS ID as the column name (with a NULL value).  6. Any crashes that come through while we have the same signature in the Indexes CF's retrace row will be written to the Awaiting``Retrace CF, with the stacktrace address signature as the row key, and the OOPS ID as the column name (with a NULL value).
Line 65: Line 175:
 8. As each bucket is modified, the counter column values in the DayBucketCounts CF are incremented. This CF is then consumed by the report web interface.  8. As each bucket is modified, the counter column values in the Day``Bucket``Counts CF are incremented. This CF is then consumed by the report web interface.

Crash Database Flow

Terminology

  • A stacktrace address signature is a signature generated from the addresses of a stack trace (bzr:apport/apport/report.py:1215). Because this isn't as complete as a full crash signature, generated by retracing the core dump with all the needed debug symbols installed, we will have multiple crash signatures for the same stack trace. We can then request a full core dump if and only if we haven't already received a core dump for the submitted stacktrace address signature. This saves the vast majority of users from having to upload a massive file.
  • A regular crash signature is simply the concatenation of the executable path, the signal number, and the top few lines of the stack trace (bzr:apport/apport/report.py:1128).

Column Families

AwaitingRetrace CF

  • '/bin/bash:11:x86_64:[vdso]+70c:...'

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

Bucket CF

  • '/bin/bash:11:read:main'

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

BucketMetadata CF

  • '/bin/bash:11:read:main'

    'FirstSeen'

    'LastSeen'

    'Source'

    '~Ubuntu 15.04:FirstSeen'

    '~Ubuntu 15.04:LastSeen'

    '4.3-11ubuntu2'

    '4.3-11ubuntu2'

    'bash'

    '4.3-11ubuntu2'

    '4.3-11ubuntu2'

BucketRetraceFailureReason CF

  • 'failed:/bin/bash:11:read:main'

    'MissingDebugSymbols'

    'OutdatedPackages'

    'Reason'

    'missing_ddeb_count'

    'oops'

    'outdated_pkg_count'

    'package names'

    'package names'

    'No crash signature after retracing.'

    '23'

    '7aefa002-4a7046a3-a02d-644b9d847e0d' '

    '8'

BucketVersionSystems2 CF

  • '/bin/bash:11:read:main'

    'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd'

    ...

    ...

BucketVersions CF

  • ...

BucketVersionsCount CF

  • '/bin/bash:11:read:main'

    'Ubuntu 15.04:4.3-11ubuntu2'

    ...

    '8'

    ...

BucketVersionsDay CF

  • 20150130

    '/bin/bash:11:read:main:Ubuntu 15.04:4.3-11ubuntu2'

    ...

    ...

BucketVersionsFull CF

  • '/bin/bash:11:read:main:Ubuntu 15.04:4.3-11ubuntu2'

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

BugToCrashSignatures CF

  • '112358'

    '/bin/bash:11:read:main'

    ...

    ...

CouldNotBucket CF

  • 20150121

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

Counters CF

  • oopses:Crash:ubuntu-touch/devel-proposed:apport:2.14.7-0ubuntu8.1:armhf

    20150121

    ...

    8

    ...

    oopses:Crash:Ubuntu 12.04:gcalctool

    20150121

    ...

    3

    ...

See daisy/utils.py's get_fields_for_bucket_counters() for all the possible counters.

DayBuckets CF

  • '20150213:/bin/bash/:11:read:main'

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

DayBucketCounts CF

  • 20120213

    'Buckets'

    'DistroVersion:Ubuntu 12.04'

    'Architecture:i386'

    ...

    30

    15

    3

    ...

DayOOPS CF

  • 20140213

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

DayUsers CF

  • Crash:Ubuntu 15.04:bash:4.3-11ubuntu2:20140213

    'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd'

    ...

    ...

ErrorsByRelease CF

  • Ubuntu 15.04:20140213 00:00:00+0000

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

FirstError CF

  • Ubuntu 15.04

    'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd'

    ...

    ...

Hashes CF

  • bucket_b

    62616561333664356464393435623765373266363365316138623065626539376163353030626662

    ...

    '/bin/bash:11:read:main'

    ...

Indexes CF

  • 'crash_signature_for_stacktrace_address_signature'

    '/bin/bash:11:x86_64:[vdso]+70c:...'

    '/bin/bash:11:read:main...'

    ...

    ...

    'retracing'

    '/bin/bash:11:x86_64:[vdso]+70c:...'

    ...

    ...

    ...

    ...

OOPS CF

  • uuid.uuid1()

    Date

    Architecture

    DistroRelease

    ...

    'Wed Feb 29 22:56:12 2012'

    'amd64'

    'Ubuntu 12.04'

    ...

RetraceStats CF

  • 20140213

    'Ubuntu 15:04:failed'

    'Ubuntu 15:04:success'

    ...

    '4'

    '16'

    ...

SourceVersionBuckets CF

  • (bash, 4.3-11ubuntu2)

    '/bin/bash:11:read:main'

    ...

    ...

Stacktrace CF

  • '/bin/bash:11:x86_64:[vdso]+70c:...'

    Date

    Stacktrace

    ...

    'Wed Feb 29 22:56:12 2012'

    #0 0x0f in foo () at /foo.c:1 ...

    ...

SystemImages CF

  • channel

    'ubuntu-touch/devel-proposed'

    ...

    ...

    device_name

    'mako'

    ...

    ...

Row keys include: channel, device_name, rootfs_build, device_image.

SystemOOPSHashes CF

  • ...

SystemsForErrorsByRelease CF

  • Ubuntu 15.04:2014-02-13 00:00:00+0000

    'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd'

    ...

    ...

UniqueSystemsForErrorsByRelease CF

  • ...

    ...

UniqueUsers90Days CF

  • Ubuntu 15.04

    20140213

    20140214

    ...

    5

    10

    ...

UserBinaryPackages CF

  • foundations-bugs

    apt

    ubuntu-release-upgrader

    ...

    ...

Used to speed up lookups for packages to which a user is subscribed for users subscribed to a large number of packages.

UserOOPS CF

  • 'deadbeefbd89cc5da1f045978a550ad4b38782c1390ccd8c265c5ddc803be1d8bc0d0d5657350ce06c611975d77dac5a28db5e78921476be5430e0b6c1a44dfd'

    '7aefa002-4a7046a3-a02d-644b9d847e0d'

    ...

    ...

The UserOOPS CF is for an eventual feature whereby you can remove your data from the crash database, given a pairing of your system UUID and a UUID generated and stored in /etc/default/whoopsie.

The DayOOPS CF comes from lp:oops-repository, but is yet to be used here.

A typical run through of the diagram may occur as follows:

  1. A crash comes in and is determined to be either a binary or Python code (server acceptance).
  2. If it is a binary, a lookup is done with the stacktrace address signature as a column name to the crash_signature_for_stacktrace_address_signature row in the Indexes ColumnFamily. If this signature has already been retraced with a previous crash, the Stacktrace CF will contain the fully retraced report. We then add the OOPS ID to the bucket queue.

  3. If the above lookup returns NULL, it is assumed that we don't have a retraced report for this signature, and a full core dump is requested. If the Indexes CF 'retracing' column key has the signature, then we know that we have this core dump already and are just waiting for it to be retraced. We add the OOPS ID to the AwaitingRetrace CF in both cases, as we may not get a core dump from this user.

  4. When the reporting daemon (whoopsie) sends the full core dump on behalf of the user, it's pushed onto the respective architecture queue, then eventually processed. It does this by first pulling the rest of the crash data back down from the OOPS ColumnFamily, then running the now fully formed report through apport-retrace. The fully retraced report is then pushed into the Stacktrace ColumnFamily, row keyed on the stacktrace address signature.

    The newly generated crash signature is also written to the Indexes CF, as the column value for the stacktrace address signature column name.

    The stacktrace address signature is removed from the Indexes CF's retrace row, indicating that we're no longer waiting for this signature to be retraced. Subsequent reports with this signature can be written straight to the bucket queue.

    Finally, the OOPS ID (uuid1) for this crash is pushed onto the bucket queue. All the OOPS IDs with the same stacktrace address signature are also pulled from the AwaitingRetrace CF and pushed onto the bucket queue.

  5. If retracing fails, then the crash address signature is removed from the retrace row key in the Indexes CF, indicating that we need another core file to process. This is one example of where we can use ConsistencyLevel.ONE. We don't care if the code accepting crashes is made aware of needing to accept another core file straight away.

  6. Any crashes that come through while we have the same signature in the Indexes CF's retrace row will be written to the AwaitingRetrace CF, with the stacktrace address signature as the row key, and the OOPS ID as the column name (with a NULL value).

  7. The bucket queue is processed by what is currently a small Python program to take the report, generate a full crash signature, and add the OOPS ID of that report to the respective bucket.


    Obviously, a single issue may have multiple stacktraces, and likewise, many issues may share a single stacktrace. Therefore, this bucketing algorithm will grow more comprehensive with time, including other fields which will be concatenated onto the row key as needed (or set to empty if they no longer serve as a differentiating factor).

    If we didn't use a separate bucketing step, even for the relatively simple Python crashes, I fear we'd spend too much time in the crash acceptance code, reassembling the report from the BSON data to generate a crash signature.

  8. As each bucket is modified, the counter column values in the DayBucketCounts CF are incremented. This CF is then consumed by the report web interface.

ErrorTracker/ServerArchitecture (last edited 2015-03-03 17:15:20 by brian-murray)