TestingProcessesUbuntuPersonal

Differences between revisions 5 and 19 (spanning 14 versions)
Revision 5 as of 2015-06-11 11:26:10
Size: 2491
Editor: 107
Comment:
Revision 19 as of 2015-12-03 13:56:01
Size: 9120
Editor: 1
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
WIP
Line 4: Line 2:
<<TableOfContents()>> ||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents(2)>>||
Line 17: Line 15:
 * Flash the phone with the latest development image.
 * citrain device-upgrade <silo number> <password>
 * Overlay PPA
 * Flash the phone with the latest development image (instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] )
 * `citrain device-upgrade <silo number> <password>`
 * /!\ This section may not be needed now if you use an up-to-date distro as citrain has built it in /!\ Overlay PPA
Line 21: Line 19:
   * Create /etc/apt/preferences.d/silo.pref file.
   * Add these lines:

{{{
   * Create this file: `sudo vi /etc/apt/preferences.d/silo.pref`
   * Add these lines (silo number must be 0-padded, e.g. landing-001): {{{
Line 29: Line 25:

   * Save the file.
   * Update the package list: `sudo apt-get update`
   * Upgrade the packages: `sudo apt-get upgrade`

=== Trello Board Edits ===
  * The trello board is located [[https://trello.com/b/AE3swczu/qa-testing-requests-for-questions-ping-ubuntu-qa-on-ubuntu-ci-eng|here]]
  * Grab a ticket from the "Ready for Testing" Lane
  * Add yourself to the ticket (Hover over the ticket and tap space, or select Members and add yourself from there)
  * From the testplan for the silo under test add the title for each section to a ticket comment and mark them as passed or failed. Example: {{{
NO ACCOUNT: PASS.
EXISTING ACCOUNT: PASS.
Security Information: PASS.
Security Information: Paypal payment: PASS.
Security Information: Add Credit/debit card: FAIL (cancel presents blank page instead Preview).
Cancel button flow: PASS.
Buy Now button flow: Logged in: PASS.
Add credit/debit card flow: Cancel adding card: FAIL (cancel presents blank page instead Preview).
Add credit/debit card flow: Add a card: FAIL ("Cancel" in credit card confirmation page -bank website- doesn't add the credit card).
The blank page is a regression, it works fine in the current pay-ui.
}}}
  * If there is a failure in a separate comment clearly make a note of the issue.
Line 35: Line 53:
Any relevant silo bug should be added to the Regression Test Suite as draft.  * Any relevant silo bugs (both new and fixed) should be added to the Regression Test Suite as "To Review".
 * Testplan levels Set the level relevant to the level to access the item, High anything on an initial page section, Medium on a secondary, low anything below that. (Example browser app, opens, connects to web, opens a page, open a new tab, save a bookmark = high, open a bookmark, history, private mode = medium, rename a bookmark, clear history = low)

=== More Info ===
[[https://wiki.ubuntu.com/LandingTeam/SiloTestingGuidelines|More info can be found here]]
Line 39: Line 61:
Images are scheduled [[https://docs.google.com/spreadsheets/d/1LiPHq39XvPaTQoE8vmi-dqSVMTGGLfCh7bFY01BcNNk/edit#gid=50|here]]
Line 40: Line 63:
For Custom Tarballs Read: https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Testing_Custom_Tarballs <<BR>>
For Device Tarballs Read: https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Testing_Device_Tarballs
Line 41: Line 66:
See Sanity Testing Below for device tarballs <<BR>>
For Custom Tarballs look at practitest (cases need adding here)
Line 42: Line 69:
All new cases must be added to the "To Review" Queue in practitest
=== More Info ===
https://wiki.ubuntu.com/QATeam/ChannelsToFlash
Line 46: Line 76:
 * Follow the instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] to get the latest image install
 * Then run the sanity suite
Line 47: Line 79:
Sanity test are located in practitest currently
Line 49: Line 82:
=== More Info ===
https://wiki.ubuntu.com/QATeam/ChannelsToFlash
Line 53: Line 88:
 * Follow the instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] to get the latest image install
 * Then run the regression suite
Line 54: Line 91:
Regression tests are located in Practitest Currently
Line 56: Line 94:
=== More Info ===
https://wiki.ubuntu.com/QATeam/ChannelsToFlash
Line 57: Line 97:
/!\ Should be carried out by canonical staff only
Line 58: Line 99:
The area you wish to test is actually functioning.
Line 59: Line 101:
 * Follow the instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] to get the latest image install
 * Work on the area you have selected
Line 60: Line 104:
Use the web form [[https://docs.google.com/a/canonical.com/forms/d/1rqF4V8VMleQQliEaEqzrUke4aiblSbsczK5BWL33DTU/viewform?edit_requested=true#|here]] to keep track of what you test
Line 63: Line 108:
== Click Package Testing ==
=== Prechecks ===
 1. Goto [[https://docs.google.com/spreadsheets/d/1LiPHq39XvPaTQoE8vmi-dqSVMTGGLfCh7bFY01BcNNk/edit?pli=1#gid=50|here]] and check for a new click package
 1. Download the click and note it's location (Default is ~/Downloads)
=== Installing ===
 1. Fresh flash the device see [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Flashing_the_Current_Release|Flashing Current Release]]
 1. Once flash setup the device
 1. Now goto system settings → about device → developer mode and enable developer mode
 1. While the screen is on do {{{
adb push Download/click_package_name.click /home/phablet
}}}
 1. Follow this up with {{{
adb shell
}}}
 1. In that shell do {{{
pkcon install-local --allow-untrusted click_package_name.click
sudo reboot
}}}
 1. Click package is now ready to test
=== Testing ===
There are various test plans dotted around but on the whole the spread sheet will point you to the right one, there are also the case in Practitest.
Line 68: Line 134:
=== More info ===
== Miscellaneous ==
=== Faking Your Location for Scopes Testing ===
<<Anchor(FakingYourLocation)>>
For the Nearby scope, some functionality is supported only enabled cities. You can “fake location” to London which supports both Songkick and Timeout as follows.
  1. Set image as writable:{{{
      phablet-config writable-image
}}}
  1. phablet-shell into the device
  1. Create this file /custom/custom.prop with below contents:{{{
      $ sudo vim.tiny /custom/custom.prop
      custom.location.fake=true
      custom.location.lon=-0.0999525
      custom.location.lat=51.5062348
}}}
  1. Reboot
  1. Refresh scope AFTER first display
Or you can fake your location to one of these Timeout-supported cities: Accra, Amsterdam, Barcelona, Berlin, Boston, Chicago, Kuala Lumpur, Las Vegas, London, Los Angeles, Madrid, Miami, México DF, New York, Paris, Washington DC.

== FAQ ==
 1. How can the community help?
  * The community are the main users of the currently supported devices so:
    * Knowing what is wrong (bug reports, see point 2)
    * Knowing what is right (ubuntu-phone@lists.launchpad.net send a mail here the devs would love to know what you love)
    * Knowing images are working (Bug reports when the don't, see point 2)
    * Knowing if something is missing (Bug reports, please don't be dismayed if they get marked as wishlist lot to do, alternatively talking to the development team/creator of you favourite app about a port to Ubuntu is good too :) )
 1. What info do you need in bug reports?
  * https://wiki.ubuntu.com/QATeam/BugReportInclusions lists the basics and the start for wifi and phone logs more to come.
  * Most packages are listed in the https://wiki.ubuntu.com/Avengers Please look for a bug similar to yours before reporting a new one.

There are a lot of new terms, tools and procedures we want to use for Testing on Ubuntu Personal. This page is here to document those processes.

Silo Testing

Prechecks

  • All merge proposals must have been reviewed and approved.
  • All automated tests must pass.
  • New features must come with automated tests.
  • New strings must be marked for translation.

Installation

Install phablet-tools package in your desktop machine.

  • Flash the phone with the latest development image (instructions here )

  • citrain device-upgrade <silo number> <password>

  • Warning /!\ This section may not be needed now if you use an up-to-date distro as citrain has built it in Warning /!\ Overlay PPA

    • In case of Vivid + Overlay PPA, the silo packages need to be pinned to a higher priority. Otherwise, silo packages won't be installed.
    • Create this file: sudo vi /etc/apt/preferences.d/silo.pref

    • Add these lines (silo number must be 0-padded, e.g. landing-001):

      Package: *
      Pin: release o=LP-PPA-ci-train-ppa-service-landing-<silo number>
      Pin-Priority: 1002
    • Save the file.
    • Update the package list: sudo apt-get update

    • Upgrade the packages: sudo apt-get upgrade

Trello Board Edits

  • The trello board is located here

  • Grab a ticket from the "Ready for Testing" Lane
  • Add yourself to the ticket (Hover over the ticket and tap space, or select Members and add yourself from there)
  • From the testplan for the silo under test add the title for each section to a ticket comment and mark them as passed or failed. Example:

    NO ACCOUNT: PASS.
    EXISTING ACCOUNT: PASS.
    Security Information: PASS.
    Security Information: Paypal payment: PASS.
    Security Information: Add Credit/debit card: FAIL (cancel presents blank page instead Preview).
    Cancel button flow: PASS.
    Buy Now button flow: Logged in: PASS.
    Add credit/debit card flow: Cancel adding card: FAIL (cancel presents blank page instead Preview).
    Add credit/debit card flow: Add a card: FAIL ("Cancel" in credit card confirmation page -bank website- doesn't add the credit card).
    The blank page is a regression, it works fine in the current pay-ui.
  • If there is a failure in a separate comment clearly make a note of the issue.

Test Plan Locations

Reporting New Cases

  • Any relevant silo bugs (both new and fixed) should be added to the Regression Test Suite as "To Review".
  • Testplan levels Set the level relevant to the level to access the item, High anything on an initial page section, Medium on a secondary, low anything below that. (Example browser app, opens, connects to web, opens a page, open a new tab, save a bookmark = high, open a bookmark, history, private mode = medium, rename a bookmark, clear history = low)

More Info

More info can be found here

Tarball Testing

Prechecks

Images are scheduled here

Installation

For Custom Tarballs Read: https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Testing_Custom_Tarballs
For Device Tarballs Read: https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Testing_Device_Tarballs

Test Plan Locations

See Sanity Testing Below for device tarballs
For Custom Tarballs look at practitest (cases need adding here)

Reporting New Cases

All new cases must be added to the "To Review" Queue in practitest

More Info

https://wiki.ubuntu.com/QATeam/ChannelsToFlash

Sanity Testing

Prechecks

All silos are landed and all tarballs are landed.

Installation

  • Follow the instructions here to get the latest image install

  • Then run the sanity suite

Test Plan Locations

Sanity test are located in practitest currently

Reporting New Cases

Sanity is a deliberately selected small subset of tests targeting software interactions with hardware. Any additional tests felt required should be discussed with the entire team via the mailing list or standups.

More Info

https://wiki.ubuntu.com/QATeam/ChannelsToFlash

Regression Testing

Prechecks

Sanity Must of been run to ensure the image is at least testable

Installation

  • Follow the instructions here to get the latest image install

  • Then run the regression suite

Test Plan Locations

Regression tests are located in Practitest Currently

Reporting New Cases

New Cases should not be found while testing regression, however you may want make a note here(add a link later) to remind you to look at something during exploratory testing.

More Info

https://wiki.ubuntu.com/QATeam/ChannelsToFlash

Exploratory Testing

Warning /!\ Should be carried out by canonical staff only

Prechecks

The area you wish to test is actually functioning.

Installation

  • Follow the instructions here to get the latest image install

  • Work on the area you have selected

Recording Test and Session

Use the web form here to keep track of what you test

Reporting New Cases

If there is automation required here it should be reported as a bug for the development team.
If there are new manual cases they should be put into draft for review in practitest.

Click Package Testing

Prechecks

  1. Goto here and check for a new click package

  2. Download the click and note it's location (Default is ~/Downloads)

Installing

  1. Fresh flash the device see Flashing Current Release

  2. Once flash setup the device
  3. Now goto system settings → about device → developer mode and enable developer mode
  4. While the screen is on do

    adb push Download/click_package_name.click /home/phablet
  5. Follow this up with

    adb shell
  6. In that shell do

    pkcon install-local --allow-untrusted click_package_name.click
    sudo reboot
  7. Click package is now ready to test

Testing

There are various test plans dotted around but on the whole the spread sheet will point you to the right one, there are also the case in Practitest.

Automated Testing

Prechecks

Installation

Test Plan Locations

Reporting Of Failures

More info

Miscellaneous

Faking Your Location for Scopes Testing

For the Nearby scope, some functionality is supported only enabled cities. You can “fake location” to London which supports both Songkick and Timeout as follows.

  1. Set image as writable:

          phablet-config  writable-image
  2. phablet-shell into the device
  3. Create this file /custom/custom.prop with below contents:

          $ sudo vim.tiny /custom/custom.prop
          custom.location.fake=true
          custom.location.lon=-0.0999525
          custom.location.lat=51.5062348
  4. Reboot
  5. Refresh scope AFTER first display

Or you can fake your location to one of these Timeout-supported cities: Accra, Amsterdam, Barcelona, Berlin, Boston, Chicago, Kuala Lumpur, Las Vegas, London, Los Angeles, Madrid, Miami, México DF, New York, Paris, Washington DC.

FAQ

  1. How can the community help?
    • The community are the main users of the currently supported devices so:
      • Knowing what is wrong (bug reports, see point 2)
      • Knowing what is right (ubuntu-phone@lists.launchpad.net send a mail here the devs would love to know what you love)

      • Knowing images are working (Bug reports when the don't, see point 2)
      • Knowing if something is missing (Bug reports, please don't be dismayed if they get marked as wishlist lot to do, alternatively talking to the development team/creator of you favourite app about a port to Ubuntu is good too Smile :) )

  2. What info do you need in bug reports?

QATeam/TestingProcessesUbuntuPersonal (last edited 2015-12-03 13:56:01 by 1)