TheStages
|
Size: 7414
Comment:
|
Size: 6938
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ## page was copied from QATeam/Overview | |
| Line 4: | Line 5: |
| = More Information = | |
| Line 6: | Line 6: |
| This section provides further information and useful links for testers. As with all alphas and betas they are not suitable for a production environment, please take the time to read [[http://ubuntuforums.org/showthread.php?t=1594833 | Common Questions for Testing]] | = The Stages = |
| Line 8: | Line 8: |
| == Announcements == | There are some differences between how the ubuntu family carry out their testing cycles. Some only release at LTS, some use alpha testing, some use only beta testing and some use only release candidate. Add to this the sequence that is cadence testing, it can get confusing. You need to check with the team(s) who you are testing for to know which of the various options they are using. The dates of when important things happen? |
| Line 10: | Line 10: |
| At each milestone release, the notes at [[https://wiki.ubuntu.com/QuantalQuetzal/TechnicalOverview | technical overview]] are updated. | '''[[https://wiki.ubuntu.com/ReleaseSchedule | Release Schedule.]]''' |
| Line 12: | Line 12: |
| == Bugs == | == Testing == Testing is split into distinct, but joined areas. The '''Daily Builds''', the '''QA-testing of Milestone releases''' and the '''Milestone releases''' themselves. |
| Line 15: | Line 17: |
| === Reporting Bugs === Discussing a bug on IRC, email, forum or facebook areas etc. does not raise a bug. The devs do not monitor those areas. By all means use them to discuss a problem. Once your discussion about the bug is at a stage where you can report it, then for it to be actioned by the devs you '''must''' raise a bug. You can find information on how to report bugs at [[https://wiki.ubuntu.com/Lubuntu/ReportingBugs | Reporting Bugs]] |
As new features are added at the beginning of the cycle, Quality assurance and the testers check for bugs and test for correct function. == Daily Builds == |
| Line 18: | Line 20: |
| === Dealing with Bug Reports === Triaging (dealing) with bug reports is a really important area. When new fixes come through it is important to let people know. [[https://wiki.ubuntu.com/Bugs/Responses#Old_untouched_bugs | please try a newer version]] is one such answer. There is also [[https://wiki.ubuntu.com/Bugs/Responses | full list of responses]]. It is important that the person who raised the bug, no matter if it is incomplete, a duplicate or any other reason is made to feel that their bug is important. Simply put, to them, it is the most important bug in the world and should be treated as such. |
The daily images between milestone releases are tested on a cadence (in the case of quantal, every 2 weeks) by the ubuntu QA community. |
| Line 21: | Line 22: |
| === The Bug Squad === If you'd like to know more about the treatment of bugs, please head over to [[https://wiki.ubuntu.com/BugSquad/KnowledgeBase | The Bug Squad]] new comers are always welcome. |
The most important goal with these images is simply to find as many bugs as possible and report them with enough detail that they can be fixed. Finding the bugs ahead of the milestone crunch is helpful as it gives us more time to fix them. Of course finding and reporting bugs in the released milestones is also useful for improving the overall quality of the distribution. |
| Line 24: | Line 24: |
| == Testing == Testing is split into distinct, but joined areas. The '''Daily Builds''', the '''QA-testing of Milestone releases''' and the '''Milestone releases''' themselves. |
As there is less time pressure on testing these images than the pre-release ones, we would encourage more in-depth testing on a range of hardware to flush out the deeper hidden bugs. If possible, please perform installs using a variety of methods and focus on the quality and depth of testing as opposed to the quantity. |
| Line 27: | Line 26: |
| To not get overly complicated, think of it as that we have a [[https://wiki.ubuntu.com/QuantalQuetzal/ReleaseSchedule | schedule]] to keep to. A few days before a Milestone is due, the daily is plucked and becomes the QA (Quality Assurance) test version for the Milestone release. Once it is confirmed that the QA version works, it then becomes the [[https://wiki.ubuntu.com/Lubuntu/Testing#Milestone_Releases | Milestone]]. | The Dailies are automatically generated every 24 hours, the get suspended as the QA cycles kick in. |
| Line 29: | Line 28: |
| All release stages are tracked by [[http://iso.qa.ubuntu.com/ | ISO Tracker]] where you can get the latest builds, see and allocate any [[https://wiki.ubuntu.com/Lubuntu/ReportingBugs | Bugs]]. | == Pre-Alpha == |
| Line 31: | Line 30: |
| So, in order of how they happen. | Scary, expect them to break! At this point in the cycle the devolopers (devs) are pushing the boundaries to the absolute limit to what they would ''like'' to be in the next release. Testing is always important during this time, so that the devs have some feed back as to how things are going. The use of [[https://wiki.ubuntu.com/Lubuntu/Testing/TheStages#Virtual_Machines | Virtual Machines]] is handy at this point. |
| Line 33: | Line 32: |
| === Daily Builds === | == Alpha's == |
| Line 35: | Line 34: |
| These iso's are automatically generated every 24 hours using the latest updates on the system from the devs. They are available from [[http://iso.qa.ubuntu.com/|ISO tracker]]. Using the [[https://help.ubuntu.com/community/ZsyncCdImage | zsync (or rsync)]] option allows you to update your [[https://help.ubuntu.com/community/BurningIsoHowto |iso]] to any of the various dailies you choose to follow without having to re-download the entire iso. They are there to check that bugs that are resolved between the Milestone releases do not break the install. They also are used to confirm that any fix for a bug that seriously affects an initial install which is released for testing now works. | With the launch of the alpha's, we are into the area of 'it should not break too often'. The pre Alpha testing is to ensure that they boot. With a bootable Alpha 1 out, the devs then go to try adding things to it and updating things to the latest versions available for things. Things may well get broken. With the test version of the Alpha 2, things are little less concentrated on getting 'new' stuff in, but checking what is fixable. |
| Line 37: | Line 36: |
| === QA testing of Milestone releases === | The purpose of the Alpha milestones is to encourage wider public testing of the image builds. Before we can recommend the images for public testing we must at least ensure that they have all built correctly (are not in themselves corrupt) and are at least known to boot on some machines. Some form of installation should also be tested. We do not require that these images be tested for any possible minor defect or that all possible install methods work, since that is rather the goal of the wider image testing itself. Find the show stopper bugs and get them fixed. |
| Line 39: | Line 38: |
| Alpha, Beta and Release Candidates (RC) are also tested using [[http://iso.qa.ubuntu.com/|ISO tracker]]. If you want to help out in this important area of testing, please read through [[https://wiki.ubuntu.com/Testing/ISO/Procedures | Procedures]] for further details. These appear a couple of days before the actual Milestone release so that we can check they are okay to become Milestone releases. | When an important error is found, the current frozen images are rejected and new ones are built after the fix has been applied. It is therefore important to make sure that you are testing the most recent images. |
| Line 41: | Line 40: |
| '''QA testing is to ensure the actual install iso works, if you can, please get involved in the qa testing''' | If the bug is specific to one flavour, only those will be re-built and the other flavour builds remain valid. So if we find a serious bug in the Ubuntu version we will rebuild those images, which will take several hours in total. Likewise, if we find a bug that exists across all flavors, but only on i386, we will re-build i386 and all amd64 and PPC images remain valid. |
| Line 43: | Line 42: |
| === Milestone Releases === | == Beta and Final Release Candidates == |
| Line 45: | Line 44: |
| Once a Milestone release passes the QA testing, it becomes a Milestone Release and is listed on the [[http://cdimage.ubuntu.com/lubuntu/releases/ | Releases]] as such. | The first glimpse of what the final release may well look like. With the Betas, we mainly switch from testing on [[https://wiki.ubuntu.com/Lubuntu/Testing/TheStages#Virtual_Machines | Virtual Machines]] to 'Real Computers'. No changes to what applications that will be shipped from this point should be altered without a very good reason. The first Beta is for people to widely check on as many computers as possible to check if certain types of computer and their associated internal parts are having problems. The second Beta is the prelude for the final version, things are now concentrated on bug-fixing. |
| Line 47: | Line 46: |
| If you would to know more about how this all works, have a read of [[https://wiki.ubuntu.com/Lubuntu/Testing/TheStages | Stages of testing]]. | For Beta and Final milestones we have extended testing periods where we try to find as many bugs as we can. With the Beta we want to put our best foot forward across the distribution and want to fix as many bugs as we have the manpower for, large or small. For the Final release on the other hand, we will only fix the remaining release critical bugs. This is because seemingly small changes can have unintended consequences and break other things. Minor changes that are merely a good idea to fix will be deferred to downloadable updates. |
| Line 49: | Line 48: |
| == General Testing == During the release cycle, things will get broken. You can really reduce these occurrences by taking the time to read [[http://ubuntuforums.org/showthread.php?t=1479146|Partial Upgrades]]. |
The procedure of rejecting known bad images and continuing testing of unaffected ones also applies here as with the Alphas. |
| Line 52: | Line 50: |
| === Known Issues === All the known issues for a particular release are mentioned in the Announcement email, and are available to see at [[http://iso.qa.ubuntu.com/ | ISO Tracker]]. |
== Final Release == |
| Line 55: | Line 52: |
| === Manual test of ISO and CD === On the help-pages of Ubuntu there is an extensive guide on [[https://help.ubuntu.com/community/HowToMD5SUM|how to MD5SUM]]. |
And that, is how we test & do things. If you fancy a roller-coaster ride then do get on. You will have the ability when the Release is made, to smile to yourself and to others. And to say "'''I was a part of that'''". |
| Line 58: | Line 54: |
| == Specific Testing == | === Thank-you === |
| Line 60: | Line 56: |
| === QA tests === You can do more specific tests, like ones done for the Ubuntu ISO : http://iso.qa.ubuntu.com/qatracker/test/5090 |
So, you have got this far? Thank-you! We were all once new to the family that is Ubuntu and felt worthless because we had so many questions, yet felt we could never answer them for others. In testing the [[https://launchpad.net/~ubuntu-testing | Ubuntu-QA team]] welcomes you. You are a new tester, you have hundreds of questions running through your mind. There is no such thing as a dumb question, if they are not answered on our [[https://wiki.ubuntu.com/QATeam | QA page]], please just ask! We will, as testers, answer any S.O.S. request from any of our family. We cannot stress how important you people are. Please do feel free to ask questions, these make the instructions better for everyone |
| Line 63: | Line 58: |
| === Laptop Testing === | == Virtual Machines == |
| Line 65: | Line 60: |
| Laptops never cease to have their little 'quirks'. You can help on this important area by heading over to [[https://wiki.ubuntu.com/Testing/Laptop/Procedures | Laptop Testing]] for full details. | As some testing is carried out on Virtiual Machines (VMs), a quick note on them. |
| Line 67: | Line 62: |
| * Advantage: If it dies, you can restart / re-install it easily without affecting your 'main' machine. * Dis-advantage: A VM does not fully check out your own computers' hardware setup. |
|
| Line 68: | Line 65: |
| === Performance tests === You can test some aspects of Ubuntu performance with the following programs : * [[http://gtkperf.sourceforge.net/ | gtkperf]] : Test the performance of the gtk theme. * [[http://www.phoronix-test-suite.com/ | phoronix-test-suite ]]: General benchmarks and test suite. * [[https://wiki.ubuntu.com/BootCharting | bootchart]] : Test boot process. |
Each cycle we hold tutorial / classroom sessions on these invaluable tools. If you'd like more information click on the link below to take you to that area. |
| Line 74: | Line 67: |
| === Unwanted packages === Some packages can be automatically installed, but are not wanted on a default installation. To find the package which automatically installed the package that you don't want : * Install apt-rdepends * run "apt-rdepends -r --show=Depends the_unwanted_package" => It will show which packages depend on the_unwanted_package. * run "apt-rdepends -r --show=Recommends the_unwanted_package" => It will show which packages recommend the_package_unwanted (Recommended packages are installed by default). * You may have to run the commands several times to see the complete chain of depends / recommends. === PPA Testing === Sometimes during the test cycle one of the developers may ask you to test something specific for them. Most often this is from a '''staging''' area. To temporarily add a '''staging''' area (below is an example using the lubuntu-staging ppa and the application lxinput), install what they want you to test and then remove the staging area following the below example. {{{ sudo apt-add-repository ppa:lubuntu-dev/staging sudo apt-get update sudo apt-get install lxinput sudo apt-add-repository -r ppa:lubuntu-dev/staging sudo apt-get update }}} This will 'grab' the lxinput from the staging area and then remove staging. If the new '''lxinput''' broke your system, then you need to get rid of it and re-install the current version {{{ sudo apt-get remove lxinput sudo apt-get install lxinput }}} This will remove the new '''lxinput''' we got from the ppa, and now that ppa is no longer in use, the install will 'grab' the one from the normal area. If you are working on a bug, it is important to test with these packages to verify it has been fixed. |
'''[[https://wiki.ubuntu.com/Testing/Activities/Classroom | All about tutorial / classroom sessions.]]''' |
The Stages
There are some differences between how the ubuntu family carry out their testing cycles. Some only release at LTS, some use alpha testing, some use only beta testing and some use only release candidate. Add to this the sequence that is cadence testing, it can get confusing. You need to check with the team(s) who you are testing for to know which of the various options they are using. The dates of when important things happen?
Testing
Testing is split into distinct, but joined areas. The Daily Builds, the QA-testing of Milestone releases and the Milestone releases themselves.
As new features are added at the beginning of the cycle, Quality assurance and the testers check for bugs and test for correct function.
Daily Builds
The daily images between milestone releases are tested on a cadence (in the case of quantal, every 2 weeks) by the ubuntu QA community.
The most important goal with these images is simply to find as many bugs as possible and report them with enough detail that they can be fixed. Finding the bugs ahead of the milestone crunch is helpful as it gives us more time to fix them. Of course finding and reporting bugs in the released milestones is also useful for improving the overall quality of the distribution.
As there is less time pressure on testing these images than the pre-release ones, we would encourage more in-depth testing on a range of hardware to flush out the deeper hidden bugs. If possible, please perform installs using a variety of methods and focus on the quality and depth of testing as opposed to the quantity.
The Dailies are automatically generated every 24 hours, the get suspended as the QA cycles kick in.
Pre-Alpha
Scary, expect them to break! At this point in the cycle the devolopers (devs) are pushing the boundaries to the absolute limit to what they would like to be in the next release. Testing is always important during this time, so that the devs have some feed back as to how things are going. The use of Virtual Machines is handy at this point.
Alpha's
With the launch of the alpha's, we are into the area of 'it should not break too often'. The pre Alpha testing is to ensure that they boot. With a bootable Alpha 1 out, the devs then go to try adding things to it and updating things to the latest versions available for things. Things may well get broken. With the test version of the Alpha 2, things are little less concentrated on getting 'new' stuff in, but checking what is fixable.
The purpose of the Alpha milestones is to encourage wider public testing of the image builds. Before we can recommend the images for public testing we must at least ensure that they have all built correctly (are not in themselves corrupt) and are at least known to boot on some machines. Some form of installation should also be tested. We do not require that these images be tested for any possible minor defect or that all possible install methods work, since that is rather the goal of the wider image testing itself. Find the show stopper bugs and get them fixed.
When an important error is found, the current frozen images are rejected and new ones are built after the fix has been applied. It is therefore important to make sure that you are testing the most recent images.
If the bug is specific to one flavour, only those will be re-built and the other flavour builds remain valid. So if we find a serious bug in the Ubuntu version we will rebuild those images, which will take several hours in total. Likewise, if we find a bug that exists across all flavors, but only on i386, we will re-build i386 and all amd64 and PPC images remain valid.
Beta and Final Release Candidates
The first glimpse of what the final release may well look like. With the Betas, we mainly switch from testing on Virtual Machines to 'Real Computers'. No changes to what applications that will be shipped from this point should be altered without a very good reason. The first Beta is for people to widely check on as many computers as possible to check if certain types of computer and their associated internal parts are having problems. The second Beta is the prelude for the final version, things are now concentrated on bug-fixing.
For Beta and Final milestones we have extended testing periods where we try to find as many bugs as we can. With the Beta we want to put our best foot forward across the distribution and want to fix as many bugs as we have the manpower for, large or small. For the Final release on the other hand, we will only fix the remaining release critical bugs. This is because seemingly small changes can have unintended consequences and break other things. Minor changes that are merely a good idea to fix will be deferred to downloadable updates.
The procedure of rejecting known bad images and continuing testing of unaffected ones also applies here as with the Alphas.
Final Release
And that, is how we test & do things. If you fancy a roller-coaster ride then do get on. You will have the ability when the Release is made, to smile to yourself and to others. And to say "I was a part of that".
Thank-you
So, you have got this far? Thank-you! We were all once new to the family that is Ubuntu and felt worthless because we had so many questions, yet felt we could never answer them for others. In testing the Ubuntu-QA team welcomes you. You are a new tester, you have hundreds of questions running through your mind. There is no such thing as a dumb question, if they are not answered on our QA page, please just ask! We will, as testers, answer any S.O.S. request from any of our family. We cannot stress how important you people are. Please do feel free to ask questions, these make the instructions better for everyone
Virtual Machines
As some testing is carried out on Virtiual Machines (VMs), a quick note on them.
- Advantage: If it dies, you can restart / re-install it easily without affecting your 'main' machine.
- Dis-advantage: A VM does not fully check out your own computers' hardware setup.
Each cycle we hold tutorial / classroom sessions on these invaluable tools. If you'd like more information click on the link below to take you to that area.
QATeam/Overview/TheStages (last edited 2015-01-23 16:17:15 by xdsl-83-150-81-40)