TheStages

Differences between revisions 11 and 23 (spanning 12 versions)
Revision 11 as of 2013-07-16 23:51:04
Size: 6938
Editor: host-89-241-247-22
Comment:
Revision 23 as of 2015-01-23 16:17:15
Size: 6174
Editor: xdsl-83-150-81-40
Comment: Remove the mention to virtual machines; it isn't related to the stages/schedule.
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
There are some differences between how the ubuntu family carry out their testing cycles. Some only release at LTS, some use alpha testing, some use only beta testing and some use only release candidate. Add to this the sequence that is cadence testing, it can get confusing. You need to check with the team(s) who you are testing for to know which of the various options they are using. The dates of when important things happen? There are some differences between how the ubuntu family carry out their testing cycles. Some only release at LTS, some use alpha testing, some use only beta testing and everyone has to use release candidate. You need to check with the team(s) who you are testing for to know which of the various options they are using. The dates of when important things happen?
Line 10: Line 10:
'''[[https://wiki.ubuntu.com/ReleaseSchedule | Release Schedule.]]'''  * '''[[https://wiki.ubuntu.com/ReleaseSchedule|Release Schedule]]'''
Line 16: Line 16:
As new features are added at the beginning of the cycle, Quality assurance and the testers check for bugs and test for correct function.
Line 17: Line 18:
As new features are added at the beginning of the cycle, Quality assurance and the testers check for bugs and test for correct function.
Line 20: Line 20:
The daily images between milestone releases are tested on a cadence (in the case of quantal, every 2 weeks) by the ubuntu QA community. The daily images are built automatically each 24 hours when not 'turned off' for milestone testing.
Line 26: Line 26:
The Dailies are automatically generated every 24 hours, the get suspended as the QA cycles kick in. The Dailies are automatically generated every 24 hours, they get suspended as the QA cycles kick in.
Line 30: Line 30:
Scary, expect them to break! At this point in the cycle the devolopers (devs) are pushing the boundaries to the absolute limit to what they would ''like'' to be in the next release. Testing is always important during this time, so that the devs have some feed back as to how things are going. The use of [[https://wiki.ubuntu.com/Lubuntu/Testing/TheStages#Virtual_Machines | Virtual Machines]] is handy at this point. Scary, expect them to break! At this point in the cycle the developers (devs) are pushing the boundaries to the absolute limit to what they would ''like'' to be in the next release. Testing is always important during this time, so that the devs have some feed back as to how things are going. The use of [[https://help.ubuntu.com/community/VirtualMachines | Virtual Machines]] is handy at this point.
Line 44: Line 44:
The first glimpse of what the final release may well look like. With the Betas, we mainly switch from testing on [[https://wiki.ubuntu.com/Lubuntu/Testing/TheStages#Virtual_Machines | Virtual Machines]] to 'Real Computers'. No changes to what applications that will be shipped from this point should be altered without a very good reason. The first Beta is for people to widely check on as many computers as possible to check if certain types of computer and their associated internal parts are having problems. The second Beta is the prelude for the final version, things are now concentrated on bug-fixing. The first glimpse of what the final release may well look like. With the Betas, we mainly switch from testing on [[https://help.ubuntu.com/community/VirtualMachines | Virtual Machines]] to 'Real Computers'. No changes to what applications that will be shipped from this point should be altered without a very good reason. The first Beta is for people to widely check on as many computers as possible to check if certain types of computers and their associated internal parts are having problems. The second Beta is the prelude for the final version, things are now concentrated on bug-fixing.
Line 46: Line 46:
For Beta and Final milestones we have extended testing periods where we try to find as many bugs as we can. With the Beta we want to put our best foot forward across the distribution and want to fix as many bugs as we have the manpower for, large or small. For the Final release on the other hand, we will only fix the remaining release critical bugs. This is because seemingly small changes can have unintended consequences and break other things. Minor changes that are merely a good idea to fix will be deferred to downloadable updates. For Beta and Final milestones we have extended testing periods where we try to find as many bugs as we can. With the Beta we want to put our best foot forward across the distribution and want to fix as many bugs as we have the manpower for, large or small. For the Final release on the other hand, we will only fix the remaining critical bugs from the previous release. This is because seemingly small changes can have unintended consequences and break other things. Minor changes that are merely a good idea to fix will be deferred to downloadable updates.
Line 49: Line 49:

== Point Releases ==

For [[https://wiki.ubuntu.com/LTS | L.T.S.'s]], These are what are is called '''point-releases'''. For example 12.04 LTS has had 4 point releases and is now on 12.04.4 In the meantime, 14.04 approaches it's first point relase as I write this. The testing time for point releases is very short and we do ask people to get involved. The changes for point releases have undergone [[https://wiki.ubuntu.com/QATeam/PerformingSRUVerification | SRU validation]]. This testing is to double, triple and quadruple check that there are no regressions in all the work that has been applied to bug fixes and updates for newer hardware.
Line 53: Line 57:

=== Thank-you ===

So, you have got this far? Thank-you! We were all once new to the family that is Ubuntu and felt worthless because we had so many questions, yet felt we could never answer them for others. In testing the [[https://launchpad.net/~ubuntu-testing | Ubuntu-QA team]] welcomes you. You are a new tester, you have hundreds of questions running through your mind. There is no such thing as a dumb question, if they are not answered on our [[https://wiki.ubuntu.com/QATeam | QA page]], please just ask! We will, as testers, answer any S.O.S. request from any of our family. We cannot stress how important you people are. Please do feel free to ask questions, these make the instructions better for everyone

== Virtual Machines ==

As some testing is carried out on Virtiual Machines (VMs), a quick note on them.

 * Advantage: If it dies, you can restart / re-install it easily without affecting your 'main' machine.
 * Dis-advantage: A VM does not fully check out your own computers' hardware setup.

Each cycle we hold tutorial / classroom sessions on these invaluable tools. If you'd like more information click on the link below to take you to that area.

 '''[[https://wiki.ubuntu.com/Testing/Activities/Classroom | All about tutorial / classroom sessions.]]'''

The Stages

There are some differences between how the ubuntu family carry out their testing cycles. Some only release at LTS, some use alpha testing, some use only beta testing and everyone has to use release candidate. You need to check with the team(s) who you are testing for to know which of the various options they are using. The dates of when important things happen?

Testing

Testing is split into distinct, but joined areas. The Daily Builds, the QA-testing of Milestone releases and the Milestone releases themselves.

As new features are added at the beginning of the cycle, Quality assurance and the testers check for bugs and test for correct function.

Daily Builds

The daily images are built automatically each 24 hours when not 'turned off' for milestone testing.

The most important goal with these images is simply to find as many bugs as possible and report them with enough detail that they can be fixed. Finding the bugs ahead of the milestone crunch is helpful as it gives us more time to fix them. Of course finding and reporting bugs in the released milestones is also useful for improving the overall quality of the distribution.

As there is less time pressure on testing these images than the pre-release ones, we would encourage more in-depth testing on a range of hardware to flush out the deeper hidden bugs. If possible, please perform installs using a variety of methods and focus on the quality and depth of testing as opposed to the quantity.

The Dailies are automatically generated every 24 hours, they get suspended as the QA cycles kick in.

Pre-Alpha

Scary, expect them to break! At this point in the cycle the developers (devs) are pushing the boundaries to the absolute limit to what they would like to be in the next release. Testing is always important during this time, so that the devs have some feed back as to how things are going. The use of Virtual Machines is handy at this point.

Alpha's

With the launch of the alpha's, we are into the area of 'it should not break too often'. The pre Alpha testing is to ensure that they boot. With a bootable Alpha 1 out, the devs then go to try adding things to it and updating things to the latest versions available for things. Things may well get broken. With the test version of the Alpha 2, things are little less concentrated on getting 'new' stuff in, but checking what is fixable.

The purpose of the Alpha milestones is to encourage wider public testing of the image builds. Before we can recommend the images for public testing we must at least ensure that they have all built correctly (are not in themselves corrupt) and are at least known to boot on some machines. Some form of installation should also be tested. We do not require that these images be tested for any possible minor defect or that all possible install methods work, since that is rather the goal of the wider image testing itself. Find the show stopper bugs and get them fixed.

When an important error is found, the current frozen images are rejected and new ones are built after the fix has been applied. It is therefore important to make sure that you are testing the most recent images.

If the bug is specific to one flavour, only those will be re-built and the other flavour builds remain valid. So if we find a serious bug in the Ubuntu version we will rebuild those images, which will take several hours in total. Likewise, if we find a bug that exists across all flavors, but only on i386, we will re-build i386 and all amd64 and PPC images remain valid.

Beta and Final Release Candidates

The first glimpse of what the final release may well look like. With the Betas, we mainly switch from testing on Virtual Machines to 'Real Computers'. No changes to what applications that will be shipped from this point should be altered without a very good reason. The first Beta is for people to widely check on as many computers as possible to check if certain types of computers and their associated internal parts are having problems. The second Beta is the prelude for the final version, things are now concentrated on bug-fixing.

For Beta and Final milestones we have extended testing periods where we try to find as many bugs as we can. With the Beta we want to put our best foot forward across the distribution and want to fix as many bugs as we have the manpower for, large or small. For the Final release on the other hand, we will only fix the remaining critical bugs from the previous release. This is because seemingly small changes can have unintended consequences and break other things. Minor changes that are merely a good idea to fix will be deferred to downloadable updates.

The procedure of rejecting known bad images and continuing testing of unaffected ones also applies here as with the Alphas.

Point Releases

For L.T.S.'s, These are what are is called point-releases. For example 12.04 LTS has had 4 point releases and is now on 12.04.4 In the meantime, 14.04 approaches it's first point relase as I write this. The testing time for point releases is very short and we do ask people to get involved. The changes for point releases have undergone SRU validation. This testing is to double, triple and quadruple check that there are no regressions in all the work that has been applied to bug fixes and updates for newer hardware.

Final Release

And that, is how we test & do things. If you fancy a roller-coaster ride then do get on. You will have the ability when the Release is made, to smile to yourself and to others. And to say "I was a part of that".

QATeam/Overview/TheStages (last edited 2015-01-23 16:17:15 by xdsl-83-150-81-40)