ubuntu-session

Differences between revisions 21 and 22
Revision 21 as of 2016-02-29 21:55:35
Size: 5527
Editor: localhost
Comment: add note about kernel modules and link to BasicTests
Revision 22 as of 2016-04-27 13:32:37
Size: 10553
Editor: kzapalowicz
Comment:
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
== Description ==

This is a test plan for bluez as used by Ubuntu Touch.  It ''does not'' cover scenarios and/or test cases for bluez installed on the desktop.
= Description =

This is a test plan for bluez as used by Ubuntu Touch as well as it covers, to small extent, scenarios and/or test cases for bluez installed on the desktop.
Line 14: Line 14:
== Test Plan ==

Test target device: Nexus 4, BQ Aquaris E4.5
Secondary/backup device: TBD

Initial set up:
== Initial set up ==
Line 27: Line 22:
== Kernel Bluetooth Management Layer tests == This section contains the detailed description of the manual testing process with explanation of the scope as well as the list of test cases.

== Test Plan Template ==

The Bluetooth Test Plan template (https://docs.google.com/a/canonical.com/spreadsheets/d/1ifvm4HkCbbDe3zizSRuIsKUJDgQonhK3jx33lvEjfjg/edit?usp=sharing) defines the test cases for the Bluetooth. It covers the high (user) and low level (kernel) tests as well as it defines the tests for Bluetooth profiles.

Test target Ubuntu devices: Nexus 4, BQ Aquaris E4.5, etc... using as many Bluetooth devices as possible. Remember that the variety of devices used increases the test coverage.

=== Description ===

In particular it contains two major test sets labeled as '''Smoke Tests''' and '''Full Tests'''. The purpose of which is first to provide a quick way of checking for regressions by verification of the basic functionality, second to provide an in-depth guidance for the full testing of the Bluetooth related components.

It is recommended that the '''Smoke Tests''' are executed whenever there is a minor update which does not have a big impact on the Bluetooth susbsystem but is rather limited in it's scope. By contrast the '''Full Tests''' shall be performed for a major updates which do have a big impact. An example of such change would be updating BlueZ version. In addition to that the minor changes related to one domain such as for example Music over Bluetooth [A2DP/AVRCP] might require, in addition to smoke testing everything performing full check on the of the media.

||Smoke Tests||This short collection of tests that is designed to act as a quick vay to verify if introduced change either does not cause any regressions or do not have negative impact on the user experience. These are meant to be performed after every minor update.||
||Full Tests||This is the full tests matrix with much more detailed scenarios than the Smoke Tests. It is especially designed for testing for major updates or releases such as updating the BlueZ version and/or kernel.||

=== Usage ===

The good way of working with this sheet would be to copy it for a test run and obviously save it with a meaningfull name, then use the tabs to track test results per device. The goal is to have a single tab for device labeled, for example, as "Full Logitech k480" which would mean that the full test have been executed using Logitech k480 device. The single sheet should contain the tabs for each tested device as this is important for tracking the test results.

Try to test with as many devices as possible so that the coverage is good.

Way of working:
 * save the tamplate sheet under new, meaningfull name
 * copy a relevant tab [Smoke or Full] and name it after the device A
 * execute the test steps using device A and save the results to the sheet
 * copy a relevant tab [Smoke or Full] and name it after the device B
 * execute the test steps using device B and save the results to the sheet
 * ...
 * copy a relevant tab [Smoke or Full] and name it after the device N
 * execute the test steps using device N and save the results to the sheet

At the end it is recommended to link the test results on the [[/TestResults|Test Results]] page so that they are easilly tracked.

== Test Sets ==

This section provides an explanation of each of the test sets.

|| '''Test set''' || '''Brief Description''' ||
|| User ||The User set of tests is to cover the use cases valid for a regular user who would normally not use the Bluetooth in ony other way [using bluetoothctl for example]. These tests are limited to the interaction that goes through the Ubuntu System Settings. In particular the functionality such as discovering devices, connecting to devices or Bluetooth state are checked here to verify that there are no major flaws.||
|| Kernel ||The Kernel test set tests the Kernel Bluetooth Management Layer. This is to test the Bluetooth stack which is encapsulated in the Linux Kernel. These tests verify that the low-level facilities are working correctly which is a prerequisite for any other set of tests. Note that these tests shall be performed only when the kernel is updated or BlueZ version changed and can be skipped if the update touches only the userland components.||
|| General ||The General set of tests focuses on the commandline Bluetooth manager which is a layer between the System Settings and the Kernel. These tests repeat the use-cases from the User set of tests.||
|| Music ||These tests are here to verify that the A2DP and AVRCP Bluetooth profiles are operating as expected allowing the end users to listen to music using their Bluetooth enabled headsets and speakers||
|| HFP ||These tests are here to verify that the Handsfree functionality is implemented providing means to make phone calls using the remote Bluetooth accessories.||
|| HID ||These tests are here that the Bluetooth enabled [Classic and Low Energy] keyboards and mices can be used with the Ubuntu based devices such as: ubuntu phone, ubuntu tablet and the desktop.||

=== Kernel a.k.a. Kernel Bluetooth Management Layer ===
Line 78: Line 120:
=== Adapter ===

As first thing we need to verify the Bluetooth adapter was correctly initialized and is ready to be used. Check with
'''Adapter'''

To verify that the Bluetooth adapter was correctly initialized and is ready to be used use the hciconfig. Check with
Line 86: Line 128:
that an HCI interface is present (normally hci0). Output should be similar to this that an HCI interface is present (normally hci0). Output should be similar to this:
Line 97: Line 139:
'''NOTE:''' The state of the interface should correspond to the Bluetooth indicator state in the UI. In the example above the interface is '''''DOWN''''' so the indicator should show disabled Bluetooth too. 

As next step we verify the adapter is correctly connected the in-kernel management layer.
'''NOTE:''' The state of the interface should correspond to the Bluetooth indicator state in the UI. In the example above the interface is '''''DOWN''''' so the indicator should show disabled Bluetooth too.



To
verify that the adapter is correctly connected the in-kernel management layer use the btmgmt.
Line 107: Line 151:
The output should be similar to this The output should be similar to this:
Line 121: Line 165:
=== HCI === '''HCI'''
Line 129: Line 173:
=== Management Interface === '''Management Interface'''
Line 139: Line 183:
=== RFCOMM === '''RFCOMM'''
Line 149: Line 193:
=== BT User channel === '''BT User channel'''
Line 159: Line 203:
=== BT BNEP === '''BT BNEP'''
Line 169: Line 213:
== Bluetooth management daemon ==

This tests the bluetooth management daemon running as part of the userland. Its binary name is ''bluetoothd''.
=== General a.k.a Bluetooth management daemon ===

This tests the bluetooth management daemon running as part of the userland. It's binary name is ''bluetoothd''.

Description

This is a test plan for bluez as used by Ubuntu Touch as well as it covers, to small extent, scenarios and/or test cases for bluez installed on the desktop.

NOTE: This only covers BlueZ 5.x and doesn't respect 4.x in any way.

Initial set up

  • Install latest image on phone
  • Install BlueZ from the silo PPA
  • Reboot the phone

Manual Tests

This section contains the detailed description of the manual testing process with explanation of the scope as well as the list of test cases.

Test Plan Template

The Bluetooth Test Plan template (https://docs.google.com/a/canonical.com/spreadsheets/d/1ifvm4HkCbbDe3zizSRuIsKUJDgQonhK3jx33lvEjfjg/edit?usp=sharing) defines the test cases for the Bluetooth. It covers the high (user) and low level (kernel) tests as well as it defines the tests for Bluetooth profiles.

Test target Ubuntu devices: Nexus 4, BQ Aquaris E4.5, etc... using as many Bluetooth devices as possible. Remember that the variety of devices used increases the test coverage.

Description

In particular it contains two major test sets labeled as Smoke Tests and Full Tests. The purpose of which is first to provide a quick way of checking for regressions by verification of the basic functionality, second to provide an in-depth guidance for the full testing of the Bluetooth related components.

It is recommended that the Smoke Tests are executed whenever there is a minor update which does not have a big impact on the Bluetooth susbsystem but is rather limited in it's scope. By contrast the Full Tests shall be performed for a major updates which do have a big impact. An example of such change would be updating BlueZ version. In addition to that the minor changes related to one domain such as for example Music over Bluetooth [A2DP/AVRCP] might require, in addition to smoke testing everything performing full check on the of the media.

Smoke Tests

This short collection of tests that is designed to act as a quick vay to verify if introduced change either does not cause any regressions or do not have negative impact on the user experience. These are meant to be performed after every minor update.

Full Tests

This is the full tests matrix with much more detailed scenarios than the Smoke Tests. It is especially designed for testing for major updates or releases such as updating the BlueZ version and/or kernel.

Usage

The good way of working with this sheet would be to copy it for a test run and obviously save it with a meaningfull name, then use the tabs to track test results per device. The goal is to have a single tab for device labeled, for example, as "Full Logitech k480" which would mean that the full test have been executed using Logitech k480 device. The single sheet should contain the tabs for each tested device as this is important for tracking the test results.

Try to test with as many devices as possible so that the coverage is good.

Way of working:

  • save the tamplate sheet under new, meaningfull name
  • copy a relevant tab [Smoke or Full] and name it after the device A
  • execute the test steps using device A and save the results to the sheet
  • copy a relevant tab [Smoke or Full] and name it after the device B
  • execute the test steps using device B and save the results to the sheet
  • ...
  • copy a relevant tab [Smoke or Full] and name it after the device N
  • execute the test steps using device N and save the results to the sheet

At the end it is recommended to link the test results on the Test Results page so that they are easilly tracked.

Test Sets

This section provides an explanation of each of the test sets.

Test set

Brief Description

User

The User set of tests is to cover the use cases valid for a regular user who would normally not use the Bluetooth in ony other way [using bluetoothctl for example]. These tests are limited to the interaction that goes through the Ubuntu System Settings. In particular the functionality such as discovering devices, connecting to devices or Bluetooth state are checked here to verify that there are no major flaws.

Kernel

The Kernel test set tests the Kernel Bluetooth Management Layer. This is to test the Bluetooth stack which is encapsulated in the Linux Kernel. These tests verify that the low-level facilities are working correctly which is a prerequisite for any other set of tests. Note that these tests shall be performed only when the kernel is updated or BlueZ version changed and can be skipped if the update touches only the userland components.

General

The General set of tests focuses on the commandline Bluetooth manager which is a layer between the System Settings and the Kernel. These tests repeat the use-cases from the User set of tests.

Music

These tests are here to verify that the A2DP and AVRCP Bluetooth profiles are operating as expected allowing the end users to listen to music using their Bluetooth enabled headsets and speakers

HFP

These tests are here to verify that the Handsfree functionality is implemented providing means to make phone calls using the remote Bluetooth accessories.

HID

These tests are here that the Bluetooth enabled [Classic and Low Energy] keyboards and mices can be used with the Ubuntu based devices such as: ubuntu phone, ubuntu tablet and the desktop.

Kernel a.k.a. Kernel Bluetooth Management Layer

NOTE: You only have to run those test when you test a new kernel. If just the userland components were updated these test steps are not required to be executed.

BlueZ comes with several utilities to test the in-kernel bluetooth management layer. Those are

  • bnep-tester
  • hci-tester
  • mgmt-tester
  • rfcomm-tester
  • userchan-tester

Each of them tests a different functionality part of the kernel bluetooth management layer. We're going to run all of them as part of this test plan.

Note - the following will not work on Touch devices as of Jan '16, as most if not all Touch devices have kernel modules turned off by default. Please skip the following testing unless you're preparing a new kernel release for a device...

As preparation for some of them we have to do the following:

  • Stop the bluetoothd daemon

$ sudo service bluetooth stop
  • Make sure bluetooth is turned off but rfkill not blocked.

$ sudo rfkill unblock bluetooth
$ sudo hciconfig hci0 down
  • Load VHCI kernel module as this one will not be loaded automatically as only used for testing

$ sudo modprobe hci_vhci

FIXME: As of right now the tools used below are not part of any package. You have to build bluez manually with --enable-experimental to get them. This was reported as a bug at Debian and should be fixed soon. Another way to get those tools is:

$ wget https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-043/+files/bluez-test-scripts_5.33-0ubuntu6_armhf.deb
$ dpkg-deb -x bluez-test-scripts_5.33-0ubuntu6_armhf.deb .
$ find usr/bin
usr/bin
usr/bin/bnep-tester
usr/bin/l2cap-tester
usr/bin/sco-tester
usr/bin/mgmt-tester
usr/bin/smp-tester
usr/bin/gap-tester
usr/bin/hci-tester
usr/bin/userchan-tester
usr/bin/rfcomm-tester

All tools are then available in the usr/bin folder without the need to install anything.

Adapter

To verify that the Bluetooth adapter was correctly initialized and is ready to be used use the hciconfig. Check with

$ hciconfig

that an HCI interface is present (normally hci0). Output should be similar to this:

phablet@ubuntu-phablet:~$ hciconfig
hci0:   Type: BR/EDR  Bus: UART
        BD Address: 4C:74:03:64:82:1F  ACL MTU: 1021:4  SCO MTU: 184:1
        DOWN 
        RX bytes:620 acl:0 sco:0 events:31 errors:0
        TX bytes:411 acl:0 sco:0 commands:31 errors:0

NOTE: The state of the interface should correspond to the Bluetooth indicator state in the UI. In the example above the interface is DOWN so the indicator should show disabled Bluetooth too.

To verify that the adapter is correctly connected the in-kernel management layer use the btmgmt.

NOTE: The btmgmt utility is only available on systems with BlueZ 5.x

$ btmgmt info

The output should be similar to this:

Index list with 1 item
hci0:   Primary controller
        addr 4C:74:03:64:82:1F version 6 manufacturer 70 class 0x000000
        supported settings: powered connectable fast-connectable discoverable bondable link-security ssp br/edr hs le advertising secure-conn debug-keys privacy static-addr 
        current settings: br/edr 
        name MTK MT6582 #1
        short name

If no controller is listed the test step should be considered as failed.

HCI

Run the HCI tester with

$ sudo hci-tester

Afterwards we have to evaluate the output. Failed commands with status 0x1 (Unknown HCI command) can be ignored as those are then just not supported by the Bluetooth version the device supports. All other failed tests with a different status code should be considered as failed.

Management Interface

Run the management interface tester with

$ sudo mgmt-tester

All tests should pass.

RFCOMM

Run the RFCOMM tester with

$ sudo rfcomm-tester

All tests should pass.

BT User channel

Run the user channel tester with

$ sudo userchan-tester

All tests should pass.

BT BNEP

Run the BNEP tester with

$ sudo bnep-tester

All tests should pass.

General a.k.a Bluetooth management daemon

This tests the bluetooth management daemon running as part of the userland. It's binary name is bluetoothd.

The following is the list of function-specific test plans:

Process/Merges/TestPlans/ubuntu-session (last edited 2017-09-05 10:17:00 by jibel)