Test plan for component: network-manager
Component Checklist: https://wiki.ubuntu.com/Process/Merges/Checklists/Common
Trunk URL: http://cgit.freedesktop.org/NetworkManager/NetworkManager/
Ubuntu Packaging Branches:
Ubuntu Package URL (LP):
Description
This is a test plan for network-manager as used by Ubuntu Touch. It does not cover scenarios and/or test cases for network-manager as installed on the desktop. Please see https://wiki.ubuntu.com/NetworkManager/DistroTesting.
Basic Tests
These tests should be run for every upload, as they cover the basic functionality that ofono provides. They should be performed with a single valid ( ie. active data plan ), non-roaming, unlocked SIM card.
Note - make sure to check /var/crash to ensure that the new version of network-manager hasn't crashed and/or triggered other component crashes.
- Test that the device automatically establishes a mobile data connection on boot.
- The network indicator should reflect the type of connection ( eg. 3g, H, ... ).
The command nmcli d should show the modem ( eg. ril_0 ) as 'connected'. It also will show the ofono context used for the connection.
The command ip route should show a default route for the low-level rild device ( eg. ccmniX, or rmnet_usbX ). Note - depending on the APN settings, an optional host route for the same device may exist for a MMS Proxy or MessageCenter.
Check that IPv6 is disabled using the command nmcli c show <name|uuid|path>, and verify that the value of GENERAL.DEFAULT6 is no.
- Use the Browser app to verify that the Internet is reachable.
Optional - restart the device multiple times to ensure that the mobile data connection is always established on boot. A dedicated stress test should be added and run on devices shipping Ubuntu.
Test that Wi-Fi can be enabled ( via System Settings and the network indicator ) and that access points are populated in the network menu. Note - some devices may only support 802.11b, so verify the device's capabilities before doing any Wi-Fi tests involving 5 GHz access points.
- Test that the device can be associated with a Wi-Fi access point.
The network indicator should show a Wi-Fi-Connected icon.
The command nmcli d should show the Wi-Fi device ( eg. wlan0 ) as connected in addition to the low-level rild device ( see above ).
The command ip route should show a new default and network route for the Wi-Fi device, with a lower priority than the default route for the mobile data device. Note - depending on the phone, it's possible a network route may be visible for the low-level rild device ( see above ). Also as mentioned above, a host route for the low-level rild device may exist due to APN MMS settings.
Run the command iwconfig and verify that Power Management is not off, and that a non-zero period is configured.
- Use the Browser app to verify that the Internet is reachable.
- Restart the device and verify that the same Wi-Fi AP is automatically connected using the above checks.
- Disable Wi-Fi and ensure that the mobile data connection is now active and the Internet reachable.
- Change the modem's radio technology ( eg. 2g only ) via the Cellular System Settings. This causes the modem to re-register to the current operator with the new technology ( if available ), and thus NM will need to re-activate the ofono context once this happens.
- Some devices enable the 3g capability to be assigned to a specific SIM slot. If this causes a modem reset on the device, this scenario should be tested to ensure that the mobile data connection is restored after such a change.
Enable FlightMode
enable Wi-Fi while FlightMode is enabled, verify connection, disable Wi-Fi
- in lieu of an actual stress test for flight-mode, manually repeat this test ~5-10 times
Disable FlightMode and verify that the mobile data connection is restored.
- Periodic Wi-Fi scanning occurs
- Ensure that Wi-Fi is enabled
Note - Network`Manager adjusts the scan interval from between 20s to 120s, so the timing of the events will be depend on where in this cycle the scan interval currently sits.
As root, run the command wpa_cli, and ensure that periodic scanning is occurring. This should happen regardless of whether or not the device is connected to an access point. Optional - enter the command scan_results and verify that valid scan results are output. Verification of periodic scanning consists of watching for wpa_cli to output:
<3>CTRL-EVENT-SCAN-STARTED
<3>CTRL-EVENT-SCAN-RESULTSRun dbus-monitor --system --profile to ensure that when scanning occurs, there aren't duplicate PropertiesChanged signals sent for each AccessPoint object path ( ie. there should be a single DBus PropertiesChanged signal for each /AccessPoint/X ). See bug #1480877.
As the Location Service's Geoclue provider relies on these signals, re-run dbus-monitor without the --profile argument, to ensure that LastSeen is included in the PropertiesChanged signals after a scan, and that the DBus type of LastSeen is int32.
APs are promptly removed from the scan list ( bug #1425172 ):
- This case tests that access points are removed from the available access point list in a timely manner.
Enable a hotspot using Android or iOS and wait till it appears in NM's scan list ( nmcli d wifi list ).
- Disable the hotspot and time how long it takes for the AP to disappear from the scan list ( using the same command as in the previous step ).
Note - in general, an AP/hotspot should be removed <= 2m from the time it was disabled.
- Wi-Fi out-of-range, auto-switch to mobile data
- Wi-Fi available, auto-switch from mobile data to Wi-Fi
Force mobile data disconnect, verify that mobile data connection is re-established. Note - forcing mobile data to disconnect can be done by wrapping the device in tinfoil, and/or putting the device inside a metal container. This requires examination of the syslog to verify that the device actually disconnected.
- Force an ofono ordered restart ("sudo restart ofono" for mako) and check that cellular data is recovered
NOTE - when running this test on arale, krillin, or vegata, the environment variable OFONO_RIL_DEVICE=mtk must be added to the end of the restart command. Likewise, on krillin or other devices with multiple SIM slots, the environment variable OFONO_RIL_NUM_SIM_SLOTS must be set to the correct number of SIMs supported by the device. This is due to the design whereby these variables are set by an upstart system job (rild.conf) which runs before the ofono job:
restart ofono OFONO_RIL_DEVICE=mtk OFONO_RIL_NUM_SIM_SLOTS=2
Kill (with "kill -9 `pgrep ofonod`") ofono and check that cellular data is recovered
Hotspot
These tests ensure that Hotspot functions correctly on supported devices. These tests should be performed with a SIM present that has an active/working data plan, and the mobile data enabled on the device.
- Using the System Settings, Enable Hotspot on the device without changing the default configuration, verify:
- the hotspot is visible to other devices ( preferably at least one non-Ubuntu device ).
- that one or more devices can connect to the hotspot, and that they're able to access the internet.
- that devices using the hotspot are able to do so for pro-longed periods of time ( ie. ensure that connected devices are able to renew their DHCP leases ).
- Disable Hotspot and verify:
- that any connected device is disconnected.
- that the hotspot no longer appears visible to other devices.
- that the device itself still has a valid mobile data connection.
that WiFi can be re-enabled on the device and connected to an access point.
- Verify that Hotspot can be used with varying configurations; Ex.
- disabled security ( ie. no password ).
- long passphrases ( ie. one with spaces and/or non-alphanumeric characters ).
- different SSIDs.
Preferred APNs
These tests ensure that Network`Manager will only attempt to activate ofono GPRS contexts which have a Preferred=true property. Clearing the property should have no immediate effect, although after doing so, the next time the connection is disconnected, NM will try all available contexts for a particular SIM in round-robin fashion. Lastly, setting Preferred=true on a context when another context is already activated for mobile data will cause NM to disconnect from the current context and attempt to activate the preferred context. If this fails, NM will not retry any other contexts.
Note - APN and context are used interchangeably below.
To test, you need to ensure that ofono has multiple gprs contexts defined for the SIM being used to test. The ofono script list-contexts can be used to examine the current SIM's provisioned contexts. This script and the other ofono scripts can be found in /usr/share/ofono/scripts. It's best if more than one of these contexts are working contexts ( ie. they can be used to provide an active data call ). If the SIM you're using only provisions a single context, it's possible to add additional contexts with the same required properties, and a different Name in order to simulate an operator that provides multiple APNs for Internet. This can be done by using the ofono create-internet-context, or by stopping ofono and editing the correct SIM-specific gprs settings file ( /var/lib/ofono/<IMSI>/gprs ) directly, and then restarting ofono.
The only other script needed to exercise these test cases is the ofono set-context-property script:
set-context-property [modem] [context_number] Preferred true|false
Set active context Preferred property to true; this should have no effect on the connection.
- Reboot device, ensure that the preferred context is activated. If it fails, no other contexts should be activated.
As an extra step, you can review syslog and ensure that NM only added a system connection for the preferred context.
Set the active context Preferred property to false; ensure that this has no effect.
As an extra step, reboot, and then review syslog and ensure that NM adds system connections for all contexts read from ofono.
Set the Preferred property of a known working context to true. This should cause the mobile data connection to drop, and be re-established with the new preferred context. Note, it may take 30-45s for the connection to drop, and the new context to be activated.
- Reboot the device, and ensure that the preferred context is activated.
- Repeat, but this time choose a non-valid context. Ensure that the mobile data connection isn't restored. Reboot, and verify the same.
Verify that if multiple contexts are set as Preferred ( which is an invalid use case ), the first read from ofono is used. Note, perhaps we should change ofono to throw an Invalid`Parameter if this happens. - awe
VPN Testing
These tests ensure functioning VPN support. These tests are valid on any Touch device with a WiFi adapter. A valid SIM is not required to test VPN.
- Using system-settings, configure an OpenVPN connection, specifying:
Server/[Port] - the hostname ( or IP address ) and port ( optional ) of the OpenVPN service ( eg. us.openvpn.mycompany.com )
Use this VPN for - Its own network ( don't change this unless specifically instructed for a specific VPN )
Type - OpenVPN ( currently the only type supported )
Protocol - UDP ( again unless instructed otherwise )
Authentication Type - Certificates (TLS)
Client Certificate - as supplied by the VPN provider
Private Key - as supplied by the VPN provider
Key Password (optional) - as supplied by the VPN provider
CA Certificate - as supplied by the VPN provider
Use additional TLS authentication - checked ( unless instructed otherwise )
TLS Key - as supplied by the VPN provider
Key direction - as supplied by the VPN provider
Verify peer certificate - checked ( unless instructed otherwise )
Peer certificate TLS type - Server
Cipher - as supplied by the VPN provider
Compress Data - checked ( unless instructed otherwise ).
Note - this might be referred to as LZO compression.
- Activate the VPN; the output of nmcli should look like this:
DEVICE TYPE STATE CONNECTION
tun0 tun connected tun0
wlan0 wifi connected hyperion - At this point, you should verify that the internet is accessible, and that resources on the internal VPN network(s) are reachable and/or resolvable ( ie. pinging a hostname at least resolves to an IP address.
- Disable VPN; confirm that internal resources can no longer be accessed, and that the internet is still accessible.
Enterprise WiFi
Add test cases for enterprise WiFi.
Dual SIM Tests
We assume that the initial situation is cellular data using SIM1 enabled, SIM1 owns 3G capabilities, and no WiFi.
- Switch data from SIM1 to SIM2. Check that cellular data is recovered.
- Enable 3G for SIM2. Check that cellular data is recovered.
- Switch data from SIM2 to SIM1. Check that cellular data is recovered.
- Enable 3G for SIM1. Check that cellular data is recovered.
- Switch cellular data off. Switch it to use SIM1. Check that cellular data is recovered.
Statistics Interface Testing
This tests are for testing the statistics interface, which sends signals about network traffic. Steps for testing are:
1. Identify modem and WiFi object paths. This can be done using
dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.DBus.Properties.Get string:"org.freedesktop.NetworkManager" string:Devices
and then, for each device
dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager/Devices/<N> org.freedesktop.DBus.Properties.Get string:"org.freedesktop.NetworkManager.Device" string:DeviceType
(see https://developer.gnome.org/NetworkManager/unstable/nm-dbus-types.html#NMDeviceType for types).
2. Activate statistics for modem and WiFi.
Statistics can be activated with a maximum reporting rate of <milliseconds> by doing:
dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager <device_object_path> org.freedesktop.DBus.Properties.Set string:"org.freedesktop.NetworkManager.Device.Statistics" string:RefreshRateMs variant:uint32:<milliseconds>
After that, PropertiesChanged signals will be emitted if the given statistics change. We should see the signals by monitoring the system bus with
dbus-monitor --system
The current statistics implemented in org.freedesktop.NetworkManager.Device.Statistics are TxBytes and RxBytes.
An example of what be should see is:
signal sender=:1.13 -> dest=(null destination) serial=1404 path=/org/freedesktop/NetworkManager/Devices/6; interface=org.freedesktop.NetworkManager.Device.Statistics; member=PropertiesChanged
- array [ ]
3. Stop reporting.
To stop reporting, we must set RefreshRateMs to 0:
dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager <device_object_path> org.freedesktop.DBus.Properties.Set string:"org.freedesktop.NetworkManager.Device.Statistics" string:RefreshRateMs variant:uint32:0