network-manager

Revision 20 as of 2015-10-02 19:46:22

Clear message

Description

This is a test plan for network-manager as used by Ubuntu Touch. It does not cover scenarios and/or test cases for network-manager as installed on the desktop. Please see https://wiki.ubuntu.com/NetworkManager/DistroTesting.

Basic Tests

These tests should be run for every upload, as they cover the basic functionality that ofono provides. They should be performed with a single valid ( ie. active data plan ), non-roaming, unlocked SIM card.

Note - make sure to check /var/crash to ensure that the new version of network-manager hasn't crashed and/or triggered other component crashes.

  • Test that the device automatically establishes a mobile data connection on boot.
    • The network indicator should reflect the type of connection ( eg. 3g, H, ... ).

    • The command nmcli d should show the modem ( eg. ril_0 ) as 'connected'. It also will show the ofono context used for the connection.

    • The command ip route should show a default route for the low-level rild device ( eg. ccmniX, or rmnet_usbX ). Note - depending on the APN settings, an optional host route for the same device may exist for a MMS Proxy or MessageCenter.

    • Check that IPv6 is disabled using the command nmcli c show <name|uuid|path>, and verify that the value of GENERAL.DEFAULT6 is no.

    • Use the Browser app to verify that the Internet is reachable.

    • Optional - restart the device multiple times to ensure that the mobile data connection is always established on boot. A dedicated stress test should be added and run on devices shipping Ubuntu.

  • Test that Wi-Fi can be enabled ( via System Settings and the network indicator ) and that access points are populated in the network menu. Note - some devices may only support 802.11b, so verify the device's capabilities before doing any Wi-Fi tests involving 5 GHz access points.

  • Test that the device can be associated with a Wi-Fi access point.

    • The network indicator should show a Wi-Fi-Connected icon.

    • The command nmcli d should show the Wi-Fi device ( eg. wlan0 ) as connected in addition to the low-level rild device ( see above ).

    • The command ip route should show new default and network routes for the Wi-Fi device. Note - depending on the phone, it's possible a network route may be visible for the low-level rild device ( see above ). Also as mentioned above, a host route for the low-level rild device may exist due to APN MMS settings.

    • Use the Browser app to verify that the Internet is reachable.
    • Restart the device and verify that the same Wi-Fi AP is automatically connected using the above checks.
  • Disable Wi-Fi and ensure that the mobile data connection is now active and the Internet reachable.
  • Change the modem's radio technology ( eg. 2g only ) via the Cellular System Settings. This causes the modem to re-register to the current operator with the new technology ( if available ), and thus NM will need to re-activate the ofono context once this happens.
    • Some devices enable the 3g capability to be assigned to a specific SIM slot. If this causes a modem reset on the device, this scenario should be tested to ensure that the mobile data connection is restored after such a change.

  • Enable FlightMode

    • enable Wi-Fi while FlightMode is enabled, verify connection, disable Wi-Fi

    • in lieu of an actual stress test for flight-mode, manually repeat this test ~5-10 times

  • Disable FlightMode and verify that the mobile data connection is restored.

  • Periodic Wi-Fi scanning occurs
    • Ensure that Wi-Fi is enabled

    • Note - Network`Manager adjusts the scan interval from between 20s to 120s, so the timing of the events will be depend on where in this cycle the scan interval currently sits.

    • As root, run the command wpa_cli, and ensure that periodic scanning is occurring. This should happen regardless of whether or not the device is connected to an access point. Optional - enter the command scan_results and verify that valid scan results are output. Verification of periodic scanning consists of watching for wpa_cli to output: 

           <3>CTRL-EVENT-SCAN-STARTED
           <3>CTRL-EVENT-SCAN-RESULTS

  • Run dbus-monitor --system --profile to ensure that when scanning occurs, there aren't duplicate PropertiesChanged signals sent for each AccessPoint object path ( ie. there should be a single DBus PropertiesChanged signal for each /AccessPoint/X ). See bug #1480877.

  • APs are promptly removed from the scan list ( bug #1425172 ):

    • This case tests that access points are removed from the available access point list in a timely manner.

    • Enable a hotspot using Android or iOS and wait till it appears in NM's scan list ( nmcli d wifi list ).

    • Disable the hotspot and time how long it takes for the AP to disappear from the scan list ( using the same command as in the previous step ).

    • Note - in general, an AP/hotspot should be removed <= 2m from the time it was disabled.

  • Wi-Fi out-of-range, auto-switch to mobile data
  • Wi-Fi available, auto-switch from mobile data to Wi-Fi

  • Force mobile data disconnect, verify that mobile data connection is re-established. Note - forcing mobile data to disconnect can be done by wrapping the device in tinfoil, and/or putting the device inside a metal container. This requires examination of the syslog to verify that the device actually disconnected.

  • Force an ofono ordered restart ("sudo restart ofono" for mako) and check that cellular data is recovered

    • NOTE - when running this test on arale, krillin, or vegata, the environment variable OFONO_RIL_DEVICE=mtk must be added to the end of the restart command. Likewise, on krillin or other devices with multiple SIM slots, the environment variable OFONO_RIL_NUM_SIM_SLOTS must be set to the correct number of SIMs supported by the device. This is due to the design whereby these variables are set by an upstart system job (rild.conf) which runs before the ofono job. This step is not required for the next test case.

  • Kill (with "kill -9 `pgrep ofonod`") ofono and check that cellular data is recovered

Preferred APNs

These tests ensure that Network`Manager will only attempt to activate ofono GPRS contexts which have a Preferred=true property. Clearing the property should have no immediate effect, although after doing so, the next time the connection is disconnected, NM will try all available contexts for a particular SIM in round-robin fashion. Lastly, setting Preferred=true on a context when another context is already activated for mobile data will cause NM to disconnect from the current context and attempt to activate the preferred context. If this fails, NM will not retry any other contexts.

The above code is currently being reviewed and will land post-OTA5 -- awe

Note - APN and context are used interchangeably below.

To test, you need to ensure that ofono has multiple gprs contexts defined for the SIM being used to test. The ofono script list-contexts can be used to examine the current SIM's provisioned contexts. It's best if more than one of these contexts are working contexts ( ie. they can be used to provide an active data call ). If the SIM you're using only provisions a single context, it's possible to add additional contexts with the same required properties, and a different Name in order to simulate an operator that provides multiple APNs for Internet. This can be done by using the ofono create-internet-context, or by stopping ofono and editing the correct SIM-specific gprs settings file ( /var/lib/ofono/<IMSI>/gprs ) directly, and then restarting ofono.

The only other script needed to exercise these test cases is the ofono set-context-property script: 

set-context-property [modem] [context_number] Preferred true|false

  • Set active context Preferred property to true; this should have no effect on the connection.

  • Reboot device, ensure that the preferred context is activated. If it fails, no other contexts should be activated.

    • As an extra step, you can review syslog and ensure that NM only added a system connection for the preferred context.

  • Set the active context Preferred property to false; ensure that this has no effect.

    • As an extra step, reboot, and then review syslog and ensure that NM adds system connections for all contexts read from ofono.

  • Set the Preferred property of a known working context to true. This should cause the mobile data connection to drop, and be re-established with the new preferred context. Note, it may take 30-45s for the connection to drop, and the new context to be activated.

    • Reboot the device, and ensure that the preferred context is activated.
    • Repeat, but this time choose a non-valid context. Ensure that the mobile data connection isn't restored. Reboot, and verify the same.

  • Verify that if multiple contexts are set as Preferred ( which is an invalid use case ), the first read from ofono is used. Note, perhaps we should change ofono to throw an Invalid`Parameter if this happens. - awe

Dual SIM Tests

We assume that the initial situation is cellular data using SIM1 enabled, SIM1 owns 3G capabilities, and no WiFi.

  • Switch data from SIM1 to SIM2. Check that cellular data is recovered.
  • Enable 3G for SIM2. Check that cellular data is recovered.
  • Switch data from SIM2 to SIM1. Check that cellular data is recovered.
  • Enable 3G for SIM1. Check that cellular data is recovered.
  • Switch cellular data off. Switch it to use SIM1. Check that cellular data is recovered.