KrillinTestReport20141119

krillin/rtm #162, fresh flash, wipe, AT&T SIM, US ( limited to edge )

basic tests

call out / in - ok
sms out / in - ok
call/ sms wake phone - ok
gprs connection - ok
verify no ofono crashes - ok

** made image writable; reboot ***

auto-pilot tests

** installed dialer-app-autopilot ** ** installed message-app-autopilot **

ran dialer tests: phablet-run dialer-app
- 8/8 FAILED; missing python3-evdev

** installed python3-evdev **

7/8 FAILED
- unlocked screen, re-ran: 6/8 FAILED
- pinned display on ( powerd-cli ): 8/8 OK!!!
  - checked /var/crash: found new urfkill crash
ran messaging tests:
- 12/15 FAILED
- tried again, same result

** installed indicator-network-autopilot **

unity8 must be stopped before running tests...
ran indicator-network-autopilot tests:
- ran one test & it failed; mail to Antti

** re-installed rtm #162 --wipe **

** re-install just messaging_app_autopilot & python3-evdev; reboot **

ran message_app tests:
- ran 15/15 - OK!!

** re-install dialer_app_autopilot; reboot **

ran dialer_app tests:
- failed 6/8!! ->>>>>>>> MISTAKE; screen !unlocked or pinned. ;(
  - ran 8/8 OK!
  - ran 8/8 OK! --> reboot
ran messaging_app tests:
- ran 15/15 OK!

** installed indicator-network-autopilot; reboot **

ran indicator_network tests:
- fails same as before...

** re-installed rtm **#163** **

sim

sim presence

1. no sim(s) - modem online - FAIL

[ no services stopped ]

FAIL: Muted = 0 ( differs from mako which is 1 ) ACTION: file bug
ACTION: file bugs for 'Fake Manufacturer' and 'Fake Model'
test-script w/verify possible: yes
- requires manual setup

2. no sim(s) - modem offline - OK

NOTE: krillin and mako still differ ( see SIM test page ); any verification script needs to be quirked.
see existing bug: https://bugs.launchpad.net/ubuntu-rtm/+source/ofono/+bug/1366186
test-script w/verify possible: yes ( w/hardware quirks ); see next test case
- requires manual setup

3. sim(s) present - modem offline - OK

test-script: rilmodem/test-modem-offline
- requires manual setup
NOTE: verification is almost identical to the previous case, as when offline, no SimManager interface exists for krillin.
ACTION: improve test script to mimic the previous test case for krillin; it currently only validates the top-level ModemManager properties.

5. SIM Removal Notification - OK**

due to the architecture of krillin, SIM removal causes the teardown and rebuilding of the SimMananger interface, therefore a PropertyChange message is never received for Present ( unlike mako ).

SIM PIN/PUK Tests

1. SIM PIN Locked - OK

only tested for slot 1 ( /ril_0 )

2. SIM PIN Retries - OK**

only tested for slot 1 ( /ril_0 )
** - SimManager::Retries property is only initialized/updated after some kind of PIN operation occurs ( eg. a failed PIN/PUK attempt, resetting a PIN, ... ). This is krillin-specific quirk.

3. SIM PUK Locked - OK

only tested for slot 1 ( /ril_0 )

4. Enable PIN Locking - OK

only tested for slot 1 ( /ril_0 )

5. Disable PIN Locking - OK

only tested for slot 1 ( /ril_0 )

6. Changing a PIN - OK

only tested for slot 1 ( /ril_0 )

Voicecall

Basic Calling

1. Outgoing Call - Unanswered - OK

2. Outgoing Call - Dropped - OK **

** - the test case isn't clear whether or not the call is rejected or answered and then hung up.

Emergency Calling

Not run due to lack of infrastructure.

Call Hold/Swap

only tested for slot 1 ( /ril_0 )

1. Hold an Active Call - OK

2. Swap Between Calls - OK

3. Hangup Call On-Hold - OK

4. Hangup Active Call & Swap to Call On-Hold - OK

5. Hangup Active Call While Call on Hold - OK

6. Verify Held Call Notification - N/A

Can't test in US. AT&T doesn't support Call Hold Suppl service.

Multiparty

1. Create Multi-Party Call - OK

2. Split Multi-Party Call - OK

Call Barring

Note - no test cases run due to lack of support for call barring by US operators.

Call Forwarding

1. Verify Interface Availablility / Default Properties - OK

2. Test basic querying and property setting - FAIL

Note - the script is broken with multiple errors ( mixed tabs/spaces, problems specifying modem ).

3. Test call forwarding unconditional - OK

4. Test call forwarding when mobile is busy - OK**

** - if the subscriber line is configured for voicemail, clearing VoiceBusy via "##67#" may just reset VoiceBusy back to a voicemail number, so it's not possible to get a busy signal from the DUT.

5. Test call forwarding when user does not respond - OK

6. Test call forwarding when mobile is not reachable - OK**

** - if the subscriber line is configured for voicemail, then even though the phone is turned off, the call will not be forwarded. The only way to do this is to disable VM. If possible, this needs to be added to the test case.

7. Test setting globally call forwarding - OK

8. Test setting globally conditional call forwarding - OK

9. Test that call forwarding works - FAIL