Exposure Notifications testing and quality considerations

This documentation provides an overview of a recommended test strategy and quality considerations for developers using the Exposure Notification API. This API is provided by Apple and Google to build apps to track exposure of app users to others who are diagnosed with COVID-19.

Background and motivations

Our goal is to ensure that Exposure Notification works for everyone. This is both a broad and deep challenge due to the significant diversity of devices in the marketplace, the ways in which these devices are used, and the environments that these devices exist in. Across Android devices, there is significant variation in performance and capability in those devices ranging from $20 USD up to flagship devices. The Exposure Notification API on Android is supported for devices running Android API level 23 and higher.

The considerations fall into three broad categories:

This document provides a high level overview of a recommended testing strategy to help app/server developers focus their time on the appropriate dimensions of testing and verification.

A mix of automated and manual testing techniques is encouraged.

Functional testing considerations

These considerations revolve around the core functionality of the software stack, from server, to client, to API. If these are met, each component should perform to specification.

Foundational automated testing

Summary: You should follow best practices with unit and functional tests.

  • Write automated unit tests and your code/branch coverage should reach levels that match industry accepted standards.
  • Run your automated tests as often as possible in a CI/CD environment.

BLE testing

Summary: Your application should be able to handle different situations with Bluetooth Low Energy (BLE).

  • Test for precision and recall with multiple devices.
  • Test for expected behavior with different radio modes (airplane, Bluetooth [BT] off, BT on, and so on).
  • Test for concurrent Wi-Fi/BT radio use (such as music playback).

Interoperability testing

Summary: Your applications on Android and iOS should interoperate. This is important to ensure epidemiological viability.

  • Test Android and iOS devices together.

End-to-end functional testing

Summary: Your software stack should perform to specification.

  • Test your server stack with your client app and the API.
  • Test all high-priority user scenarios.
  • Test server key signing with your client app.
  • Test that you are able to get matches on the client app with an end-to-end flow.

UX/UI testing

Summary: Your UI/UX should perform to specification.

  • You should build automated UI/UX tests with and without mocking, for functional tests and for end-to-end tests.
  • Consider testing with an automated crawler to detect UI changes and other non-functional behavior of the app, such as crashes.
  • Ensure that your UI/UX is accessible and meets local/regional regulations and guidelines.

Server functional tests and/or A/B testing

Summary: Your server stack should perform to specification.

  • You should test your server stack independent of your client.
  • Consider A/B-testing your server stack with canned or recorded traffic.

Security, privacy, and counter-abuse

Summary: Your software should be secure, respect user privacy, and be tested against abuse.

  • Adhere to standard security practices for the server and app.
  • Perform a detailed review of the app’s behavior against applicable privacy laws and policies.
    • Design your app so that you can programmatically verify these checks and ensure programmatic checks exist to verify the app is behaving as expected.
  • Plan for and test against abuse scenarios on the app and server.

Non-functional testing considerations

These considerations revolve around the non-functional performance of the software stack. If these are met, the user should have an experience that is not only functional, but performs well.

Power and performance

Summary: Your software should minimize power usage and perform well.

  • Note that tests for power may require hardware power measurement devices.
  • Check that wakeups and wakelocks match expectations.
  • Test performance (for example, CPU, memory utilization, and latency) via automated methods across a mix of device models that are representative of your user population.
  • For Android, compare battery usage patterns when exposure notifications are enabled and disabled using Battery Historian.

Degraded network or poor network connectivity

Summary: Ensuring that communications between server and client are resilient to poor network connections is important so that matching occurs at a constant frequency.

  • Ensure server/client communication, especially for large data payloads, is able to resume and recover from lost packets, lost connectivity, high latency, and so on.

No Wi-Fi connectivity

Summary: Ensure that users on mobile-only access have full functionality of the app.

  • Because some users will not have access to Wi-Fi, ensure you do not rely on Wi-Fi to download keys from the server.

Low RAM and/or storage

Summary: Ensure that users who have low RAM or storage are able to use the app and service.

  • Ensure that you perform memory testing.

  • Ensure that your app is able to work on low storage or alert on it.

Low battery and battery saver

Summary: Ensure that users who have low battery or have battery saver turned on continue to experience proper behavior.

  • Test on devices with low battery.
  • Test on devices with battery saver turned on.

Server load

Summary: Ensure that server performs under load.

  • Test the performance of the server under load.

Hardening and edge cases

Summary: Ensure that the software is hardened.

  • Consider using techniques like mutation or fuzz testing to harden (that is, make resilient to vulnerabilities) the server/client code.

Android versions

Summary: Ensure that your app functions on supported Android versions.

  • Test compatibility from Android API level 23 through the upcoming Android 11 release. The Android Developer site has instructions for installing Android 11 Beta.

Scaling & release considerations

These considerations revolve around scaling the software to all devices in a certain region as well as any release considerations.

App testing lab

Summary: Ensure that you have the proper devices to test.

  • Have the most popular devices for your region.
  • Have the most popular low-end devices.
  • Procure and run iOS and Android devices in the lab.
  • Consider building automated test harnesses to run tests continuously.
  • Consider automating multi-device workflow tests.

Telemetry and logging

Summary: Ensure that telemetry and logging conform to specifications and comply with all applicable privacy regulations and policy.

  • Implement logging and telemetry tests.
  • Consider creating dashboards to monitor field telemetry to detect anomalies not seen in automated testing, such as crashes.
  • Ensure you are able to collect and triage user feedback.
  • See Security, privacy, and counter-abuse.

Release

Summary: Ensure that any release operations are tested.

  • Test rollouts, rollbacks, and roll-forwards.
  • Test app storage compatibility across versions.
  • For Android, consider following best practices for App releases.

Modeling and simulation

Summary: Ensure that the software performs in real-world usage scenarios.

  • Consider testing in settings where you expect the software to be used (schools, workplaces, hospitals, restaurants, transit, rush hour traffic at different densities, high density housing, and so on).
  • Consider testing precision and recall of BLE signals.

Beta testing and pilots

Summary: Ensure that you test with a significant number of users to ensure functional performance of the system.

  • Consider adding a beta test or a pre-production channel for early access.
  • Consider a pilot with hundreds of users.