Test Runner Documentation

  • The test_runner.py script runs unit test binaries across multiple platforms, requiring at least --platform and --config arguments.

  • To run a specific test, use the --target_name parameter with the name of the test binary.

  • Tests can be built using the -b flag and run immediately after using the -br flag.

  • The TEST_TARGETS variable in starboard_configuration.py lists all test binaries to be run by the test runner, unless filtered out.

  • Tests can be filtered for specific platforms, configurations, or targets using GetTestFilters() in the TestFilters subclass.

The scripts in this folder comprise a cross-platform unit test runner. The components of this runner are as follows:

Components

1.) test_runner.py

This script is used to run unit test binaries on multiple platforms. To see a full list of parameters that can be supplied to the script, run:

python test_runner.py --help

Running Tests:

To run all tests for a given platform, execute the test_runner.py script and provide at minimum the --platform and --config arguments.

Example:

python test_runner.py --platform=android-x86 --config=devel

Running this command will run all unit tests for the devel build of the android-x86 platform. To specify a device to run the tests on, provide the --device_id argument, as shown below:

python test_runner.py --platform=android-x86 --config=devel \
  --device_id=emulator-4

Running a Single Test:

If you would like to run a single unit test binary and view its output, you can do so by using the --target_name parameter and providing a test binary name, as shown below:

python test_runner.py --platform=android-x86 --config=devel \
  --device_id=emulator-4 --target_name=audio_test

Building Tests:

You can also use this script to build your unit test binaries before running them. To do this, provide the -b command line flag. If you would like to build the tests and then run them, provide the flags -br.

2.) Master list of test binaries

In your application's starboard_configuration.py file, define a variable called TEST_TARGETS. It should be a list containing the names of all of the test binaries that you want the test runner to run. An example is shown below:

TEST_TARGETS =[
    'audio_test',
    'bindings_test',
]

If your 'starboard_configuration.py' file contains this list, then every platform you support in Starboard will try to run these test binaries unless they are filtered out as described below.

Filtering Tests

To filter out tests that you do not want to run for a specific platform, implement a method within the platform's TestFilters subclass called GetTestFilters(). The TestFilters subclass lives in the test_filters.py file for each platform. If the tests are application-specific, you may define GetTestFilters() on an optional ApplicationConfiguration subclass, which will be found in the <platform-directory>/<application-name>/configuration.py subdirectory. See this Linux implementation for an example.

The GetTestFilters() function should return a list of TestFilter objects, which can be constructed by importing starboard.tools.testing.test_filter. To make a TestFilter object, provide the constructor with the test target name, the name of the actual unit test that the target runs, and optionally, the build configuration from which the test should be excluded. An example is shown below:

test_filter.TestFilter('math_test', 'Vector3dTest.IsZero', 'debug')

If a configuration is not provided, then the test will be excluded from all configurations.

To filter out all tests for a particular target, provide test_filter.FILTER_ALL as the test name.

To disable unit testing for all targets and all configurations, return a list containing test_filter.DISABLE_TESTING.

Environment Variables

If a platform requires extra environment variables in order to run tests properly, implement a method called GetTestEnvVariables() in the same PlatformConfiguration or ApplicationConfiguration mentioned above. The application-level variables will be merged on top of the platform-level variables. There is an example of this method in the provided Linux implementation. The method should return a dictionary that maps test binary names to dictionaries of environment variables that they need.

Example:

def GetTestEnvVariables(self):
  return {
      'base_unittests': {'ASAN_OPTIONS': 'detect_leaks=0'},
      'crypto_unittests': {'ASAN_OPTIONS': 'detect_leaks=0'},
      'net_unittests': {'ASAN_OPTIONS': 'detect_leaks=0'}
  }