Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Smoke testing Application Services against end-user apps

This is a great way of finding integration bugs with application-services. The testing can be done manually using substitution scripts, but we also have scripts that will do the smoke-testing for you.

Dependencies

Run pip3 install -r automation/requirements.txt to install the required Python packages.

Usage

You can easily run a smoke test against iOS and Fenix by running the following:

./automation/build_against_all.py --firefox-dir ../firefox --action build-without-testing --allow-clears

  • In this case, firefox-dir must point to a bootstrapped and working installation of mozilla-central (see instructions here). It is used for the compilation and test of the Android and HNT builds.

  • The --action argument can be either build-without-testing or run-tests.

  • The --allow-clears argument allows the various subscripts to clear their various caches as appropriate (such as XCode cleaning iOS caches).

You can also run against specific platforms directly with the following examples:

  • iOS:

    ./automation/build_against_ios.py --action build-without-testing --clear-previous-bindings --clean-ios-caches --use-local-firefox-ios ../firefox-ios

    • By default, this script creates a tmp directory for firefox-ios and uses it. If you have a running firefox-ios, you can add the optional argument --use-local-firefox-ios ../firefox-ios (as shown), which may result in speed gains and the ability to run it on XCode immediately after a failure. This can also be passed to ./build_against_all.py.

    • You can also customize the run scheme (--ios-scheme) or test plan (--ios-test-plan). Available schemes include Fennec (default) and Firefox. Available test plans include: Smoketest, FullFunctionalTestPlan and UnitTest.

      • A full list of schemes and their corresponding test plans can be found in the firefox-ios respository.

      • Both of these arguments can be used in ./build_against_all.py, and they will be passed to the underlying build_against_ios.py call.

  • Fenix:

    ./automation/build_against_fenix.py --action build-without-testing --firefox-dir ../firefox --prefix-ff fenix --clear-previous-bindings

    • The --prefix-ff argument here refers to the prefix passed to commands like ./gradlew fenix:assembleDebug. It can be omitted, but may cause failures on non-fenix projects.

    • The --clear-previous-bindings argument here runs a ./gradlew prefix:clear before recompiling. It is not always necessary, and can be excluded for some speed gains, but can result in some cache reuse.

  • Desktop / HNT (Home & New Tab):

    ./automation/build_against_hnt.py --action build-without-testing --firefox-dir ../firefox

    • This script uses the symlink method described at the local A-S against HNT tutorial, cleaning up after a successful run. You can avoid this cleanup process (specifically on a successful run) by passing --no-clean-up, which will keep the symlinks. For example, you might run with --action build-without-testing --no-clean-up to experiment after with ./mach run.

      • Unlike the other tests, HNT has the additional action variant of --action run, because it can be run from the terminal directly.

      • You can pass --hnt-test (eg: --hnt-test dom/notification) to pass a set of tests to use if the action argument is run-tests. Otherwise, ./mach test --auto will be used, which takes a guess at which tests would be best to run. You can pass --hnt-test to the build_against_all.py script as well.

All test scripts also accept the --verbose argument to show the output of run subprocesses (such as ./mach build).

Limitations

Note that these tests are primarily smoke tests against the building and compilation of application-services. There are a wide array of possible regressions that can only be caught with tests, including ones that crash the build immediately on running. To ensure any regressions for your component are caught, tests should be created for them rather than just building.

Deprecated tests

Android Components

The automation/smoke-test-android-components.py script will clone (or use a local version) of android-components and run a subset of its tests against the current application-services worktree. It tries to only run tests that might be relevant to application-services functionality.

Fenix

The automation/smoke-test-fenix.py script will clone (or use a local version) of Fenix and run tests against the current application-services worktree.

Firefox iOS

The automation/smoke-test-fxios.py script will clone (or use a local version) of Firefox iOS and run tests against the current application-services worktree.