share

ENGINEERING

5min read

Keep Espresso Tests Stable with Android Test Orchestrator

Introduction

If you have ever executed Espresso tests in your Android Project, I bet you had a lot flakiness and shared states in separated tests. In spite of using the tools that make tests more stable (read more about it in one of our previous posts by Tomasz), you can still experience some kind of instability.

Fortunately, Google has responded to the problem. Since the Android Testing Support Library 1.0 is here, we’ve got a perfect solutions to all those problems—Android Test Orchestrator.

Android Test Orchestrator is a gradle test option that helps in testing thanks to the supporting packages available on the maven repository. Applying Android Test Orchestrator from scratch in our projects increased the reliability of our automated test suites.

In this article, I would like to show you an example of how this new Android tool can be used for Gradle builds.

Android Studio and Gradle dependencies

The support for Android Test Orchestrator arrived with Android Studio 3.0 beta build, so for now, it is needed to use IDE from that release channel.

That’s not all. If you use Gradle build tools in any version below 3.0 you also have to update dependency setup for your project:

classpath 'com.android.tools.build:gradle:3.0.0-beta7'

In order to configure latest Espresso (3.0.2) with Android Test Orchestrator put these dependencies in your app/build.gradle file:

testImplementation 'junit:junit:4.13'
androidTestImplementation 'com.android.support.test:runner:1.0.2'
androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
androidTestUtil 'com.android.support.test:orchestrator:1.0.2

With Android Test Support Library 1.0 comes the androidTestImplementation keyword which resolve dependency apk file and install via adb.

Solution #1 Running test with Android Test Orchestrator from command line

Based on Google’s documentation, enabling Android Test Orchestrator for older gradle build tools versions (not supporting androidTestUtil) involves a different approach. In order to do it, one needs to call install steps for Orchestrator and Test-services apk files and execute shell commands with specified test runner. You can put those files in the directory inside project repository (it is about 1 MB) and trigger each test execution with the set of commands:

adb install ./apk_directory/orchestrator-1.0.2.apk
adb install ./apk_directory/test-services-1.0.2.apk
adb shell 'CLASSPATH=$(pm path android.support.test.services) app_process / \
  android.support.test.services.shellexecutor.ShellMain am instrument -w -e -v \
  targetInstrumentation YOUR_PACKAGE_NAME/YOUR_TEST_RUNNER_CLASS \
  android.support.test.orchestrator/.AndroidTestOrchestrator'

The thing that can be regarded as inconvenient in this scenario is that you can’t use Fastlane for tests execution, therefore you still face some problems with sharing the results in a smart way. In order to get some statistics from the execution, you will need to flush the output of the third command and get the results in a programmatic manner.

Solution #2 Based on Build.gradle

The recommended setup claims to add a single line to the testOptions section inside the build.gradle module related file:

 testOptions {
    execution 'ANDROID_TEST_ORCHESTRATOR'
  }

That is really simple. If you don’t want to let Android Test Orchestrator to be on in each test run you can overcome this by passing the optional argument for gradle task e.g.:

./gradlew connectedAndroidTest -Porchestrator=true

You can specify the project variables inside the build.gradle project related file:

project.ext {
    orchestrator = project.hasProperty('orchestrator') ? project.property('orchestrator') : false
}

And then determine testOptions on orchestrator variable value:

 testOptions {
        if (orchestrator == true) {
            println("With android test orchestrator")
            execution "ANDROID_TEST_ORCHESTRATOR"
        }
 }

Thanks to this setup instruction you will get Android Test Orchestrator disabled. Also, you will be able to run configurations without orchestrator=true argument passed for Android Studio. Android Test Orchestrator will be enabled for all executions with that argument passed — using command line or just “Run” button in Android Studio. After this you will get an output from all required steps attached to the mechanism usage:

Testing started at 14:00 ...

09/19 14:00:02: Launching MainActivityTest
$ adb push .../app/build/outputs/apk/debug/app-debug.apk /data/local/tmp/YOUR_PACKAGE_NAME
$ adb shell pm install -t -r "/data/local/tmp/YOUR_PACKAGE_NAME"
Success


$ adb push .../app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk /data/local/tmp/YOUR_PACKAGE_NAME.test
$ adb shell pm install -t -r "/data/local/tmp/YOUR_PACKAGE_NAME.test"
Success


$ adb push .../orchestrator-1.0.2.apk /data/local/tmp/android.support.test.orchestrator
$ adb shell pm install -t -r "/data/local/tmp/android.support.test.orchestrator"
Success


$ adb push .../test-services-1.0.1.apk /data/local/tmp/android.support.test.services
$ adb shell pm install -t -r "/data/local/tmp/android.support.test.services"
Success


Running tests

$ adb shell CLASSPATH=$(pm path android.support.test.services) app_process / android.support.test.services.shellexecutor.ShellMain am instrument -r -w -e targetInstrumentation YOUR_PACKAGE_NAME.test/YOUR_PACKAGE_NAME.BaseAndroidJUnitRunner   -e debug false -e class YOUR_PACKAGE_NAME.MainActivityTest android.support.test.orchestrator/android.support.test.orchestrator.AndroidTestOrchestrator

Client not ready yet..
Started running tests
Tests ran to completion.

Fastlane integration for running suites with Android Test Orchestrator

Thanks to Gradle plugin for Fastlane we can enjoy the simplicity of lanes also as far as Android builds are concerned. We can use project variable defined in the previous scope and execute Gradle task with the following argument passed:

desc "Runs orchestrator tests"
   lane :orchestrator do
       gradle(task: "connectedAndroidTest", properties: {
         "orchestrator" => true
       })

Conclusion

The malformed test sessions had been pain in the neck to everyone who wrote them until now. With a great tool from Google we can forget about all of the shared states problems forever. I hope you’ve enjoyed my tutorial and won’t have any issues with Android Test Orchestrator in your project!

share


AdamTest Engineer

LEARN MORE

Contact us if you have any questions regarding the article or just want to chat about technology, our services, job offers and more!

POLIDEA NEWSLETTER

Sign in and expect sharp insights, recommendations, ebooks and fascinating project stories delivered to your inbox

The controller of the personal data that you are about to provide in the above form will be Polidea sp. z o.o. with its registered office in Warsaw at ul. Przeskok 2, 00-032 Warsaw, KRS number: 0000330954, tel.: [0048795536436], email: [hello@polidea.com] (“Polidea”). We will process your personal data based on our legitimate interest and/or your consent. Providing your personal data is not obligatory, but necessary for Polidea to respond to you in relation to your question and/or request. If you gave us consent to call you on the telephone, you may revoke the consent at any time by contacting Polidea via telephone or email. You can find detailed information about the processing of your personal data in relation to the above contact form, including your rights relating to the processing, HERE.

Data controller:

The controller of your personal data is Polidea sp. z o.o. with its registered office in Warsaw at ul. Przeskok 2, 00-032 Warsaw, KRS number: 0000330954, tel.: [0048795536436], email: [hello@polidea.com] (“Polidea”)

Purpose and legal bases for processing:

 

Used abbreviations:

GDPR – Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the free movement
of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

ARES – Polish Act on Rendering Electronic Services dated 18 July 2002

TL – Polish Telecommunications Law dated 16 July 2004

1)        sending to the given email address a newsletter including information on Polidea’s new projects, products, services, organised events and/or general insights from the mobile app business world |art. 6.1 a) GDPR, art. 10.2 ARES and art. 172.1 TL (upon your consent)

Personal data:name, email address

2)       statistical, analytical and reporting purposes |art. 6. 1 f) GDPR (based on legitimate interests pursued by Polidea, consisting in analysing the way our services are used and adjusting them to our clients’ needs, as well as developing new services)

Personal data:name, email address

Withdrawal of consent:

You may withdraw your consent to process your personal data at any time.

Withdrawal of the consent is possible solely in the scope of processing performed based on the consent. Polidea is authorised to process your personal data after you withdraw your consent if it has another legal basis for the processing, for the purposes covered by that legal basis.

Categories of recipients:

Your personal data may be shared with:

1)       authorised employees and/or contractors of Polidea

2)       persons or entities providing particular services to Polidea (accounting, legal, IT, marketing and advertising services) – in the scope required for those persons or entities to provide those services to Polidea

 

Retention period:

1)       For the purpose of sending newsletter to the given email address – for as long as the relevant consent is not withdrawn

2)       For statistical, analytical and reporting purposes – for as long as the relevant consent is not withdrawn

Your rights:

 

Used abbreviation:

GDPR – Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the free movement
of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

According to GDPR, you have the following rights relating to the processing of your personal data, exercised by contacting Polidea via [e-mail, phone].

1)       to access to your personal data (art. 15 GDPR) by requesting sharing and/or sending a copy of all your personal data processed by Polidea

2)       to request rectification of inaccurate personal data
(art. 16 GDPR) by indicating the data requiring rectification

3)       to request erasure of your persona data (art. 17 GDPR); Polidea has the rights to refuse erasing the personal data in specific circumstances provided by law

4)       to request restriction of processing of your personal data (art. 18 GDPR) by indicating the data which should be restricted

5)       to move your personal data (art. 20 GDPR) by requesting preparation and transfer by Polidea of the personal data that you provided to Polidea to you or another controller in a structured, commonly used machine-readable format

6)       to object to processing your personal data conducted based on art. 6.1 e) or f) GDPR, on grounds relating to your particular situation (art. 21 GDPR)

7)       to lodge a complaint with a supervisory authority,
in particular in the EU member state of your habitual residence, place of work or place of the alleged infringement if you consider that the processing
of personal data relating to you infringes the GDPR
(art. 77.1 GDPR)

No obligation to provide data:

Providing your personal data is not obligatory, but necessary for Polidea to provide you the newsletter service

Refusal to provide the above data will result in inability to receive the newsletter service.

Profiling

In the process of providing the newsletter service, we make decisions in an automated way, including profiling, based on the data you provide.

 

“Profiling” means automated processing of personal data consisting of the use of your personal data to evaluate certain personal aspects relating to you, in particular to analyze or predict aspects concerning your personal preferences and interests.

 

The automated decisions are taken based on the analysis of clicked and viewed content. They affect the targeting of specific newsletter content to selected users registered to receive the newsletter service, based on the anticipated interests of the recipient.