Share

engineering

20min read

It’s Flutter Time—Introduction to UI Tests

It’s Flutter Time—Introduction to UI Tests

Flutter, the cross-platform app development tool, has become one of the most popular buzzwords in the world of mobile development. It is a real competitor for well-known frameworks like React Native or Xamarin.

Before Flutter, writing cross-platform apps was quite challenging for mobile developers—most of the frameworks grew on web development roots (languages, architectural concepts). Since Flutter’s release, native developers can write code in a pleasant way. It’s because of Dart which has many similarities to Kotlin and Swift languages. You can read more about Flutter in one of our recent blog posts written by our Android specialist—Michał.

Today, I will highlight the most important facts related to automated UI testing with Flutter.

Differences between Flutter and React Native

The main goal of React Native is to build JavaScript apps that work natively on Android and iOS. Thanks to the framework, you can write one code which looks like a native design for an actual platform. Below you can see an example of how it works:

text

For more information about the pros and cons of this technology, go here. If you have already decided to go with React Native—don’t hesitate and see my blogpost, related to testing this technology.

Flutter comes with different options for making cross-platform apps. To put it simply—you write one code, which looks the same everywhere. We can benefit from this feature thanks to the Flutter Engine, which is a portable runtime that hosts applications created with this framework. The engine implements the core of Flutter (animations, graphics, files, network I/O, and many more). This means that we have the same environment for running our apps on Android and iOS. In my opinion, it is a big plus because it reduces the effort when implementing platform-specific differences.

During the app development, we can use custom-designed elements or decide to use predefined style libraries that come with Flutter—Material (designed as Android) and Cupertino (designed as iOS). As a result, we can go with a native iOS or Android look in our views—no matter what phone brand a user has. Isn’t that impressive?

text

UI testing

Widget testing

All view elements are separated widgets that make Flutter very modular and testable. flutter_test library allows for rendering the whole application, without the need for running tests on a device or simulator. Moreover, it enables making typical operations, such as end-to-end tests, for example, tapping on elements, entering text, or making assertions.

It is possible to automate a scenario where a user goes to a search tab in the application and tries to find, let’s say, grapes. Example code (from the snippet below) should be placed in file test/widget_test.dart:

import 'package:veggieseasons/main.dart' as app;
void main(){
testWidgets('it is possible to find matching fruit from main app',
(WidgetTester tester) async {
await tester.pumpWidget(app.MainWidget());
await tester.tap(find.text("Search"));
await tester.pump();
await tester.enterText(find.byType(SearchBar), "Grapes");
await tester.pump();
expect(find.byKey(new Key("SearchItemName")),findsOneWidget);
expect(find.text("Couldn\'t have wine without them."),findsOneWidget);
});
}
view raw gistfile1.txt hosted with ❤ by GitHub

Thanks to focusing on testing single widget elements, it can render one of them separately. Below you can see an example of a test which renders a single view element with a kiwi image and description. Then it performs assertion on this widget text fields and checks if one title and one description were found.

Extremely fast and impressingly reliable.

void main(){
testWidgets('it is possible to validate small widget',
(WidgetTester tester) async {
Widget kiwi = ScopedModel<AppState>(
model: AppState(),
child: ScopedModel<Preferences>(
model: Preferences()..load(),
child: CupertinoApp(
color: Styles.appBackground,
home: VeggieCard(
new Veggie(
shortDescription: "Kiwi is sour.",
category: VeggieCategory.tropical,
accentColor: CupertinoColors.white,
seasons: <Season>[Season.winter],
id: 1,
name: "Kiwi",
imageAssetPath: 'assets/images/kiwi.jpg'),
true,
),
),
),
);
await tester.pumpWidget(kiwi);
expect(find.text("Kiwi"), findsOneWidget);
expect(find.text("Kiwi is sour."), findsOneWidget);
});
}
view raw gistfile1.txt hosted with ❤ by GitHub

Widget tests are triggered with the flutter test command. After a successful test execution, you should see an output like this:

00:12 +1: it is possible to find matching fruit from main app                                                                                                                                                                                                                                                                                               
00:12 +2: it is possible to validate small widget                                                                                                                                                      
00:12 +2: All tests passed! 

End-to-end tests

Flutter engineers also covered a need for writing automated end-to-end tests in a way commonly known from the native development. It means we can write an automated test which then can interact with a real application launched on a phone, with all device-related aspects (as WiFi or Bluetooth). We don’t have to worry about writing separated scripts for Android and iOS—one script is enough to test the same scenario on both platforms.

It’s all because, under the hood, it uses native automation tools for instrumentation—Espresso for Android and EarlGrey for iOS.

To start using this feature, we need to add the flutter driver library to the pubspec.yaml file and refresh all dependencies with the command flutter get.

dev_dependencies:
flutter_driver:
sdk: flutter
test: any
view raw gistfile1.txt hosted with ❤ by GitHub

Next, add the test_driver folder to the root of the project. It is required to place the app.dart file in this directory. Inside, you should place the enableFlutterDriverExtension() command to enable the Flutter driver library usage in the project.

import 'package:veggieseasons/main.dart' as app;
import 'package:flutter_driver/driver_extension.dart';
void main(){
enableFlutterDriverExtension();
app.main();
}
view raw gistfile1.txt hosted with ❤ by GitHub

Next, add the file with tests app_test.dart.

// Imports the Flutter Driver API
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';
void main() {
group('Veggie app', () {
FlutterDriver driver;
setUp(() async {
// in order to make app starting from the home screen each time
await driver.tap(find.text("Home"));
});
setUpAll(() async {
driver = await FlutterDriver.connect();
});
tearDownAll(() async {
if (driver != null) {
driver.close();
}
});
test('Apples are displayed', () async {
final cardTitle = find.byValueKey("cardTitleApples");
await ozzie.takeScreenshot("Apples");
await diff("Apples");
expect(await driver.getText(cardTitle),"Apples");
});
test('Plums are displayed', () async {
final cardTitle = find.byValueKey("cardTitlePlums");
await driver.scrollUntilVisible(find.byValueKey("list"), cardTitle, dyScroll: -500);
await ozzie.takeScreenshot("Plums");
await diff("Plums");
expect(await driver.getText(cardTitle),"Plums");
});
});
}
view raw gistfile1.txt hosted with ❤ by GitHub

In this particular case, our test will check if it is possible to find Apples and Plums using the scroll operation. In order to run those tests use the command flutter drive --target=test_driver/app.dart.

Improve tests with HTML reports and visual regression testing

Ozzie HTML reports

Ozzie is a tool that gives you the ability to create a HTML test report with attached screenshots from the test execution. To start using Ozzie in your project, just add it as a Flutter dependency:

dev_dependencies:
  ozzie: <latest_version_here>

Another step is to add some code lines to the test class:

  1. Ozzie library import line:
import 'package:flutter_driver/flutter_driver.dart';
  1. Ozzie object declaration:
Ozzie ozzie;
  1. Ozzie object initialization:
ozzie = Ozzie.initWith(driver, groupName: 'Veggie');
  1. HTML report generating command:
ozzie.generateHtmlReport(); 
  1. …and finally screenshot capture command:
await ozzie.takeScreenshot(“name of snap”);

Below you can see a snippet of test class with two tests. First of them checks whether the Apples fruit card is displayed on the first screen. In the second one, the Flutter driver tries to find the Plums fruit card nested deeply in listview using scroll operation. Ozzie will capture screenshots for both.

// Imports the Flutter Driver API
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';
import 'package:ozzie/ozzie.dart'; // Ozzie dependency
void main() {
group('Veggie app', () {
FlutterDriver driver;
Ozzie ozzie; //Ozzie object
setUp(() async {
await driver.tap(find.text("Home"));
});
setUpAll(() async {
driver = await FlutterDriver.connect();
ozzie = Ozzie.initWith(driver, groupName: 'Veggie'); // Initialize the Ozzie instance
});
tearDownAll(() async {
if (driver != null) {
driver.close();
ozzie.generateHtmlReport(); //Finally generate html report
}
});
test('Apples are displayed', () async {
final cardTitle = find.byValueKey("cardTitleApples");
await ozzie.takeScreenshot("Apples");
expect(await driver.getText(cardTitle),"Apples");
});
test('Plums are displayed', () async {
final cardTitle = find.byValueKey("cardTitlePlums");
await driver.scrollUntilVisible(find.byValueKey("list"), cardTitle, dyScroll: -500);
await ozzie.takeScreenshot("Plums");
expect(await driver.getText(cardTitle),"Plums");
});
});
}
view raw gistfile1.txt hosted with ❤ by GitHub

It is crucial to write tests that are resistant to languages or content changes. In my tests, I decided to localize elements using key values. Thanks to that, each fruit element can be accessed with stable and constant key locator which can be generated programmatically in code. You can extend your widget definition with this attribute as shown in this example:

<Widget>[
Text(
veggie.name,
style: Styles.cardTitleText,
key: Key('cardTitle'+veggie.name)
),
Text(
veggie.shortDescription,
style: Styles.cardDescriptionText,
key:Key('cardDescription'+veggie.name)
),
]
view raw gistfile1.txt hosted with ❤ by GitHub

After conducting the tests above, we get the HTML report that contains screenshots from these tests. However, we can do something more interesting—add the automated visual regression check.

Snapshot testing

Before you start, capture some reference images using Ozzie. In my case, I copied previously saved images from the test execution and placed them in the reference_snaps/Veggie directory.

I tried to look for some tools that could be used as an automatic pixel-perfect comparator but did not find anything for Flutter. Based on my experience with React Native, I decided to use the blink-diff tool.

To setup this tool, download it with the attached npm command and place it in the project directory.

npm install blink-diff

The code that I’m presenting next, expects to have the blink-diff location in the node_modules folder.

'node_modules/blink-diff/bin/blink-diff'

Then, add in the test class a diff method that we can use for triggering the blink-diff visual regression check. It is good to prepare the util class—coming from the test logic—and place all the methods there, which can be used around many test classes.

On the snippet attached below you can see how I implemented this visual check:

import 'dart:io';
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';
import 'package:ozzie/ozzie.dart';
import 'package:path/path.dart' as p;
Future diff(snap) async {
final diffsDir = new Directory('ozzie/Diffs');
var outputFile = new File(p.join(diffsDir.path, snap) + "-output.png");
final String blinkdiffPath = 'node_modules/blink-diff/bin/blink-diff';
await outputFile.create(recursive: true);
await Process.run(blinkdiffPath, [
'--output',
outputFile.path,
p.join('ozzie/Veggie/', snap) + ".png",
p.join(
'reference_snaps/Veggie/',
snap,
) +
".png",
'--threshold',
'10'
]).then((ProcessResult results) {
if (results.stdout.toString().contains("FAIL") == false) {
outputFile.delete();
}
expect(results.stdout.toString().contains("FAIL"), isFalse);
});
}
void main(){
group('Veggie', () {
FlutterDriver driver;
Ozzie ozzie;
setUp(() async {
await driver.tap(find.text("Home"));
});
setUpAll(() async {
driver = await FlutterDriver.connect();
ozzie = await Ozzie.initWith(driver, groupName: "Veggie");
});
tearDownAll(() async {
if (driver != null){
await driver.close();
}
await ozzie.generateHtmlReport();
});
test('Apples are displayed', () async {
final cardTitle = find.byValueKey("cardTitleApples");
await ozzie.takeScreenshot("Apples");
await diff("Apples");
expect(await driver.getText(cardTitle),"Apples");
});
test('Plums are displayed', () async {
final cardTitle = find.byValueKey("cardTitlePlums");
await driver.scrollUntilVisible(find.byValueKey("list"), cardTitle, dyScroll: -500);
await ozzie.takeScreenshot("Plums");
await diff("Plums");
expect(await driver.getText(cardTitle),"Plums");
});
});
}
view raw gistfile1.txt hosted with ❤ by GitHub

Finally, we are all done. Now, after the test execution, we will get our nice HTML report in the Ozzie directory. In this report, you can see three images for each test—an actual screenshot from our test, a reference screenshot, and a their comparison placed in the middle. All differences between these captures will be marked with a red highlight, as you can see in the report attached below:

text

Continuous Integration

Here’s an example that shows you some basic steps which you should follow in order to run Flutter tests on CI. My snippet describes the configuration of Gitlab CI—this code should be placed in .gitlab-ci.yaml in the root folder of the project.

# Flutter tests
"test:UI:Gitlab":
script:
- /bin/bash $CACHE_DIR/bundlerScript.bash
- mkdir development && wget -O development/flutter.zip https://storage.googleapis.com/flutter_infra/releases/stable/macos/flutter_macos_v1.2.1-stable.zip
- cd development
- unzip flutter.zip
- export PATH="$PATH:`pwd`/flutter/bin"
- cd ..
- open -a Simulator
- cd ios
- pod setup
- pod install
- cd ..
- flutter packages get
- flutter test
- flutter drive test_driver/*
stage: test
artifacts:
paths:
- ozzie
when: always
expire_in: 1 week
tags:
- xcode
view raw gistfile1.txt hosted with ❤ by GitHub

Notice:

  • This code is not optimized. All things related to downloading resources should be improved by the proper Gitlab CI runner setup. I just highlighted the required steps.
  • Remember to open the required simulator (or the Android emulator) first, before you launch the tests, otherwise it will fail.
  • After the tests execution, Ozzie deliverables would be stored for 1 week (this value is configurable).

Conclusion

Google has surprised us a lot with an excellent quality in Flutter: testing libraries are available right out of the gate. It’s worth highlighting that Espresso, the officially recommended framework for testing UI in Android apps, was launched years after the release of Android.

I showed you that it is possible to create a UI testing framework in Flutter, which allows for the extensive application testing. With Flutter libraries writing unit, integration and UI tests is very simple. Configuration of the development environment is equally straightforward. We can easily move our project into the CI infrastructure.

Of course, we can find many limitations and disadvantages in comparison to the cross-platform frameworks like Appium and native frameworks. For example, the lack of the real device farms support, not being able to manipulate device state, or mock HTTP in end-to-end tests (because of the UI tests running out of the main app process). Nevertheless, Flutter seems to fulfill most needs in terms of the automated testing in mobile apps.

Fingers crossed for Flutter! If you have any questions about the framework—get in touch. Don’t forget to subscribe to our newsletter to get your monthly dose of tech news and articles!

Share

Adam

Senior Test Engineer

Did you enjoy the read?

If you have any questions, don’t hesitate to ask!

Did you enjoy the read?

If you have any questions, don’t hesitate to ask!