Automated UI Testing with Gitlab CI
If you use automated UI tests in your iOS projects, you surely want to pass execution trigger effort to someone else. I know it from my own experience—I had plenty of scripts written with KIF framework but I had no idea how to trigger those tests in CI system.
After configuring a project on Gitlab Continuous Integration system using Fastlane tool, I’ve come up with some observations and ideas that may be useful for testers and developers.
In this blogpost, I would like to introduce the short guide to adapting Fastlane configuration and expanding Gitlab CI job description to get scheduled test sessions with shareable results. All done in a quick and simple way.
Project scheme adjustments
In order to run UI tests for iOS devices based on KIF iOS integration test framework, we need to adjust the scope of the test and define some crucial project properties in native iOS IDE—Xcode.
- Open Xcode project and navigate to Product -> Scheme -> New Scheme
- In the new window select Target and type Name for the new scheme created, e.g. ProjectKIF
- Select new scheme and go to Product -> Scheme -> Edit Scheme next
- In the Run section, open Arguments section and specify Environment variable that contains path to a directory where KIF Framework stores screenshots
KIF_SCREENSHOTS = $PROJECT_DIR/failed_screenshots/
- Switch to the Build section and enable Test checkbox for your new created target (scheme)
- Enable tests you want to execute for this scheme, using Test Navigator and… you’re done!
Make sure that
failed_screenshots directory is accessible from the build folder. You have to add this folder to
Slack integrations setup
For the typical agile development process, it is not needed to generate exhausting reports of each regression test suite execution. Automated test plan guarantees that the application works as tests describe but does not verify / validate if the application is good at all. As far as this task is concerned, I would recommend minimal effort. Gitlab Slack integration offers very simple integration via WebHooks, which allows to share data from external sources into Slack.
- Open Slack web interface and go to Apps & Integrations
- Go to Manage -> Custom integrations -> Incoming WebHooks
- Add new integration and define channel for notifications, name, optional image
- Keep the generated WebHook URL for further usage
Gitlab CI is limited to sharing artifacts only for the passed builds. That gives some limitations and requires more custom actions. To share artifacts for the failed builds (some tests failed and build was marked as unsuccessful), it is needed to make a workaround.
- Go to the Manage -> Custom integrations -> Bots
- Add new bot user, fill the description and keep API token
- In PROJECT_DIR create a scripts directory with a file publish_report.sh
- Open this file with your favorite script editor (surely it is vim)
Create the bash script, which detects if the directory with screenshots is empty and then sends a zip file to the slack channel with html test execution report. In other case, it attaches html report with screenshots for the failed tests.
#! /bin/bash if [ $(ls -l ./failed_screenshots/ | grep -v ^d | wc -l) == 0 ] then zip -r ./Only_report.zip ./fastlane/test_output/ && curl -F file=@./Only_report.zip -F channels=#ios-feed -F token=API_TOKEN https://slack.com/api/files.upload else zip -r ./Report_and_screenshots_for_failures.zip ./fastlane/test_output/ ./failed_screenshots && curl -F file=@./Report_and_screenshots_for_failures.zip -F channels=#ios-feed -F token=API_TOKEN https://slack.com/api/files.upload fi
Fastlane usage for testing purposes
Fastlane, being the most popular iOS and Android deployment tool, is used for running tests and managing the results of build products. Fastlane reduces the effort of running test to just one simple command.
Open your Fastfile, which is usually placed in $PROJECT_DIR/fastlane
Define a new lane with target simulator definition, workspace, scheme and Slack message payload, as well as the and previously reached WebHook url
desc 'Runs KIF tests' lane :runKIFtests do ENV["SLACK_URL"] = "https://hooks.slack.com/services/..." scan( workspace: "project.xcworkspace", scheme: ‘projectKIF’, devices: ["iPhone 4s (9.2)"] ) slack( message: "KIF Tests”, default_payloads: [:lane, :test_result, :git_branch, :last_git_commit] ) end
Gitlab CI job description using YAML
Yaml file is a kind of description of Gitlab Continuous Integration tool. All commands needed for build are stored in this file. This part describes how to trigger Fastlane lanes and define test jobs (command for CI) separated from other jobs that currently exist in the project.
Open your .gitlab-ci.yml file mainly placed in $PROJECT_DIR
Add new job definition
test_KIF: stage: test artifacts: when: always paths: - ./fastlane/test_output/report.html - ./failed_screenshots/ expire_in: 1 week script: - bundle exec fastlane runKIFtests after_script: - sh ./scripts/publish_report.sh only: - schedules
To avoid running automated tests locally and doing it manually, test job can be scheduled repeatedly.
Artifacts section contains paths to the files in the build directory and defines the period of time for which the finished artifacts should be held after the build finishes.
In order to run your lane during this job, it is required to execute fastlane command with the name of your newly created lane.
In order to enable your job only for scheduled jobs please pass the keyword schedules in only part.
As a result of this job, the test execution and fastlane slack notification happen and job artifacts are shared by Slack Bot.
Gitlab CI job scheduler
To avoid running automated tests locally and doing it manually, test job can be scheduled repeatedly. Running scheduled tests with Gitlab CI is very simple—one needs only simple cron rules for that.
- Open Gitlab CI web interface and open your project dashboard
- Go to Pipelines -> Schedules -> New schedules
- Define name for the schedule and cron rule. You can skip manual cron definition and use one of the predefined names e.g. Every day
0 4 * * *
- Save pipeline schedule
For the typical agile development process, it is not needed to generate exhausting reports of each regression test plan execution. Automated test plan guarantees that the application works as tests describe but does not verify / validate if the application is good at all. Therefore, minimal effort is recommended. Use the simplest recipes you can, like Gitlab automated testing, and keep calm with your app quality.
Senior Test Engineer