share

ENGINEERING

5min read

Automated UI Testing with Gitlab CI

Introduction

If you use automated UI tests in your iOS projects, you surely want to pass execution trigger effort to someone else. I know it from my own experience—I had plenty of scripts written with KIF framework but I had no idea how to trigger those tests in CI system.

After configuring a project on Gitlab Continuous Integration system using Fastlane tool, I’ve come up with some observations and ideas that may be useful for testers and developers.

In this blogpost, I would like to introduce the short guide to adapting Fastlane configuration and expanding Gitlab CI job description to get scheduled test sessions with shareable results. All done in a quick and simple way.

Project scheme adjustments

In order to run UI tests for iOS devices based on KIF iOS integration test framework, we need to adjust the scope of the test and define some crucial project properties in native iOS IDE—Xcode.

Let’s start:

  1. Open Xcode project and navigate to Product -> Scheme -> New Scheme
  2. In the new window select Target and type Name for the new scheme created, e.g. ProjectKIF
  3. Select new scheme and go to Product -> Scheme -> Edit Scheme next
  4. In the Run section, open Arguments section and specify Environment variable that contains path to a directory where KIF Framework stores screenshots
    KIF_SCREENSHOTS = $PROJECT_DIR/failed_screenshots/
  5. Switch to the Build section and enable Test checkbox for your new created target (scheme)
  6. Enable tests you want to execute for this scheme, using Test Navigator and… you’re done!

Make sure that failed_screenshots directory is accessible from the build folder. You have to add this folder to .gitignore file.

Slack integrations setup

For the typical agile development process, it is not needed to generate exhausting reports of each regression test suite execution. Automated test plan guarantees that the application works as tests describe but does not verify / validate if the application is good at all. As far as this task is concerned, I would recommend minimal effort. Slack offers very simple integration via WebHooks, which allows to share data from external sources into Slack.

  1. Open Slack web interface and go to Apps & Integrations
  2. Go to Manage -> Custom integrations -> Incoming WebHooks
  3. Add new integration and define channel for notifications, name, optional image
  4. Keep the generated WebHook URL for further usage

incoming_webHooks.png

Gitlab CI is limited to sharing artifacts only for the passed builds. That gives some limitations and requires more custom actions. To share artifacts for the failed builds (some tests failed and build was marked as unsuccessful), it is needed to make a workaround.

  1. Go to the Manage -> Custom integrations -> Bots
  2. Add new bot user, fill the description and keep API token
  3. In PROJECT_DIR create a scripts directory with a file publish_report.sh
  4. Open this file with your favorite script editor (surely it is vim)
  5. Create the bash script, which detects if the directory with screenshots is empty and then sends a zip file to the slack channel with html test execution report. In other case, it attaches html report with screenshots for the failed tests.

    #! /bin/bash
    if [ $(ls -l ./failed_screenshots/ | grep -v ^d | wc -l) == 0 ]
    then
      zip -r ./Only_report.zip ./fastlane/test_output/ && curl -F file=@./Only_report.zip -F channels=#ios-feed -F token=API_TOKEN https://slack.com/api/files.upload
    else
      zip -r ./Report_and_screenshots_for_failures.zip ./fastlane/test_output/ ./failed_screenshots && curl -F file=@./Report_and_screenshots_for_failures.zip -F channels=#ios-feed -F token=API_TOKEN https://slack.com/api/files.upload
    fi

tester.png

Fastlane usage for testing purposes

Fastlane, being the most popular iOS and Android deployment tool, is used for running tests and managing the results of build products. Fastlane reduces the effort of running test to just one simple command.

  1. Open your Fastfile, which is usually placed in $PROJECT_DIR/fastlane

  2. Define a new lane with target simulator definition, workspace, scheme and Slack message payload, as well as the and previously reached WebHook url

    desc 'Runs KIF tests'
    lane :runKIFtests do
        ENV["SLACK_URL"] = "https://hooks.slack.com/services/..."
        scan(
            workspace: "project.xcworkspace",
            scheme: ‘projectKIF’,
            devices: ["iPhone 4s (9.2)"]
        )
        slack(
            message: "KIF Tests,
            default_payloads: [:lane, :test_result, :git_branch, :last_git_commit]
        )
    end

Gitlab CI job description using YAML

Yaml file is a kind of description of Gitlab Continuous Integration tool. All commands needed for build are stored in this file. This part describes how to trigger Fastlane lanes and define test jobs (command for CI) separated from other jobs that currently exist in the project.

  1. Open your .gitlab-ci.yml file mainly placed in $PROJECT_DIR

  2. Add new job definition

    test_KIF:
       stage: test
       artifacts:
           when: always
           paths:
           - ./fastlane/test_output/report.html
           - ./failed_screenshots/
           expire_in: 1 week
       script:
           - bundle exec fastlane runKIFtests
       after_script:
           - sh ./scripts/publish_report.sh
       only:
           - schedules
  3. To avoid running automated tests locally and doing it manually, test job can be scheduled repeatedly.

Artifacts section contains paths to the files in the build directory and defines the period of time for which the finished artifacts should be held after the build finishes.

In order to run your lane during this job, it is required to execute fastlane command with the name of your newly created lane.

In order to enable your job only for scheduled jobs please pass the keyword schedules in only part.

As a result of this job, the test execution and fastlane slack notification happen and job artifacts are shared by Slack Bot.

Fastlane.png

Only_report.png

Gitlab CI job scheduler

To avoid running automated tests locally and doing it manually, test job can be scheduled repeatedly. Running scheduled tests with Gitlab CI is very simple—one needs only simple cron rules for that.

  1. Open Gitlab CI web interface and open your project dashboard
  2. Go to Pipelines -> Schedules -> New schedules
  3. Define name for the schedule and cron rule. You can skip manual cron definition and use one of the predefined names e.g. Every day 0 4 * * *
  4. Save pipeline schedule

Conclusion

For the typical agile development process, it is not needed to generate exhausting reports of each regression test plan execution. Automated test plan guarantees that the application works as tests describe but does not verify / validate if the application is good at all. Therefore, the minimal effort is recommended. Use the simplest recipes you can and keep calm with your app quality.

share


AdamTest Engineer

LEARN MORE

Contact us if you have any questions regarding the article or just want to chat about technology, our services, job offers and more!

POLIDEA NEWSLETTER

Sign in and expect sharp insights, recommendations, ebooks and fascinating project stories delivered to your inbox

The controller of the personal data that you are about to provide in the above form will be Polidea sp. z o.o. with its registered office in Warsaw at ul. Przeskok 2, 00-032 Warsaw, KRS number: 0000330954, tel.: 0048 795 536 436, email: hello@polidea.com (“Polidea”). We will process your personal data based on our legitimate interest and/or your consent. Providing your personal data is not obligatory, but necessary for Polidea to respond to you in relation to your question and/or request. If you gave us consent to call you on the telephone, you may revoke the consent at any time by contacting Polidea via telephone or email. You can find detailed information about the processing of your personal data in relation to the above contact form, including your rights relating to the processing, HERE.

Data controller:

The controller of your personal data is Polidea sp. z o.o. with its registered office in Warsaw at ul. Przeskok 2, 00-032 Warsaw, KRS number: 0000330954, tel.: [0048795536436], email: [hello@polidea.com] (“Polidea”)

Purpose and legal bases for processing:

 

Used abbreviations:

GDPR – Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the free movement
of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

ARES – Polish Act on Rendering Electronic Services dated 18 July 2002

TL – Polish Telecommunications Law dated 16 July 2004

1)        sending to the given email address a newsletter including information on Polidea’s new projects, products, services, organised events and/or general insights from the mobile app business world |art. 6.1 a) GDPR, art. 10.2 ARES and art. 172.1 TL (upon your consent)

Personal data:name, email address

2)       statistical, analytical and reporting purposes |art. 6. 1 f) GDPR (based on legitimate interests pursued by Polidea, consisting in analysing the way our services are used and adjusting them to our clients’ needs, as well as developing new services)

Personal data:name, email address

Withdrawal of consent:

You may withdraw your consent to process your personal data at any time.

Withdrawal of the consent is possible solely in the scope of processing performed based on the consent. Polidea is authorised to process your personal data after you withdraw your consent if it has another legal basis for the processing, for the purposes covered by that legal basis.

Categories of recipients:

Your personal data may be shared with:

1)       authorised employees and/or contractors of Polidea

2)       persons or entities providing particular services to Polidea (accounting, legal, IT, marketing and advertising services) – in the scope required for those persons or entities to provide those services to Polidea

 

Retention period:

1)       For the purpose of sending newsletter to the given email address – for as long as the relevant consent is not withdrawn

2)       For statistical, analytical and reporting purposes – for as long as the relevant consent is not withdrawn

Your rights:

 

Used abbreviation:

GDPR – Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the free movement
of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

According to GDPR, you have the following rights relating to the processing of your personal data, exercised by contacting Polidea via [e-mail, phone].

1)       to access to your personal data (art. 15 GDPR) by requesting sharing and/or sending a copy of all your personal data processed by Polidea

2)       to request rectification of inaccurate personal data
(art. 16 GDPR) by indicating the data requiring rectification

3)       to request erasure of your persona data (art. 17 GDPR); Polidea has the rights to refuse erasing the personal data in specific circumstances provided by law

4)       to request restriction of processing of your personal data (art. 18 GDPR) by indicating the data which should be restricted

5)       to move your personal data (art. 20 GDPR) by requesting preparation and transfer by Polidea of the personal data that you provided to Polidea to you or another controller in a structured, commonly used machine-readable format

6)       to object to processing your personal data conducted based on art. 6.1 e) or f) GDPR, on grounds relating to your particular situation (art. 21 GDPR)

7)       to lodge a complaint with a supervisory authority,
in particular in the EU member state of your habitual residence, place of work or place of the alleged infringement if you consider that the processing
of personal data relating to you infringes the GDPR
(art. 77.1 GDPR)

No obligation to provide data:

Providing your personal data is not obligatory, but necessary for Polidea to provide you the newsletter service

Refusal to provide the above data will result in inability to receive the newsletter service.

Profiling

In the process of providing the newsletter service, we make decisions in an automated way, including profiling, based on the data you provide.

 

“Profiling” means automated processing of personal data consisting of the use of your personal data to evaluate certain personal aspects relating to you, in particular to analyze or predict aspects concerning your personal preferences and interests.

 

The automated decisions are taken based on the analysis of clicked and viewed content. They affect the targeting of specific newsletter content to selected users registered to receive the newsletter service, based on the anticipated interests of the recipient.