This is the first part of a four-part blog post series describing how we in Base have built a modern test automation and cloud testing solution. The first part is a high-level overview of a problem that we have faced and a solution to it. The second part is going to be a deep dive into code examples and technologies used to build Android automation. The third is going to describe how our Continuous Integration platform works. Forth will be about iOS implementation of test automation.

Things you need to know

  • If, as a test automation engineer or mobile developer, you have ever tried to automate your mobile tests, you know the true meaning of Star Trek quote “resistance is futile”. Whatever you would or would not do, tests were difficult to write, hard to maintain and unstable during execution. Cloud testing was closer to sci-fi than reality.
  • A new breed of native frameworks, Espresso and EarlGrey provide us with the tools we have all been waiting for.
  • Continuous integration (CI) using Jenkins and Firebase is an absolute killer feature.
  • Years of experience with building similar solutions with other frameworks for various technologies provide us with a long list of lessons learned and well-tested patterns.

Chapter I – Before

“The past is already gone, the future is not yet here. There’s only one moment for you to live, and that is the present moment” – Buddha

Appium and Calabash failed us. Let’s be blunt about it. We all had high hopes, invested our time and resources. This was the best tool available at that time and we did our best to make it work. But it did not, not the way it was expected to work.

Cross-platform testing frameworks idea was great in theory, but a real-life application proved to be more than suboptimal. There is no point of listing out all the cons, so let’s just focus on the one thing that those tools gave us. They gave us invaluable experience and provided with perspective.

Chapter II – Now

“Do, or do not. There is no try” – Yoda

A new era of mobile automation gave us Espresso, EarlGrey and Firebase (with Flank). And all three are game changers. How do we know? Because we have tried and succeeded. Well, we are halfway there at least, our Android setup is complete, iOS is under development.

A picture is worth a thousand words, so let’s take a look at high-level solution overview.

Testing strategy:

  • Full E2E scenarios – we have a lot of unit tests, but they do not find application crashes specific to OS version or device type, nor do they find that a button is missing in a given screen. We could do integration tests, but mocking parts of a mobile application, especially our application that heavily relies on backend, looked grim. We would not be testing what we wanted to test in the first place. Our choice was to go with full end-to-end user scenarios. Let automation execute multiple simple tests that perform basic scenarios from the UI level, check if all icons are where they supposed to be, if an application does not crash and if objects are being created/updated/deleted as expected.
    An obvious drawback of this approach is execution time, this, however is mitigated by our CI setup, we simply run tests in parallel and focus on tracking longest execution times. We strive to be able to execute the whole suite in under 20 minutes.
  • Smoke test first policy – I am a firm follower of Rapid Software Testing school of thought. Our tests are not really testing an application, they are checking if nothing blows up. Testing is a human only activity, but a machine can do an amazing job with running automated checks. With each new build, there is a risk that a change done in one place, might crash an app somewhere else. It is fairly easy to write a test doing simple regression scenarios, like login, do this, do that, check if it was done, log out. Testing stuff like that manually is tedious and takes tester’s time away from where he or she might really have an impact – which is exploratory testing where human mind is required.

Testing frameworks:

  • Espresso – developed by Google for Google. What else can be said, this is a native tool built for Android, not a wrapper to a wrapper to Selenium. It “just” works.
  • EarlGrey – developed by Google for iOS platform. As I type it we are in a proof of concept phase, so I can tell you as much as “it seems to works as well as Espresso”.

Continuous integration (CI):

  • Jenkins – this is our CI of choice. While being open source and easy to customize Jenkins became our tool for frontend and backend testing. We, as a company, have a dedicated team called Quality Platform, that takes care of Jenkins and tweaks it to needs of different teams. There could be a separate blog post about it since the topic is so broad, but let me just mention that our Jenkins runs on Kubernetes that by itself makes it amazing. We use Jenkins as a place where you can run tests, check execution status, find logs and links to execution details. Mobile tests themselves are executed using Firebase. Jenkins dashboardMoreover our CI is integrated with GitHub. Tests can be triggered (using our Triggear, that integrates Jenkins and GitHub) by PR request and their status is visible in GitHub. Jenkins jobs on Github
  • Firebase – I was really surprised when it turned out that Espresso works better than expected. But a real star of this whole setup is Firebase. It gives us Test Lab that is simply amazing. The setup will be described in next parts of this blog post. Our tests can be executed on emulators or physical devices. The list of devices is long, it covers multiple vendors and OS versions. Each execution is monitored and video logs are provided. Moreover, the performance of an application at each given moment is also tracked. Firebase rules.Firebase Test Lab at Base
  • Grafana – there is an old saying “if you cannot measure it, you cannot improve it”. Well, we measure things. Details about Jenkins job executions, number of successes and failures, execution time are all available as simple dashboards. Below a screenshot of one of those dashboards.Grafana - Android test stats

Code:

  • Page Object Pattern – it stood the test of time and proved to be an industry standard. In a mobile application context, using this pattern makes a lot of sense. Screens have a many common parts, that can be inherited or shared between abstract pages. Usually, there are fewer available options compared to web, so pages are smaller and easier to maintain. What surprised me, and might also surprise you if you are a test engineer, is that developers do not know about this pattern. They will learn it in a heartbeat though.
  • Tasks Pattern – the problem with page object pattern is that it generated a lot of repetitive code in test classes. We use tasks pattern to mitigate this. Below a super short example, that shows how it works.
    package com.futuresimple.base.ui.lead
    
    import ...
    
    class LeadFromGlobalTest {
    
      @Rule
      @JvmField
      val rule = BaseEspressoTestRule(LEAD_FROM_GLOBAL)
    
      @Test
      fun leadFromGlobalTest() {
    
        val firstName = Faker.getFirstName()
        val lastName = Faker.getLastName()
        val user = rule.testAccount.users[0]
    
        loginAndSkipFux(user.email, user.password)
    
        TodayView.goToGlobalAdd(LEAD)
        createSimpleLead(firstName, lastName)
        checkItemIsFirstOnTodayActivity("${user.name} created $firstName $lastName")
        searchAndOpen("$firstName $lastName")
        LeadView.shouldBeVisible()
        clearSearchAndCheckRecentlyVisited("$firstName $lastName")
        filterLeadsListByName(firstName, "$firstName $lastName")
    
        logout()
      }
    }
    

    loginAndSkipFux is a task. Underneath it, we use page objects.

    fun loginAndSkipFux(email: String, password: String) {
      WelcomeView.goToLogin()
      LoginView.login(email, password)
      OnboardingFux.skipFux()
    }
    
  • Kotlin and Swift – both platforms provide us with a new generation of programming languages, relatively easy to learn and with a focus on minimizing a boilerplate code.

How we work:

  • Test code in the same repository as application code – the uniqueness of this solution is that both Espresso and EarlGrey must be kept in the same code repository as production code. This may be new to some, but the test code quality must be kept with the same quality as application code. Detailed Pull Request checks and reviews are needed and this is something that we have introduced as a part of our workflow.
  • Division of responsibility – with each project as complex as this, clear responsibility division helps a lot. For us, we drew the line as presented below. CI is a realm where Quality Platform Team reigns supreme. Developers are responsible for page objects and all auxiliary tooling like API or platform-specific features. Testers build test cases and tasks as well as create final tests using existing building blocks.Division of responsibility - Dev | QA

What’s next

In the next post I’ll do a deeper dive into code examples and technologies we used to build Android automation. Stay tuned.

Posted by

Michał Węgrzyn

Share this article