When the goal is to test the performance of an application while running hundreds of mobile devices and a single user’s operation can interact with dozens of microservices, testing is not trivial. This post is about what made us decide to build a farm of mobile device emulators using a public cloud, mobile automation and continuous integration platform as well and how it all works.

Base is growing. We are growing in sheer number of customers and on top of that we have a new tier of clients – enterprises. With that in mind, many things need to be taken care of, both system- and performance-wise. One of those is mobile application functional and performance testing while the other is the number of users in each customers’ account and how this will impact overall performance of our end to end solution.

But first things first, let’s start with system architecture. Base is built as a large collection of microservices. Both web client as well as mobile clients use great many of those. In our case the goal is to test the behavior of an application while running hundreds or possibly thousands of mobile devices emulating real users’ actions on a single enterprise account. A single operation on a mobile client can interact with a dozen microservices that then interact with the next set of services. How can we reliably test this?

Our approach was to build a farm of mobile device emulators using a public cloud, mobile automation and continuous integration platform.

This brings us to this blogpost, in which we’ll share lessons we learned while building an infrastructure needed to run performance tests.

Technical background aka how it works

We use the following building blocks in order to obtain our goal:

blog1
  • Appium – framework intended for mobile automation on both Android and iOS is what we use to drive the logic of our tests. We use Rspec and Page Object to make our lifes easier while doing that. Below you can find an example of a performance test scenario.

PRO TIP: One thing worth mentioning here is that in this particular case our tests are meant to only simulate end-users’ actions and not test a mobile application directly. The subject of the test is the backend. Therefore no assertions are made at any step of the test and general principle is to allow tests to run continuously for a given amount of time.

require 'spec_helper' 
  
RSpec::Steps.steps "Create a new Lead - from map" do 
  
  before(:all) do 
    on(LoginPage).login_with(Mobile.config.email, Mobile.config.password) 
  end 
   
  it "Navigate to Leads" do 
    on(MenuPage).navigate_to "Leads" 
  end 
  
  it "Query to glonass from the map in a context of a Lead" do 
    on(LeadsPage).open_map 
    random_address = get_real_geolocation_with_address 
    valid_location_gps = random_address[:gps] 
    set_location(valid_location_gps[0], valid_location_gps[1]) #latitude/longitude 
    on(LeadsPage).zoom_my_location 
    on(LeadsMapPage).add_lead_from_map 
  end  
  
  it "Create a new Lead" do 
    on(LeadsMapPage).create_lead 
    on(AddEditLeadPage).populate_page_basic 
    on(AddEditLeadPage).save 
    on(LeadDetailsPage).go_back_to_leads 
  end 
end
  • Jenkins CI – this is where the whole flow is managed. The high-level overview on our setup is that each virtual user is performing a simulated action as one Jenkins job. To create that many jobs we are using a Jenkins API client. To run all jobs (and therefore users) together there is a special job that aggregates them using Groovy script.

The diagram below illustrates the whole process from Jenkins perspective.

blog3

When all pieces are set up correctly, running a test is as easy as setting how long and how many values.

blog4
  • Digital Ocean – out of many public cloud providers we picked them. The main reason was that they support KVM virtualization that is needed to run an Android Emulator. After a few months we can really give a thumbs up to Digital Ocean. We’ve had no issues whatsoever on their end and though the user interface might not be as versatile as that of AWS, the essential capabilities are there. If you need an alternative to GUI, there is also an API at your disposal, for example a ruby implementation called droplet_kit.
  • Emulator – this one is tricky. In theory we would like to have an Android and iOS devices, since our customers use both. But first of all, if the application under test will work exactly the same if used by either, then it does not make much business sense to spend time building both. Second, more important issue is how to emulate multiple iOS devices. With Android it is fairly easy. You need a machine (Linux/Windows/MacOS) or a virtual machine capable of running an android emulator and you are golden. With iOS it is not that obvious. For us iOS at this moment is an open topic. As an emulator we are using an Android Studio 2. This is due to newly introduced changes in emulator performance as well as better support for location emulator compared to Geny Motion.

PRO TIP: By trial and error the command that we use to start an instance of an emulator is:

-gpu on -nojni -no-boot-anim -netdelay none -netspeed full -qemu -m 1536 -enable-kvm
  • Jenkins Digital Ocean plug-in – we need to connect our Jenkins instance to Digital Ocean. The setup is quite simple. After the plug-in is added to Jenkins instance, an Auth Token that can be obtained from Digital Ocean account settings needs to be provided. Next, a few customizations of droplets need to be done, like providing image name, size and region. We have tweaked the plug-in a bit so that we can request X number of machines in advance, so we can reduce the variances in startup time of Digital Ocean droplets.

PRO TIP: There is a rate limit on Digital Ocean API. Before making tweaks in plug-in code we reached this limit quite fast. Making small change in the implementation of the plug-in solved this.

Open issues and next steps

As with all technical processes this is an evolution. We see a number of possible improvements as well as challenges that current solution poses to us.

Our primary focus within next months will be on making test runs more stable. Running an Android emulator with Appium tests for hours seems to be too much for the emulator and it will randomly crash. There might be a number of possible reasons and these need to be dissected one by one.

The second topic is adding iOS emulators to the mix. This might not be the first requirement, but in the long run we see that being able to emulate both will be quite useful.

Benefits

There are multiple benefits of this approach, the most important are:

  • Real app testing with no mocking or fake services and no responses or any other system components being mocked or stubbed.
  • Any change in the backend can be easily implemented into testing suite by simply using the latest version of an app.
  • The nature of communication with mobile devices is asynchronous, this can be fully supported by real devices doing real work by both sending and receiving data.
  • The solution scales well, if we need 500 devices we spawn 500 emulators. We can easily adjust  data throughput, number of interactions and number of active devices to simulate various conditions.

Authors: Michał Węgrzyn and Tomasz Nowak

Posted by

Michał Węgrzyn

Share this article

  • Michał Cichoń

    Did you consider using a farm of real devices for performance testing? If so, why did you choose emulators?

    • Michał Węgrzyn

      There are few companies offering real devices, but for us this was not an option cost-wise. For example Amazon charges $0.17/minute. So 0,17 * 500 devices * 8h = $680 per single session. Compare this with with DO cost: $0.06/hour!