Frequent testing of your software is a key to success. As mentioned in my previous article “Mobile CI/CD 101” it’s not easy to cover an entire mobile application with just regular unit tests. Most mobile applications depend on the user interface and platform-specific features that make true Test-Driven Design (TDD) with solely unit tests extremely challenging It’s better to split automatic testing in two parts: cover the Data Access Layer with unit tests and cover the Business Logic, User Interface, and platform features with UI tests. You can do both types of testing with your Visual Studio App Center CI/CD pipeline. This article will help you learn how to more easily test Xamarin.Forms applications with MSTest based unit tests and Xamarin.UITest based UI tests.

1. Unit Testing

As I've covered in previous posts, it is not possible to cover all the code in mobile applications, as well as UI applications in general, with tests based on console Unit Tests. Most of the features require interaction with the user interface or platform functionality. If we consider a multi-layer architecture, common in mobile projects, we can distinguish the Data Access Layer (DAL, accessing to external online services or local database) which can be covered 100 percent with unit tests from the rest of the application. Let's look at the typical application architecture using Xamarin.Forms - Figure 1.

Additionally, you can cover integration to external REST-services with automated tests. This step will let you quickly detect changes and problems with external APIs used in the application. To leverage this, you should launch unit tests on the “post-clone” step of your App Center build pipeline. Here is a sample of appcenter-post-clone.sh (launch MSTest based unit tests for .NET Standard DAL library ):

2. UI Testing

In order to begin using automated UI testing in your development process, let's define the types of problems that can be identified:

• crashes of the application on all or some devices, logs and crash-reports help to find the problems faster;
• inconsistency with the user interface behavior to the required scenarios - screen transition is not performed correctly or elements of the user interface don't work;
• problems with the layout on all or some models of devices – an analysis of screenshots between different devices allows you to find errors in the interface, when something is cropped out, does not fit or is not displayed, etc.
   

   

  Also, UI testing saves the tester from the monotony of manual testing, manually performing the same tests on a large number of devices day-by-day. Depending on which platform or framework you are developing (iOS, Android, Xamarin, ReactNative), you can use the following frameworks for testing:

• Appium (JUnit/Java) for Android and iOS;
• Espresso (Java) for Android;
• Xamarin.UITest for Android and iOS;
• XCUITest for iOS.

   
  In our example, we will focus on Xamarin.UITest for Android, which allows you to write test scripts in C#.

  The main task that UI tests perform is the simulation of user actions on a real device: tapping the screen, data entry, gestures. You should keep in mind that the tests may not only interact with the application as a black box, selecting items by the text or screen coordinates contained in them, but also click, for example, UI elements with internal identifiers specified in the source code. To do this, you need to set the value in a specific field (see the documentation for test frameworks) on the UI component. In the case of Xamarin.Forms, you need to set the AutomationId property (see Figure 2), which will be automatically assigned to the native AccessibilityIdentifier fields in iOS and ContentDescription on Android. Using of AutomationId makes testing more robust, and also helps ensure fewer breaks in tests if developers will change the underlying controls.

More information on how to write UI tests can be found in the official documentation:
https://docs.microsoft.com/en-us/appcenter/test-cloud/uitest/
If you use other frameworks, you can find the correct documentation here:
https://docs.microsoft.com/en-us/appcenter/test-cloud/supported-frameworks

Smoke, Acceptance and GUI Tests

Testing should be systematic, so there is no need to "always test everything on all devices.” It can be a long and very expensive process, both in terms of time and money. If we consider the process of developing mobile applications, we can distinguish the following sets of automatic functional UI-tests:

• Smoke-tests, added as an element of the automatic CI/CD pipeline. Verify that the application simply runs on real devices. These tests will find problems after updating the dependent libraries (if they are not supported on the oldest or newest versions of the OS, or on a certain CPU architecture) and makes sure that the build will run on real device.
• Acceptance (Beta) - a set of key user scripts for which the application is created. It allows to check that at least all the key elements of the user interface and business logic are executed correctly. These tests should be launched on the Top Devices measured in App Center analytics, to focus their testing on hardware most popular to the app user base.
• GUI - an extended set of short tests to verify all screens (layout, performance, key controls) behave correctly on the widest possible number of devices.

  The proposed test suites will allow you to perform a comprehensive test of the application in an automatic manner, divided it into stages (the example shows a breakdown for 2-week sprints). Below is an example of a test plan - different types of tests on different sets of devices. Figure 3 shows how much time is spent on different types of automated tests.

Intervals for Performing Automated UI Tests

It’s also great to use Nightly UI tests that will run automatically every night to test the latest app version on a large set of devices. But this step will require some configurations of Azure Scheduler and AppCenter Public API. Nightly testing is not covered in this article.

And if it's better to make Smoke-tests an automatic pipeline step, then the Acceptance and GUI tests will be started manually by the QA engineer. Before that, it's better to debug them on the local machine.
In order to somehow distinguish different types of tests (by default, always run all the tests in the project), the tester can use the categories mechanism available in popular frameworks. For Xamarin.UITest you must specify the Category attribute for each test or for a single method - see Figure 4.

You should thereafter consider these categories when sending tests to App Center:

After the completion of each test, a notification is sent to the email with test results, which are also available on the App Center site (see Figure 6).

You can view more detailed information of each step on each test device, including a screenshot and technical information.

With App Center you can perform both unit tests and automated UI tests within a CI/CD pipeline. You should launch your unit tests on a post-clone step and UI-tests on post-build step. While planning your UI test strategy (how many testing devices, how many tests, and how often to perform them) you should consider that UI testing requires more time than unit testing. Roughly you can calculate required device time (you pay for rent of devices in App Center) in the manner of “1 script step” = “1 device minute”. For example, if you are writing scripts with 60 steps it will take up to 60 minutes on one device. To make testing wiser you should split it for Smoke, Acceptance (beta) and GUI categories.
In this article, we explained how to test your Xamarin.Forms application with App Center. In the next article, we’ll focus on distribution and monitoring of your Xamarin applications: stay tuned!