Device Farm is an app testing service that enables you to test and interact with your Android, iOS, and Web apps on real, physical phones and tablets that are hosted by Amazon Web Services (AWS). There are two main ways to use Device Farm:
- Automated testing of apps using a variety of available testing frameworks
- Remote access of devices onto which you can load, run, and interact with apps in real time
Device Farm allows you to upload your own tests or use built-in, script-free compatibility tests. Because testing is automatically performed in parallel, tests on multiple devices begin in minutes.
A test report containing high-level results, low-level logs, pixel-to-pixel screenshots, and performance data is updated as tests are completed.
Device Farm supports testing of native and hybrid Android, iOS, and Fire OS apps, including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks. It also supports remote access of Android apps for interactive testing.
Device Farm currently provides support for the following test types:
Note: For the development of the automated tests, we used Appium JUnit framework for “Calculator” application on android device.
Prepare Android Appium Java JUnit Tests for running on AWS Device Farm
Android Appium Java JUnit tests must be contained in a .zip file. POM file needs to be set up according to the official AWS Device Farm documentation for JUnit.
Following dependencies should be added in maven POM file:
Go into your Appium Maven project directory from command prompt and run the following command to package the test content.
mvn clean package -DskipTests=true
After execution of this command, “zip-with-dependencies.zip” file will be created in the target folder, which will contain .jar files of the test classes and all dependency .jar files.
Note: “zip-with-dependencies.zip” file need to be uploaded when creating a run on AWS Device Farm.
Create test run on AWS Device Farm
- Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm
2. Click on “Create a new project”, enter project name and click on “Create project” button.
3. Click on “Create a new run” button on project page.
4. On the “Choose your application” page, choose either Native application (the Android and Apple button) or Web application (the HTML5 button). Upload your application file and click on the “Next step” button.
5. On Configure a test page, select test type, ex. “Appium Java JUnit”, upload “zip-with-dependencies.zip” file. It will take a few minutes for tests upload, after the test upload is finished select “Appium version” and click on “Next step” button.
6. Select the device (ex. Huawei P9) on “Select devices” and click on “Next step” button.
7. Click on “Next step” button on “Specify device state” page.
8. On “Review and start run” page set the execution timeout per device “ex. 10 minutes” and click on “Confirm and start run”.
9. A report in Device Farm contains information about a run, which is a request for Device Farm to test a single app against one or more devices. Reports in Device Farm contain pass and fail information, crash reports, test and device logs, screenshots, and performance data. Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem.
9.1 The “Video” section displays a downloadable video recording of the test.
9.2 The “Suites” section displays all test steps that contain pass and fail information and time of execution for each test step.
9.3 The “Logs” section displays any information Device Farm logged during the test.
9.4 The “Performance” section displays information about any performance data Device Farm generated during the test.
AWS Device Farm is easy to use and offers a very detailed view of the results, providing all information needed to reproduce the bug. The main advantage is the possibility to execute tests on a wide range of devices, allowing us to discover problems, which can’t be seen on the local test devices.