Despite being considered immature and/or a waste of time by many, automated testing of Android applicationis possible. Arguably it should be in the repertoire of any serious Android developer, but today I am not going to delve into this discussion and talk about Robotium, Robolectric and so on. I am just going to show you how it just got a lot easier .
Android testing is mostly done with the testing API in the SDK around what is called instrumentation testing. It is very close to what is often called integration testing since it basically means that you write a test suite that is basically an application that uses your tested application and confirms it behaves the way it should. This whole testing approach therefore relies on the fact that both applications are deployed to a device or emulator.
Doing this sort of testing has been ok within the IDE in the past, but did not go much beyond. Specifically if you wanted to run the tests on a number of devices and emulators, have it all automated and wanted to capture the test results it quickly became painful to manage.
And this is exactly where my recent improvements are stepping up. As you might know I have been involved with the Maven Android Plugin as core committer for quite a while. Recently I have been doing a lot merging for the upcoming 3.0.0 release. That resulted in a bunch of release becoming available that contain the critical enhancements I made for automated testing.
With the move away from the adb command line call to using ddmlib directly I introduced automated multi-device handling for the deploy, undeploy and redeploy goals.
Using the latest 3.0.0-alpha-6 release of the Maven Android Plugin you can now have multiple emulators running and devices attached and issue a command like
mvn android:deploy
and it will automatically iterate through and deploy the application to all of them. The same applies to pull and push goals of the plugin copying files to or from the device.
You can also narrow down the devices affected to only all emulators or only all devices or only a specific devices or emulator.
mvn android:deploy -Dandroid.device=emulator mvn android:deploy -Dandroid.device=usb mvn android:deploy -Dandroid.device=emulator-5554 mvn android:deploy -Dandroid.device=someAvdName mvn android:deploy -Dandroid.device=deviceSerialNumber
So now you are ready to deploy your application in one sweep to all attached devices and test your awesomeness on a 10 inch tablet, a 4 inch phone and some other device all manually. Well hold on. You forgot about the automated part of testing.
If you set up your instrumentation testing with Maven like the MorseFlash project in the official samples of the Maven Android Plugin the multi device/emulator support will also work for that. So it will deploy the application and the test suite to all of them and then run the tests on the lot one after another.
And that is where my next enhancement comes in – JUnit compatible test report files. One of the problems with the instrumentation testing as supplied by the Android SDK is the fact that no report files are created. Luckily a few projects have stepped up and produced implementations of the InstrumentationTestRunner class that create JUnit compatible test reports. The problem is that by using them you have to include the class in your test project or all of the different test projects, modify build scripts and so on. All this can become a maintenance headache. Beyond that you have to deal with the fact that the report file is created on the device/emulator. That is a problem if you want to e.g. look at the files conveniently or get them post processed into nice HTML for access e.g. in a CI server like Hudson. Typically you then have to integrate adb pull operations into your build to get the file off the emulator.
So rather than producing the report file on the device I modified the TestRunListener I already implemented for logging on the host to also spit out a correctly formatted xml file for each test run on each device.
The files will be created in the standard Maven test report output location target/surefire-reports and be named so that the file is picked up by tools like Hudson automatically. The file names contain AVD name, serial number, manufacturer and model so that the are more or less human readable. E.g. for my Motorola XOOM, my Nexus S and an emulator named 233 the files look sort of like that
TEST-0212324143a0d357_Motorola_MZ604.xml TEST-123453F1031F00EC_samsung_NexusS.xml TEST-emulator-5554_233_unknown_sdk.xml
The file will contain all system properties from the host, all device properties from the emulator or device as well as any testing related data like stack traces, failure texts and so on. If you run your Android application build with a Maven build like this on the Eclipse Hudson CI server you get great report accessibility for a test class.
Stack trace and so on for failed tests:
And overall results view
as well as historic tracking of changes from build to build. And all that happens without additional configuration of the build or the test project…
I hope this will edge you all one step closer to write tests as part of your Android development and use a CI server like Hudson.
If you want to see more of that sort of stuff you should come to my testing workshop and Android application development with Maven presentation at AnDevCon later this year and register with my last name as discount code 😉
Enjoy
manfred
PS: I am a committer on Eclipse Hudson, the Maven Android Plugin and a bunch of other things and can easily be found if you need help.
PPS: This should work on other CI servers like Jenkins and so on as well.
UPDATE: It seems like a regression kicked in later that disabled the default generation of the report. You can force it on with the plugin configuration
<test> <createReport>true</createReport> </test>
or with the command line parameter -Dandroid.test.createReport=true
11 comments » Write a comment