Introduction
There is a problem that we meet
everyday in our development process. An application has a scenario to test; In
first build, this scenario already tested with Pass status. Later, in some
newer builds, some functions are added or edited. These functions somehow
related to the scenario we tested in the first build. So we need to re-test
this scenario again to make sure our application runs smoothly. And the problem
is we need to test the same scenario from build to build. It takes a lot of
time to re-test a scenario, especially, it is tested manually.
Today I’ll introduce you a new
technique to help us reduce a lot of time for re-testing, which is integrated in
XCode: Automation UI Testing.
What is XCode UI Testing?
I’m sure that all you us already know about
the concept of UI Testing. Before WWDC 2015, Xcode already has a tool included
in Instruments. Instruments is a set of tool for developer to play with their
apps. That tool is called: Automation.
Automation in Instruments
But the script in Automation is
written based on Javascript syntax and not really familiar to mobile developer,
especially iOS developer. Besides, Automation cannot help us with debugging,
detail error report and cannot put all test cases in 1 file.
Automation code is written in Javascript
But XCode UI testing can solve all
the problems above. Introduced in WWDC 2015, Xcode UI Testing is a brand new
tool which is integrated in Xcode 7. We can take a look at a small part in
Xcode UI Testing file. We can see it is more clearly, more beautiful, and more
friendly to iOS developer.
Automation code is written in Swift
Core technologies include
XCTest framework and Accessibility.
- XCTest provides the framework for UI
testing capabilities, integrated with Xcode. Creating and using UI testing
expands upon what you know about using XCTest and creating unit tests. You
create a UI test target, and you create UI test classes and UI test methods as
a part of your project. You use XCTest assertions to validate that expected
outcomes are true. Test is fully compatible with both Objective-C and Swift.
- Accessibility
is the core technology that allows disabled users the same rich experience for
iOS and OS X that other users receive. It includes a rich set of semantic data
about the UI that users can use can use to guide them through using your app.
Accessibility is integrated with both UIKit and AppKit and has APIs that allow
you to fine-tune behaviors and what is exposed for external use. UI testing
uses that data to perform its functions.
There also be UI Recording. This is
a really great feature of XCode UI Testing which help developer write
automation test code without putting too much effort. Because the code which
is generated by the recording is in
swift/objectiveC, the logic of UI element is unified to the development. This
would make our test cases built quickly and exactly.
APIs
There are 3 APs:
- XCUIApplication:
is a proxy for app being tested. It is independent from the application because
of running in a separate process.
- XCUIElement:
- XCUIElement is a proxy for UI elements.
- Elements can be different types: buttons, cells,
windows, etc.
- Elements has its own identifiers - strings
generated by the accessibility system: accessibility identifiers, labels,
titles etc.
- XCUIElementQuery:
XCUIElementQuery provides access to a specific element. The result of running
query is a set of current UI elements visible for the accessibility system.
With these 3 APIs, coding ui
automation test can emulate almost (with exceptions) end-user interaction.
XCTestAssertions
To evaluate the result, we al so
have Assertions listed by category. XCTestAssertions are categorized into 5
groups. Whenever we put assertions in our automation test code, we can evaluate
this is correctly functioned or not. For example, I delete an element, I want
to determine whether the element is existed or not, I can put XCTestAssertion
to compare value as well as existence of the element. Assertion will fail if the
value is not equal and we will get the fail report and we can know exactly
where the error is. How convenient it is!
Conclusion
XCode UI Testing
opens to iOS and OSX developer as well as tester a new way to do automation
test for iOS and OSX, easier, faster and familiar. But since it’s still a new
tool for us, it still has some limitations:
- Require iOS 9+ to run tests. Not only SDK, but latest iOS must be installed on testing device/simulator.
- Require device to be configured for development and connected to trusted host running Xcode.
- Require access to accessibility features set from privacy settings.
- Running on Mac OS only: if your QA-team have no Macs (like our do) that may be a problem.
- You could test only your main app: extensions neither watch app testing is not supported.
- Can't find any documentation for the moment.
- Can't find way to test anything outside the app. Can't switch to safari, make URL request and then go back again.
- Some other limitations may happen when running UI test
- Limitation on events, we have tap(), doubleTap() but no longPress(). Instead of
longPress(), we have to use an alternative way such as pressForDuration().
- Elements must be recognised by Accessibility. For some
complex application, this can be more complicated.
- This also required iOS 9 and above
to run test. So Testers also need a MAC to run all the test.
- Moreover, lack of documentation is
another challenge that we have to deal with.
Prospects
But over all of these challenges,
we have a fabulous tool to work with.
- If possible, we can save much time on
testing.
- In the future, Instrument Automation will be removed from Instruments
and Apple recommends are moving to Xcode UI Testing for all the UI Testing.
- 1 more important things is: this is
a tool from Apple, so the update to newer version of iOS with new UI control
will also be supported at the same time of updating. With third party, we have
to wait some time for it to update after the iOS updating.
So. It’s worth for trying right now!
