Using Our Automation SDK with Appium and Sauce Labs

Appium is a popular open source mobile automation tool and has a significant advantage with its approach to automated mobile testing. Appium is cross platform meaning you can test both iOS and Android from the same framework. I think we all can agree that minimizing the number of context switches required in a given day is always better.

With direct support for Appium available from our Automation SDK, your Appium tests can return even more value by pinpointing accessibility issues as well. This goes beyond what the iOS and Android accessibility APIs offer, by heling you find more issues that could be impacting the accessibility of your application. In addition, Evinced uses advanced algorithms to create a single actionable report that is easy to digest. Having had the pleasure of wading through a sea of 20+ reports created based on individual scans, this is a feature I am very excited about.

The one thing missing from our even more powerful testing practice is the ability to test on a wide variety of devices and operating systems. This is crucially important for both the Android and iOS platforms.

For Android, breadth of coverage is extremely important due to the large segmentation in the Android market (Samsung, Google, LG, HTC, OnePlus, Huawei, etc). All these manufacturers take the default Android OS image from Google and make it their own. This creates the need for additional testing to make sure we give the best experience possible to all users. For iOS, you may have noticed that once you upgrade your iOS device it is impossible to roll it back to a previous version. This way, proper testing requires a library of iOS devices on all the versions a potential customer might be using to ensure a great experience.

Fortunately, Sauce Labs, where Appium was born, has a comprehensive cloud based device pool where we can execute our Appium tests with the Evinced Engine to pinpoint accessibility issues on nearly any device and operating system combination.

In this post, we will discuss how to take an existing Appium test running on a Sauce Labs device and add the Evinced engine to scan for accessibility issues.

Let’s take a look at our Java JUnit Appium test running on a Sauce Labs Device:

import com.evinced.a11y.validator.appium.java.core.A11yValidatorOptions;
import com.evinced.a11y.validator.appium.java.core.EvincedA11yValidator;
import com.evinced.a11y.validator.appium.java.core.Report;
import io.appium.java_client.MobileBy;
import io.appium.java_client.ios.IOSDriver;
import io.appium.java_client.ios.IOSElement;
import io.appium.java_client.remote.MobileCapabilityType;

import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.List;

import org.openqa.selenium.remote.DesiredCapabilities;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class EvincedA11yValidatorTest {

    public static URL url;
    public static DesiredCapabilities capabilities;
    public static IOSDriver<IOSElement> driver;
    public static String sauce_username = System.getenv("SAUCE_USERNAME");
    public static String sauce_accesskey = System.getenv("SAUCE_ACCESS_KEY");
    
    @BeforeClass
    public static void setupAppiumDriver() throws MalformedURLException, IOException {
        URL sauceUrl = new URL("https://" + sauce_username +":" + sauce_accesskey + "@ondemand.us-west-1.saucelabs.com:443/wd/hub");
        capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone 12.*");
        capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, "iOS");
        capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "14.7");
        capabilities.setCapability(MobileCapabilityType.APP, "storage:b7ae6573-cab1-44a5-9b59-343f974376c3");
        
        driver = new IOSDriver<IOSElement>(sauceUrl, capabilities);
        driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
    }
    
    @AfterClass
    public static void tearDown() {
        driver.quit();
    }
    
    @Test
    public void testStationsScreenIsAccessible() {
        IOSElement firstStationTableCell = driver.findElement(MobileBy.className("StationTableViewCell"));
        firstStationTableCell.click();
        IOSElement playButton = driver.findElement(MobileBy.className("XCUIElementTypeButton"));
        playButton.click();
        Assert.assertTrue(!playButton.isDisplayed());
    }
}

The first place we want to take a look at is the @BeforeClass method. This is where we will want to initiate the EvincedAppiumSdk class by passing in the IOSDriver instance that has created an Appium session on the Sauce Labs device cloud. This will not impact any of the existing functionality of the driver, it will simply add the tools needed to add accessibility scans to our test. Check out the Sauce Labs platform configurator for all the information needed to select a cloud device. Next, add your Evinced service ID and API key to authenticate the Appium SDK. (These credentials can be found by logging into your Evinced account.)

@BeforeClass
public static void setupAppiumDriver() throws MalformedURLException, IOException {
    URL sauceUrl = new URL("https://" + sauce_username +":" + sauce_accesskey + "@ondemand.us-west-1.saucelabs.com:443/wd/hub");
    capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone 12.*");
    capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, "iOS");
    capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "14.7");
    capabilities.setCapability(MobileCapabilityType.APP, "storage:b7ae6573-cab1-44a5-9b59-343f974376c3");
    
    driver = new IOSDriver<IOSElement>(sauceUrl, capabilities);
    driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
    // Init the EvincedAppiumSdk
    evincedSdk = new EvincedAppiumSdk(driver);
    evincedSdk.setupCredentials("<Evinced Service ID>","<Evinced API Key>");
}

Now that we have added the capabilities need to the driver, the next step is to identify in our test where adding accessibility scans will give use the maximum coverage of our application. This will likely be after a tap or a swipe that takes us to a new page or opens a menu. Taking a look at our test we can identify the following points:

@Test
public void testStationsScreenIsAccessible() {
    // The main application screen has loaded. Let's scan for a11y issues!
    IOSElement firstStationTableCell = driver.findElement(MobileBy.className("StationTableViewCell"));
    firstStationTableCell.click();
    // We have clicked on a cell that has taken us to a new page. Let's scan for a11y issues!
    IOSElement playButton = driver.findElement(MobileBy.className("XCUIElementTypeButton"));
    playButton.click();
    // We have now entered the player view. Let's scan for a11y issues!
    Assert.assertTrue(!playButton.isDisplayed());
}

Now that we have identified the points in our test that would be good to scan, lets add the code:

@Test
public void testStationsScreenIsAccessible() {
    // Let's scan for a11y issues!
    evincedAppiumSdk.analyze();
    IOSElement firstStationTableCell = driver.findElement(MobileBy.className("StationTableViewCell"));
    firstStationTableCell.click();
    // Let's scan for a11y issues!
    evincedAppiumSdk.analyze();
    IOSElement playButton = driver.findElement(MobileBy.className("XCUIElementTypeButton"));
    playButton.click();
    // Let's scan for a11y issues!
    evincedAppiumSdk.analyze();
    Assert.assertTrue(!playButton.isDisplayed());
}

In this test we have added the scans directly to test code for demonstration purposes. For actual implementation we would recommend adding the scans to existing page objects methods to make maintenance simple and easy.

The last thing we need to do is generate the report of all the scans. We can do that in the @AfterClass method.

@AfterClass
public static void tearDown() {
    List<Report> reports = evincedSdk.reportStored(true);
    driver.quit();
}

This single line of code will automatically compile all of the data from the scans and generate a set easy to digest HTML and JSON report files.

We are now ready to execute our test!

Not only can we run our test on any of the hundreds of devices and operating system versions Sauce Labs provides, we also gain the additional insights provided such as videos, logs, and screenshots to make debugging our functional test quicker and easier. This is extremely powerful combined with the actionable Evinced HTML and/or JSON accessibility report containing all issues found, severity levels, descriptions, effect on end users, and a link to actionable information on how to resolve the issues.

Check out the Evinced and Sauce Labs pages for more information on getting started!