Selenium and Our Automation SDK

With the speed of modern release cycles, reliance on manual processes at any point of the development pipeline has a significant impact on developer productivity and release velocity. Functional testing has made this shift in the past decade but unfortunately accessibility testing is still heavily reliant on manual audits and reviews. There are a few legacy open source tools available for automated accessibility testing, however they are not built in a way that suits modern teams that may need to release dozens of times per day. In this post, I will discuss advantages of our Automation SDK for Selenium and how Evinced solves the problems created for modern development teams by legacy open source accessibility automation tools.

The Problems

There are a few reasons why the adoption of these legacy tools often fail:

Individual test code requires significant modification

A retail website may have an investment of well over 1000 Selenium tests to ensure it is functioning as expected. Not all of these tests will need to have accessibility scans added to them, but a significant subset will be needed to ensure all areas of the website are covered. To many developers, adding accessibility coverage to all those tests just doesn’t look to be worth the effort.

Maintenance

Each time the functionality of the site changes, the accessibility scans will also have to be changed so they continue to add value to the tests.

Managing Results

The results traditional tools provide are not consolidated and easy to digest. A single test might return multiple reports that then need to be deduplicated, archived, and reviewed in order to prioritize and fix the issues.

The Evinced Approach

Evinced has a new approach built for high velocity development teams with a modern release cadence. Our Automation SDK integrates with Selenium tests to automatically detect accessibility issues using computer vision and advanced algorithms. With the addition of just a few lines of code to your Selenium framework you can begin to analyze all the pages and DOM changes to offer a dynamic view of how your site can become more accessible. At the conclusion of the test, a rich and comprehensive JSON or HTML report is generated to track issues in any reporting tool.

Legacy Tool Implementation

Legacy accessibility tools require a scan to be manually inserted in the code after any action taken within the test that exposes additional page elements or triggers a DOM change. Not only that, after each scan we must check the Result object to see if any issues were detected, and if so, generate the report file.

In the example below, the code example on the left is a simple Selenium test that verifies the functionality of a travel site. The test navigates to the site, completes a form, clicks a button, and then verifies the form has been submitted and that we have navigated to a new page. On the right, we have the same test with the additional lines of code needed to incorporate accessibility scans to the test.

//imports ...

public class DemoTest {
  public static final String demoPage = "https://demo.evinced.com/";
  private WebDriver driver;

  @BeforeClass
  public void setUp() {
      driver = new ChromeDriver();
  }
  
  @AfterClass
  public void tearDown() {
      driver.quit();
  }
      
  @Test
  public void evincedTrvlTest() {
      // Navigate to the website under test
      driver.get(demoPage);
      // Click "Your New Home" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(1) > div > div.dropdown.line")).click();
      // Click "Where" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(2) > div > div.dropdown.line")).click();
      // Click "When" date picker
      driver.findElement(By.cssSelector(".react-date-picker")).click();
      // Click on the "Search" Button
      driver.findElement(By.className("search-btn")).click();
      // Assert we have navigated to the results page
      Assert.assertEquals("Page two | Evinced, Demos site", driver.getTitle());
  }
}

//imports ...

public class DemoTest {
  public static final String demoPage = "https://demo.evinced.com/";
  private WebDriver driver;

  @BeforeClass
  public void setUp() {
      driver = new ChromeDriver();
  }
  
  @AfterClass
  public void tearDown() {
      driver.quit();
  }

+  public void createReport(Results result) {
+      <Rule> violations = result.getViolations();
+      // If there are violations detected, create a report file
+      if (violations.size() > 0)
+        {
+         AxeReporter.writeResultsToJsonFile("/path/to/result/directory", result);
+        }
+  }
      
  @Test
  public void evincedTrvlTest() {
      // Navigate to the website under test
      driver.get(demoPage);
+      // Scan the current page state for accessibility issues
+      Results result = new AxeBuilder().analyze(driver);
+      // Check for violations and create report
+      createReport(result);
      // Click "Your New Home" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(1) > div > div.dropdown.line")).click();
+      result = new AxeBuilder().analyze(driver);
+      createReport(result); 
      // Click "Where" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(2) > div > div.dropdown.line")).click();
+      result = new AxeBuilder().analyze(driver);
+      createReport(result);
      // Click "When" date picker
      driver.findElement(By.cssSelector(".react-date-picker")).click();
+      result = new AxeBuilder().analyze(driver);
+      createReport(result);
      // Click on the "Search" Button
      driver.findElement(By.className("search-btn")).click();
+      result = new AxeBuilder().analyze(driver);
+      createReport(result);
      // Assert we have navigated to the results page
      Assert.assertEquals("Page two | Evinced, Demos site", driver.getTitle());
  }
}

Adding the scans after every interaction adds a significant amount of complexity to the test case. The code below needs to be added after any action that revealed more of the application or took us to a new page.

// Scan the current page state for accessibility issues
Results result = new AxeBuilder().analyze(driver);
// Check for violations and create report
createReport(result);

Now, I will be first to admit some refactoring could be done to clean things up a bit. Nonetheless, the time required to implement dozens of scans this way, plus the effort needed to maintain the tests going forward, equals a task that many developers simply won’t do.

In addition, within our example test we added 5 accessibility scans in order to cover all of the elements of the site. This means that at the end of this single test we will need to deduplicate and interpret 5 separate report files. A single test class could have 30+ tests leaving us with an arduous task of extracting value from over 100 files. This is a frustrating task especially when considering the amount of effort it takes to implement the legacy tools.

Implementing Our Automation SDK for Selenium

Let’s take a look at how Evinced does things differently. Here is our example test with the addition of accessibility scans using our Automation SDK. Again, our original test is on the left for easy comparison.

//imports ...

public class DemoTest {
  public static final String demoPage = "https://demo.evinced.com/";
  private WebDriver driver;

  @BeforeClass
  public void setUp() {
      driver = new ChromeDriver();
  }
  
  @AfterClass
  public void tearDown() {
      driver.quit();
  }
      
  @Test
  public void evincedTrvlTest() {
      // Navigate to the website under test
      driver.get(demoPage);
      // Click "Your New Home" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(1) > div > div.dropdown.line")).click();
      // Click "Where" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(2) > div > div.dropdown.line")).click();
      // Click "When" date picker
      driver.findElement(By.cssSelector(".react-date-picker")).click();
      // Click on the "Search" Button
      driver.findElement(By.className("search-btn")).click();
      // Assert we have navigated to the results page
      Assert.assertEquals("Page two | Evinced, Demos site", driver.getTitle());
  }
}

//imports ...

public class DemoTest {
  public static final String demoPage = "https://demo.evinced.com/";
  private EvincedWebDriver driver;

  @BeforeClass
  public void setUp() {
+      driver = new EvincedWebDriver(new ChromeDriver());
+      // Start the Evinced Engine
+      driver.evStart();
  }
  
  @AfterClass
  public void tearDown() {
+      // Stop the Evinced Engine
+      Report report = driver.evStop();
+      // Output the Accessibility results in JSON or HTML
+      EvincedReporter.writeEvResultsToFile("accessibilityReport", report, EvincedReporter.FileFormat.JSON);
      driver.quit();
  }
      
  @Test
  public void evincedTrvlTest() {
      // Navigate to the website under test
      driver.get(demoPage);
      // Click "Your New Home" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(1) > div > div.dropdown.line")).click();
      // Click "Where" dropdown
      driver.findElement(By.cssSelector("div.filter-container > div:nth-child(2) > div > div.dropdown.line")).click();
      // Click "When" date picker
      driver.findElement(By.cssSelector(".react-date-picker")).click();
      // Click on the "Search" Button
      driver.findElement(By.className("search-btn")).click();
      // Assert we have navigated to the results page
      Assert.assertEquals("Page two | Evinced, Demos site", driver.getTitle());
  }
}

As you can see below, the only setup needed to add accessibility scans to your existing Selenium tests is in the @BeforeClass and @AfterClass methods. There is no need to alter the individual test code in any way.

In the @BeforeClass method we simply need to pass the EvincedWebDriver a Selenium driver instance and call the evStart() method. This starts the Evinced engine which will continuously run in the background tracking all DOM changes and page navigations and collect accessibility violations as the tests execute.

@BeforeClass
public void setUp() {
    driver = new EvincedWebDriver(new ChromeDriver());
    // Start the Evinced Engine
    driver.evStart();
}

The last thing we need to do is stop the engine and generate the accessibility reports. We can do that in the @AfterClass method. A Report object is created when we stop the Evinced engine using the evStop() method. We can then choose to export the accessibility report as a JSON or HTML report (or both!).

@AfterClass
public void tearDown() {
    // Stop the Evinced Engine
    Report report = driver.evStop();
    // Output the Accessibility results in JSON or HTML
    EvincedReporter.writeEvResultsToFile("accessibilityReport", report, EvincedReporter.FileFormat.JSON);
    driver.quit();
}

Evinced provides a single, easy to understand, report that contains all of the information to identify, prioritize, reproduce and remediate any issue.

In sum, using our Automation SDK with Selenium is a huge leap in ease of use for developers. There’s no need to rewrite your existing tests in order to get started, and there’s no need to maintain the tests as your application and its UI changes over time. Accessibility scan results are easy to understand and prioritize, given the single consolidated report provided after the tests are complete. If you haven’t already, get an Evinced account and check out the Automation SDK for Selenium documentation to get started.