Test Automation & QA

High level Goals and Objective of the Test Automation system

Goals - reduce regression errors in the app, build developer confidence, improve quality and stability of apps

Objectives -

• 100% coverage of stable core features Login, discovery, lending, playback or render, content navigation
• Testing on 80% of mobile install base device and OS's
• 95% of PC user base for Web App

Process

• When a change is merged into the mobile codebase, new binaries for all relevant apps are built and placed in a location where the integrated test system can be initiated.
  • At a minimum, this includes the SimplyE mobile app and a "vanilla" app that works just like SimplyE but contains no DRM secrets and no code that is not open source.
• When a change is merged into the circulation-patron-web web-based client repo integrated test system can be initiatied
• Integration tests are able to use a variety of test credentials against a variety of real servers and real libraries, as necessary to fulfill the purpose of the test.
• Integration tests are able to operate on different types of data without having to hope that the server sends them appropriate data.
• As much as possible, integration tests are run through the standard mobile interface accessible to human end-users, and do not require special lower-level access to mobile APIs.

In general the automation lifecycle looks as on the picture below

GitHub Actions: GitHub Actions are used to launch test runs.  There are two test automation work-flows implemented using GitHub Actions

• Mobile Automated Test Cycles:  Workflow will be automatically started if there is a new commit into the following repositories:
  • https://github.com/NYPL-Simplified/iOS-binaries (master branch)
  • https://github.com/NYPL-Simplified/android-binaries (develop and SimplyE branches)

• Mobile Manual run using GitHub Actions: Workflow for Autotest can can be initiated manually as needed.

  1. Go to https://github.com/NYPL-Simplified/integration-tests-android/actions (NOTE: This autotest repository covers both Android and iOS platforms)
  2. Select Test Run workflow and specify the repository, tag and release version:  See detailed instructions in the following Article: Mobile Autotest solution
• Web Automated Test Cycles: 
• Web Manual run using GitHub Actions:

Unit Test:

• Circulation Manager
• Mobile

Integration, UX and Regression Tests:

Repositories:

• Mobile Integration Tests: https://github.com/NYPL-Simplified/integration-tests-android
• Web Integration Tests (CPW):  https://github.com/NYPL-Simplified/integration-tests-web
• iOS Binaries: https://github.com/NYPL-Simplified/iOS-binaries
• Android Binaries: https://github.com/NYPL-Simplified/android-binaries

Test Suite:

Test Run Reporting:

Project Epic: https://jira.nypl.org/browse/SIMPLY-3233