High level Goals and Objective of the Test Automation system
Goals - reduce regression errors in the app, build developer confidence, improve quality and stability of apps
Objectives -
- 100% coverage of stable core features Login, discovery, lending, playback or render, content navigation
- Testing on 80% of mobile install base device and OS's
- 95% of PC user base for Web App
Process
- When a change is merged into the mobile codebase, new binaries for all relevant apps are built and placed in a location where the integrated test system can be initiated.
- At a minimum, this includes the SimplyE mobile app and a "vanilla" app that works just like SimplyE but contains no DRM secrets and no code that is not open source.
- When a change is merged into the circulation-patron-web web-based client repo integrated test system can be initiatied
- Integration tests are able to use a variety of test credentials against a variety of real servers and real libraries, as necessary to fulfill the purpose of the test.
- Integration tests are able to operate on different types of data without having to hope that the server sends them appropriate data.
- As much as possible, integration tests are run through the standard mobile interface accessible to human end-users, and do not require special lower-level access to mobile APIs.
In general the automation lifecycle looks as on the picture below
GitHub Actions: There are two test automation work-flows implemented using GitHub Actions
- Automated Test Cycles: Workflow will be automatically started if there is a new commit into the following repositories:
- https://github.com/NYPL-Simplified/iOS-binaries (master branch)
- https://github.com/NYPL-Simplified/android-binaries (develop and SimplyE branches)
Unit Test:
- Circulation Manager
- Mobile
Integration, UX and Regression Tests:
Repositories:
- Mobile Integration Tests: https://github.com/NYPL-Simplified/integration-tests-android
- Web Integration Tests (CPW): https://github.com/NYPL-Simplified/integration-tests-web
- iOS Binaries: https://github.com/NYPL-Simplified/iOS-binaries
- Android Binaries: https://github.com/NYPL-Simplified/android-binaries
Test Suite:
Test Run Reporting:
Project Epic: https://jira.nypl.org/browse/SIMPLY-3233