Versions Compared

Old Version 25

changes.mady.by.user Tim Donohue

Saved on

compared with

New Version 26

changes.mady.by.user Tim Donohue

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • JUnit 4 for writing tests
  • Mockito for mocking objects or responses within tests
  • H2 In-Memory Database for the database backend within test environment
  • Spring Boot Test for testing the Server Webapp / REST API, which itself uses Hamcrest (matchers), JsonPath (for JSON parsing) and other testing tools.
  • Travis CI GitHub Actions - This is our continuous integration (CI) system of choice. It automatically runs all code style checks & unit/integration tests for all Pull Requests. Our configuration for Travis CI GitHub Actions can be found in our .travis [src]/.github/workflows/build.yml
  • Coveralls  - Travis CI GitHub Actions runs test code coverage checks which it passes to Coveralls.io for easier display/analysis. If Coveralls finds code coverage has decreased significantly (which means tests were not implemented for new code), it will place a warning on a Pull Request.

...

Checklist for building good tests

  •  Integration Tests must be written for any methods/classes which require database access to test.  We feel integration tests are more important than unit tests (and you'll see this in our codebase).
  •  Unit Tests must be written for any (public or private) methods/classes which don't require database-level access to test.  For example, a utility method that parses a string should have a unit test that proves the string parsing works as expected.
  •  Include tests for different user types to prove access permissions are working. This includes testing as (1) an Anonymous user, (2) an Authenticated User (non-Admin), (3) an Administrator and/or Community/Collection Admin (as necessary).
  •  Include tests for known error scenarios & error codes.  If the code throws an exception or returns an 4xx response, then a test should prove that is working.
  •  Bug fix PRs should include a test that reproduces the bug to prove it is fixed.  For clarity, it may be useful to provide the test in a separate commit from the bug fix.
  •  Every test method must cleanup any test data created. See guidelines below for "187173784Cleaning up test data".
  •  Use "context.turnOffAuthorisationSystem()" sparingly, and always follow-up  with a "context.restoreAuthSystemState()" as soon as possible (and in the same test method).  As turning off the authorization system can affect the behavior of tests, only use these methods when you need to create or delete test data.
  •  If a test needs to temporarily modify a configuration property's value (in any *.cfg file), see the guidelines below for "187173784Changing configuration properties in tests"

Writing Integration Tests

...

  • All integration test classes must end in "IT". For example: "ItemRestRepositoryIT.java" or "StructBuilderIT.java"
  • All integration test classes should extend one of the Abstract classes provided.  These include:
    • For dspace-server-webapp, two Abstract classes exist, based on the type of integration test
      • org.dspace.app.rest.test.AbstractControllerIntegrationTest : This style of Integration Test is for testing any class which is an @Controller (which is the majority of classes in the REST API / Server Webapp).  This type of integration test uses Spring's built in caching and MockMVC to speed up the integration tests.  One example is the ItemRestRepositoryIT (which tests the "/api/core/items" endpoints in the REST API).
      • org.dspace.app.rest.test.AbstractWebClientIntegrationTest: This style of Integration Test is for testing classes which require a full webserver to run. As such, it will run slower, and is primarily used for non-REST API classes, such as testing SWORD or OAI-PMH. Those each require a full webserver to be started, as they are not Spring Controllers (and are not built on Spring Boot).  One example is OAIpmhIT (which tests the dspace-oai module)
    • For dspace-api, all Integration Tests should extend org.dspace.AbstractIntegrationTest
  • As Integration Tests generate a lot of test data, all Integration tests must be sure to follow our guidelines for "187173784Cleaning up test data" (see below).
  • Many example Integration Tests can be found in the dspace-server-webapp (see org.dspace.app.rest.*) and dspace-api modules.  We recommend looking at them for example code, etc.

...

Integration Tests necessarily have to create test data to verify the code they are testing is working properly.  But, just as importantly, they must cleanup any test data they create.  Integration tests which do not cleanup after themselves often result in random or odd Travis CI failures. These odd failures in Travis CI builds may occur anytime Travis the CI environment runs tests tests in a different order than your local machine and test data from an earlier test directly affects the results of a later test. Keep in mind, JUnit has no defined order of execution for tests. So, if you are seeing tests succeed on your system, but fail in Travis CI (or another system), then it's almost certainly because tests are running in a different order on the two systems...and one order is succeeding while another is failing (likely cause of test data not being cleaned up in prior tests).

...

  • Jasmine for writing unit/spec tests and Karma for running those tests
  • Protractor for writing integration / end-to-end (e2e) tests
  • Travis CI GitHub Actions - This is our continuous integration (CI) system of choice. It automatically runs all code style checks & unit/integration tests for all Pull Requests. Our configuration for Travis CI GitHub Actions can be found in our .travis [src]/.github/workflows/build.yml
    • Our Travis GitHub Actions configuration also starts up a Docker-based REST API backend for running end-to-end (e2e) tests against.
  • Coveralls  - Travis GitHub Actions CI runs test code coverage checks which it passes to Coveralls.io for easier display/analysis. If Coveralls finds code coverage has decreased significantly (which means tests were not implemented for new code), it will place a warning on a Pull Request.

...

Checklist for building good tests

  •  Unit Tests (i.e. specs) must be written for every Angular Component or Service.  In other words, every "*.component.ts" file must have a corresponding "*.component.spec.ts" file, and every "*.service.ts" file must have a corresponding "*.service.spec.ts" file.
  •  Integration (e2e) Tests are recommended for new features but not yet required.
  •  Include tests for different user types (if behaviors differ per user type). This includes testing as (1) an Anonymous user, (2) an Authenticated User (non-Admin), (3) an Administrator and/or Community/Collection Admin (as necessary).
  •  Include tests for known error scenarios.  For example, tests should validate when errors/warnings are expected to appear, and/or validate when buttons are expected to be enabled/disabled.
  •  Bug fix PRs should include a test that reproduces the bug to prove it is fixed.  For clarity, it may be useful to provide the test in a separate commit from the bug fix.

...

  • All e2e tests should be in the /e2e/src/ directory, per the Angular CLI best practices.
  • A single e2e test consist of two related files:
    • A Page object (ends in "po.ts") which is a class describing a high level page view, including logic for finding all elements on the page and navigating to the page URL. 
    • An e2e test file (ends in "e2e-spec.ts") which include Jasmine-based tests using the Page object.
  • Examples of writing Angular e2e tests can be found in this Introduction to e2e Testing with Angular CLI and Protractor blog post, or by referencing the existing tests in our codebase.
  • NOTE: be aware that when e2e tests run, they require using a REST API backend & test data. Therefore in Travis GitHub Actions CI, our tests run against a Docker based REST API (preloaded with basic test data) defined in the docker-compose-travisci.yml