...

• A mock of BrowseCreateDAOOracle has been done due to an incompatibility between H2 and the "upper" function call. This will affect tests related to case sensitivity in indexes.
• Many objects (like SupervisedItem) lack a proper definition of the "equals" method, which makes comparison between objects and unit testing harder
• Update method of many objects doesn't provide any feedback, we can only test if it raises an exception or not, but we can't be 100% sure if it worked 
• Many objects have methods to change the policies related to the object or children objects (like Item), it would be good to have some methods to retrieve these policies also in the same object (code coherence)
• There are some inconsistencies in the calls to certain methods. For example getName returns an empty String in a Collection where the name is not set, but a null in an Item without name
• DCDate: the tests raise many errors. I can't be sure if it's due to misunderstanding of the purpose of the methods or due to faulty implementation (probably the previous). In some cases extra encapsulation of the internals of the class would be advisable, to hide the complexities of the Calendars (months starting by 0, etc)
• The Authorization system gets a bit confusing. We have AuthorizationManager, AuthorizationUtils, methods that raise exceptions and methods that return booleans. Given the number of checks we have to do for permissions, and that some classes call methods that require extra permissions not declared or visible at first, this makes creation of tests (and usage of the API) a bit complex. I know we can ignore all permissions via context (turning on and off authorizations) but usually we don't want that
• Community: set logo checks for authorization, but set metadata doesn't. It's in purpose?
• Collection: methods create and delete don't check for authorization
• Item: there is no authorization check for changing policies, no need to be an administrator
• ItemIterator: it uses ArrayLists in the methods instead of List
• ItemIterator: we can't verify if the Iterator has been closed
• Metadata classes: usually most classes have a static method to create a new instance of the class. Instead, for the metadata classes (Schema, Field and Value) the method create is part of the object, thus requiring you to first create an instance via new and then calling create. This should be changed to follow the convention established in other objects (or the other objects should be amended to behave like the Metadata classes)
• Site: this class extends DSpaceObject. With it being a Singleton, it creates potential problems, for example when we use DSpaceObject methods to store details in the object. It's this relation necessary?

Fixes done:

- Bitstream: added an "isDeleted" method to verify if a bitstream has been deleted
- Bundle: added methods to check the policies of bundle and its bitstreams
- Collection: just a comment: delete requires authorization to remove the template Item and write, not to remove. Is that correct?
- Community: when a community is created with a parent, it's added as a child community immediately
- DCLanguage: added checks for null name in languages
- FormatIdentifier: fixed the check for filename == null in guessFormat 
- SiteTest: the test is in the abstract DSpaceObjectTest so I've make it inherit AbstractUnitTest.  I see the class has almost no usage so we could remove the inheritance from DSpaceObject, but I'm not sure if to do this. It's something that we should ask the developers?
- Several equals and hashCode methods added for other issues in tests
Pending of a review by a DSpace developer:
- DCDate: Here many tests fail because I'm not sure of the purpose of the class. I would expect it to hide the implementation of Calendar (with all those things like months starting by 0 and other odd stuff) so it's easier to use, but it seems that's not the case...

• Bitstream: added an "isDeleted" method to verify if a bitstream has been deleted
• Bundle: added methods to check the policies of bundle and its bitstreams
• Collection: just a comment: delete requires authorization to remove the template Item and write, not to remove. Is that correct?
• Community: when a community is created with a parent, it's added as a child community immediately
• DCLanguage: added checks for null name in languages
• FormatIdentifier: fixed the check for filename == null in guessFormat 
• SiteTest: the test is in the abstract DSpaceObjectTest so I've make it inherit AbstractUnitTest.  I see the class has almost no usage so we could remove the inheritance from DSpaceObject, but I'm not sure if to do this. It's something that we should ask the developers?
• Several equals and hashCode methods added for other issues in tests

Proposals:

To solve the previous issues, some proposals are done:

...

JMockit 0.998 has been added to the project to provide a mocking framework to the tests.

ContiPerf

ContiPerf is a lightweight testing utility that enables the user to easily leverage JUnit 4 test cases as performance tests e.g. for continuous performance testing.

...

The project makes use of H2 version 1.2.137

Unit Tests Implementation

These are tests which test just how one object works. Typically test each method on an object for expected output in several situations. They are executed exclusively at the API level.

...

The fact that we load the tests configuration from a dspace-test.cfg file means we are only testing the classes against a specific set of configurations. We probably would like to have tests that runs with multiple settings for the specific piece of code we are testing. This will require some extra classes to modify the configuration system and the way this is accessed by DSpace.

How to build new tests

To build a new Unit Test, create the corresponding class in the project dspace-test, under the test folder, in the package where the original class belongs. Tests for all the projects (dspace-api, dspace-jspui-api, etc) are stored in this project, to avoid duplication of code. Name the class following the format <OriginalClass>Test.java.

...

Be aware that tests of classes that extend DSpaceObject should extend AbstractDSpaceObjectTest instead due to some extra methods and requirements implemented in there.

How to run the tests

The tests can be activated using the commands

mvn package -Dmaven.test.skip=false  //builds DSpace and runs tests

  or
mvn test -Dmaven.test.skip=false     //just runs the tests

or by changing the property "activeByDefault" at the profile (skiptests) in the main pom.xml file, at the root of the project and then running

mvn package  //builds DSpace and runs tests
  or
mvn test     //just runs the tests

...

The only difference right now between Unit Tests and Integration Tests is that the later include configuration settings for ContiPerf. These is a performance testing suite that allows us to reuse the same methods we use for integration testing as performance checks. Due to limitations mentioned in the following section we can't make use of all the capabilities of ContiPerf (namely, multiple threads to run the tests) but they can be still be useful.

Limitations

Tests structure

These limitations are shared with the unit tests.

The solution to copy the file system is not a very elegant one, so we appreciate any insight that can help us to replicate the required files appropriately.

The fact that we load the tests configuration from a dspace-test.cfg file means we are only testing the classes against a specific set of configurations. We probably would like to have tests that runs with multiple settings for the specific piece of code we are testing. This will require some extra classes to modify the configuration system and the way this is accessed by DSpace.

Events Concurrency Issues

...

Due to these concurrency issues, ContiPerf can only be run with one thread. This slows the process considerably, but until the concurrency issue is solved this can't be avoided.

How to build new tests

To build a new Integration Test, create the corresponding class in the project dspace-test, under the test folder, in the package where the original class belongs. Tests for all the projects (dspace-api, dspace-jspui-api, etc) are stored in this project, to avoid duplication of code. Name the class following the format <RelatedClasses>IntegrationTest.java.

There are some common imports and structure, you can use the following code as a template:

//Add DSpace licensing here at the top!
package org.dspace.content;

import java.sql.SQLException;
import org.dspace.core.Context;
import org.junit.*;
import static org.junit.Assert.* ;
import static org.hamcrest.CoreMatchers.*;
import mockit.*;
import org.apache.log4j.Logger;
import org.dspace.core.Constants;
/**
 * This is an integration test to validate the metadata classes
 * @author pvillega
 */
public class MetadataIntegrationTest  extends AbstractIntegrationTest
{
    /** log4j category */
    private static final Logger log = Logger.getLogger(MetadataIntegrationTest.class);


    /**
     * This method will be run before every test as per @Before. It will
     * initialize resources required for the tests.
     *
     * Other methods can be annotated with @Before here or in subclasses
     * but no execution order is guaranteed
     */
    @Before
    @Override
    public void init()
    {
        super.init();
    }

    /**
     * This method will be run after every test as per @After. It will
     * clean resources initialized by the @Before methods.
     *
     * Other methods can be annotated with @After here or in subclasses
     * but no execution order is guaranteed
     */
    @After
    @Override
    public void destroy()
    {
        super.destroy();
    }

    /**
     * Tests the creation of a new metadata schema with some values
     */
    @Test
    @PerfTest(invocations = 50, threads = 1)
    @Required(percentile95 = 500, average= 200)
    public void testCreateSchema() throws SQLException, AuthorizeException, NonUniqueMetadataException, IOException
    {
        String schemaName = "integration";

        //we create the structure
        context.turnOffAuthorisationSystem();
        Item it = Item.create(context);

        MetadataSchema schema = new MetadataSchema("htpp://test/schema/", schemaName);
        schema.create(context);
        [...]
        
        //commit to free locks on tables
        context.commit();

        //verify it works as expected
        assertThat("testCreateSchema 0", schema.getName(), equalTo(schemaName));
        assertThat("testCreateSchema 1", field1.getSchemaID(), equalTo(schema.getSchemaID()));
        assertThat("testCreateSchema 2", field2.getSchemaID(), equalTo(schema.getSchemaID()));              [...]
        //clean database
        value1.delete(context);
        [...]

        context.restoreAuthSystemState();
        context.commit();
    }

}

The sample code contains common imports for the tests and common structure (init and destroy methods as well as the log). You should add any initialization required for the test in the init method, and free the resources in the destroy method. 

The sample test shows the usage of the assertThat clause. This clause (more information in JUnit help) allows you to check for condition that, if not true, will cause the test to fail. We name every condition via a simple schema (method name plus an integer indicating order) as the first parameter. This allows you to identify which specific assert if failing whenever a test returns an error.

Please be aware methods init and destroy will run once per test, which means that if you create a new instance every time you run init, you may end up with several instances in the database. This can be confusing when implementing tests, specially when using methods like findAll.

If you want to add code that it's executed once per test class, edit the parent AbstractUnitTest and its methods initOnce and destroyOnce. Be aware these methods contain code used to recreate the structure needed to run DSpace tests, so be careful when adding or removing code there. Our suggestion is to add code at the end of initOnce and at the beginning of destroyOnce, to minimize the risk of interferences between components.

How to run the tests

The tests can be activated using the commands

mvn package -Dmaven.test.skip=false  //builds DSpace and runs tests
  or
mvn test -Dmaven.test.skip=false     //just runs the tests

or by changing the property "activeByDefault" at the profile (skiptests) in the main pom.xml file, at the root of the project and then running

mvn package  //builds DSpace and runs tests  
 or
mvn test     //just runs the tests

Be aware that this command will launch both unit and integration tests.

Code Analysis Tools

Due to comments in the GSoC meetings, some static analysis tools have been added to the project. The tools are just a complement, a platform like Sonar should be used as it integrates better with the structure of DSpace and we could have the reports linked to Jira.

...

The Selenium RC environment was discarded due to the complexity it would add to the testing process. If you are a DSpace developer and have the resources and expertise to set it up, we clearly recommend so. You can reuse the scripts generated by the Selenium IDE and add extra tests to be run with JUnit alongside the existing unit and integration tests, which allows you to build more detailed and accurate tests. Even if we recommend it and it is a more powerful (and desirable) option, we are aware of the extra complexity it would add to just run the functional tests, as not everybody has the resources to deploy the required applications. That's why we have decided to provide just the Selenium IDE scripts, as they require much less effort to be set up and will do the job.

Structure

Selenium tests consist of two components:

...

To install the Selenium IDE, first install Firefox and then try to download it from Firefox itself. The browser will recognize the file as an addon and it will install it.

Limitations

The resulting tests have several limitations:

• Tests are recorded against a specific run on a particular machine. This means some steps may include specific values for some variables (id numbers, paths to files, etc) that link the test to a particular installation and state in the system. As a consequence we have to ensure we have the same state in the system every time we run the tests. That may mean run the tests in a certain order and probably starting from a wiped Dspace installation.
• For the same reason as above, some scripts may require manual changes before being able to be run (to ensure the values expected by Selenium exist). This is specially important in tests which include a reference to a file (like when we create an item) as the path to the file is hard coded in the script.
• Tests have to be launched manually and can only be run in Firefox using the Eclipse IDE. We can launch all our tests at once, as a test suite, but we have to do so manually.
• Due to they way Selenium works (checking HTML code received and acting upon it) high network latency or slowness in the server may cause a test to fail when it shouldn't. To avoid this, it is recommended to run the tests at minimum speed and (if required) to increase the time Selenium waits for an answer (it can be set in the Options panel).
• Tests are linked to a language version of the system. As Selenium might reference, in some situations, a control by its text, a test will fail if we change the language of the application. Usually this is not a big problem as the functionality of the application will be the same independently of the language the user has selected, but if we want to test the application including its I18N components, we will need to record the actions once per language enabled in the system.

How to build new tests

Building a test in Selenium IDE is easy. Open the IDE (in Firefox, Tools > Selenium IDE) and navigate to your DSpace instance in a tab of Firefox (i.e.: http://localhost:8080/jspui/). Press the record button (red circle at top-right corner) in the Selenium IDE and navigate through your application. Selenium will record every click you do and text you write in the screen. If you do a mistake, you can right-click over an entry in the list and remove it.

...

You can use the Selenium IDE to generate the tests or write them manually by using the Selenese commands.

How to run the tests

To run the tests simply open the Selenium IDE (in Firefox, Tools > Selenium IDE) and navigate to your DSpace instance in a tab of Firefox (i.e.: http://localhost:8080/jspui/). Then, in the Selenium IDE, click on File > Open and select a test case. You can open as many files as you want, they will be run in the order you opened them.

...

A very common reason why a test fails is because the server returned the HTML slowly and Selenium was trying to locate and HTML element before having all the HTML. To avoid this make sure that Selenium speed is set to slow and increase the default timeout value in Options.

Provided tests

We have included some sample Selenium tests in Dspace so developers can experiment with them. The tests are located under "<dspace_root>/dspace-test/src/test/resources/Selenium scripts". They are HTML files following the format described above. They are doing some assumptions:

...

You can edit the tests (see the format above) and change the required values (like user and path to a file) to values which are valid in your system.

Advanced Usage

If you set up Selenium RC, you can reuse the test scripts to be run as JUnit tests. Selenium can export them automatically to Java classes using JUnit. For this open the Selenium IDE (in Firefox, Tools > Selenium IDE), click on File > Open and select a test case. Once the test case is loaded, click on File > Export Test Case As > Java (JUnit) - Selenium RC. This will create a Java class that reproduces the test case, as the following:

...