...
These are tests which come from user-based use cases. Such as a user wants to search DSpace to find material and download a pdf. Or something more complex like a user wants to submit their thesis to DSpace and follow it through the approval process. These are stories about how users would preform tasks and cover a wide array of components within Dspace.
This tests are currently under development
Structure
Limitations
How to build new tests
...
Be aware that in this section we don't focus on testing the layout of the UI, as this has to be done manually to ensure we see exactly the same in different browsers. In this section we only consider tests that replicate a process via the UI, to ensure it is not broken due to some links missing, unexpected errors or similar issues.
Choices taken
To decide on a specific implementation of these tests some choices have been taken. They may not be the best but at this time they seemed the appropriate ones. Contributions and criticism are welcomed.
On one hand, functional tests run against a live instance of the application. This means we need a full working environment with the database and file system. It also means we are running them against the modified UI of a specific installation. As a consequence, we want the tests to be very generic or easily customizable, but we have to be aware that due to way Maven (and particularly its packaging system) works it isn't possible to run the tests as a step of the unit and integration testing described in the above sections.
On the other hand, if we focus on the tools available, the best choice we have available is Selenium, a suite of tools to automate testing of web applications. Selenium can be run in two modes: as a Firefox plug in that will allow us to record test scripts and run them later, or as a distributed system that allows us to run the tests against several browsers in different platforms.
We have to choose a way to run the tests that is easy to set up and adapt to a particular project by the developers, while limited by the options that maven and selenium provide. The decision has been to use the Selenium IDE to run the tests. This means the tests can only be run in Firefox and they have to be launched manually, but on the other hand they are easily customizable and runnable.
The Selenium RC environment was discarded due to the complexity it would add to the testing process. If you are a DSpace developer and have the resources and expertise to set it up, we clearly recommend so. You can reuse the scripts generated by the Selenium IDE and add extra tests to be run with JUnit alongside the existing unit and integration tests, which allows you to build more detailed and accurate tests. Even if we recommend it and it is a more powerful (and desirable) option, we are aware of the extra complexity it would add to just run the functional tests, as not everybody has the resources to deploy the required applications. That's why we have decided to provide just the Selenium IDE scripts, as they require much less effort to be set up and will do the job.
Structure
Selenium tests consist of two components:
- The Selenium IDE, downloadable here , which is a Firefox addon that can record our actions and save them as a test
- The tests, which are HTML files that store a list of actions to be "replayed" by the browser. If any of the actions can't be executed (due to a field or link missing or some other reason) the test will fail.
- The recommendation is to create one test per user-action (like creating an item). Several tests can be loaded at once in the IDE and run sequentially.
To install the Selenium IDE, first install Firefox and then try to download it from Firefox itself. The browser will recognize the file as an addon and it will install it.
Limitations
The resulting tests have several limitations:
- Tests are recorded against a specific run on a particular machine. This means some steps may include specific values for some variables (id numbers, paths to files, etc) that link the test to a particular installation and state in the system. As a consequence we have to ensure we have the same state in the system every time we run the tests. That may mean run the tests in a certain order and probably starting from a wiped Dspace installation.
- For the same reason as above, some scripts may require manual changes before being able to be run (to ensure the values expected by Selenium exist). This is specially important in tests which include a reference to a file (like when we create an item) as the path to the file is hard coded in the script.
- Tests have to be launched manually and can only be run in Firefox using the Eclipse IDE. We can launch all our tests at once, as a test suite, but we have to do so manually.
- Due to they way Selenium works (checking HTML code received and acting upon it) high network latency or slowness in the server may cause a test to fail when it shouldn't. To avoid this, it is recommended to run the tests at minimum speed and (if required) to increase the time Selenium waits for an answer (it can be set in the Options panel).
- Tests are linked to a language version of the system. As Selenium might reference, in some situations, a control by its text, a test will fail if we change the language of the application. Usually this is not a big problem as the functionality of the application will be the same independently of the language the user has selected, but if we want to test the application including its I18N components, we will need to record the actions once per language enabled in the system.
How to build new tests
Building a test in Selenium IDE is easy. Open the IDE (in Firefox, Tools > Selenium IDE) and navigate to your DSpace instance in a tab of Firefox (i.e.: http://localhost:8080/jspui/
). Press the record button (red circle at top-right corner) in the Selenium IDE and navigate through your application. Selenium will record every click you do and text you write in the screen. If you do a mistake, you can right-click over an entry in the list and remove it.
Actions are stored by default as a reference to the control activated. This reference is generic, meaning that Selenium might look for an anchor link (<a>) that points to a certain url (like '/jspui/handle/1234/5678'), independently of its id, name or position in the application. This means that the test will not be affected (usually) by changes in the layout or some refactoring. That said, in some specific cases you may need to edit the test cases to change some values.
Once you are finished, press again the record button. Then, in the Selenium IDE, go to File > Save Test Case and save your test case.
In Selenium the tests cases are a HTML file that stores data in a table. The table contains a row per action, and each row has 3 columns :
- An action to be run (mandatory). This can be an action like open, click, etc.
- A path or control id against which the action is executed (mandatory). This can point to an URL, a control (input, anchor, etc) or similar.
- A text to be added or selected in an input control (optional).
A sample of a Selenium test is:
| Code Block |
|---|
open /xmlui
clickAndWait link=Subjects
clickAndWait //div[@id='aspect_artifactbrowser_Navigation_list_account']/ul/li[1]/a
type aspect_eperson_PasswordLogin_field_login_email admin@dspace.org
type aspect_eperson_PasswordLogin_field_login_password test
clickAndWait aspect_eperson_PasswordLogin_field_submit
|
The code generated by the Selenium IDE for this would be like:
| Code Block |
|---|
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head profile="http://selenium-ide.openqa.org/profiles/test-case">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="selenium.base" href="" />
<title>Create_Comm_Coll_Item_XMLUI</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">Create_Comm_Coll_Item_XMLUI</td></tr>
</thead><tbody>
<tr>
<td>open</td>
<td>/xmlui</td>
<td></td>
</tr>
<tr>
<td>clickAndWait</td>
<td>link=Subjects</td>
<td></td>
</tr>
[... more actions ...]
</tbody></table>
</body>
</html>Create_Comm_Coll_Item_XMLUI
|
You can use the Selenium IDE to generate the tests or write them manually by using the Selenese commands.
How to run the tests
To run the tests simply open the Selenium IDE (in Firefox, Tools > Selenium IDE) and navigate to your DSpace instance in a tab of Firefox (i.e.: http://localhost:8080/jspui/). Then, in the Selenium IDE, click on File > Open and select a test case. You can open as many files as you want, they will be run in the order you opened them.
Once you have selected the test cases to run, ensure the speed of Selenium is set to slow (use the slider) and press either the "Play entire test suite" or "Play current test case" button (the ones with a green arrow), according to your intentions. Selenium will run the actions one by one in the order recorded. If at some point it can't run an action, it will display an error and fail the test. You can see the reason of the error in log at the bottom of the Selenium IDE window.
A very common reason why a test fails is because the server returned the HTML slowly and Selenium was trying to locate and HTML element before having all the HTML. To avoid this make sure that Selenium speed is set to slow and increase the default timeout value in Options.
Provided tests
We have included some sample Selenium tests in Dspace so developers can experiment with them. The tests are located under "<dspace_root>/dspace-test/src/test/resources/Selenium scripts". They are HTML files following the format described above. They are doing some assumptions:
- Tests assume that you are running them against a vanilla environment with no previous data. They may work in an environment with data, but it's not assured
- Tests assume you are running the English UI, other languages may break some tests.
- Tests assume a user with user name admin@dspace.org and password test exists in the system and has administrator privileges
- Tests assume a file exists at /home/pvillega/Desktop/test.txt
You can edit the tests (see the format above) and change the required values (like user and path to a file) to values which are valid in your system.
Advanced Options
If you set up Selenium RC, you can reuse the test scripts to be run as JUnit tests. Selenium can export them automatically to Java classes using JUnit. For this open the Selenium IDE (in Firefox, Tools > Selenium IDE), click on File > Open and select a test case. Once the test case is loaded, click on File > Export Test Case As > Java (JUnit) - Selenium RC. This will create a Java class that reproduces the test case, as the following:
Future Work
This project creates a structure for testing that can expanded. Future tasks would include:
...