Using jquery and jquery-ui libraries we'll try to setup the autocomplete functionality on every textbox of advanced search page, we'll take advantage of Discovery Solr cores to add an additional search core for JSPUI.
Note: This implementation will display only words or partial words when using autocomplete funcionality because JSPUI indexes fields splitting them in words and removing some suffixes, this is bad for searching full text like author names. In a future section I will describe how to set custom text indexers like "full text" to our fields.
Once finished our textboxes must work like this:
Administrators that use XMLUI interface will notice that Discovery functionality strongly increases searching and browsing by the use of layered navigation and textbox autocomplete among others. To achieve these goals an Apache Solr search server is deployed in [dspace-source]/dspace/solr to take care of all query-reponse handling.
JSPUI interface use an Apache Lucene Directory (located on [dspace]/search ) to search and browse by, but it leaks of previously mentioned functionalities.
We'll use a servlet class to query a SOLR core to generate a JSON responses, this responses will be wired to a textbox with autocomplete functionality. Here is a simple diagram of this dataflow.
First steps: Preparing JSPUI advanced search page
First of all we'll add jquery and jquery-ui libraries to advanced search .jsp page. We can add them on head section of advanced search page [dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/search/advanced.jsp, or use a on/off switch approach to attach them as described on Adding jQuery (or other script library) support on JSPUI. In this manual we'll attach jquery libraries by appending them on head section.
First we must download jquery and jquery-ui script libraries from jQuery UI webpage, we'll download it's full version so we can use more functionalities on future improvements. All files are zipped inside a structure like this:
- Folder - css
- Folder - js
- Folder - development-bundle
- File - index.html
We'll use only js and css folders for our purposes. I'll describe selected folders content:
- js : Contains jquery-1.7.2.min.js and jquery-ui-1.8.21.custom.min.js script files
- css: Contains ui-lightness folder, here are all necessary .css style files and images to display ui-lightness visual theme, (name will vary if we selected another visual theme on jQuery UI page)
We'll create a new folder inside [dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/static/js/jqueryui and copy all previously described files inside it.. Structure will look like this:
[dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/static/js/jqueryui/jquery-1.7.2.min.js
[dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/static/js/jqueryui/jquery-ui-1.8.21.custom.min.js
[dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/static/js/jqueryui/css/...
Now edit [dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/search/advanced.jsp
...
<%
//Add this code on top section
//Generate JSPUI Solr path url
String scheme = request.getScheme(); // http
String serverName = request.getServerName(); //request.getServerName(); // hostname.com
int serverPort = request.getServerPort(); // 80
String parentContextPath = ""; //Change this value if dspace not installed on html root folder
String contextPath = "/jspui";
String servletPath = "/searchJSON";
// Reconstruct original requesting URL
StringBuffer url = new StringBuffer();
url.append(scheme).append("://").append(serverName);
if (serverPort != 80)
url.append(":").append(serverPort);
url.append(parenContextPath).append(contextPath).append(servletPath);
String solrPath = url.toString();
%>
...
<!-- Add this lines after <dspace:layout> tag-->
<link type="text/css" href="<%=request.getContextPath()%>/static/js/jqueryui/css/smoothness/jquery-ui-1.8.20.custom.css" rel="stylesheet" />
<script type="text/javascript" src="<%=request.getContextPath()%>/static/js/jqueryui/jquery-ui-1.8.20.custom.min.js"></script>
<script type="text/javascript" src="<%=request.getContextPath()%>/static/js/jqueryui/jquery-1.7.2.min.js"></script>
<script type="text/javascript" src="<%=request.getContextPath()%>/static/js/autocomplete.js"></script>
<script type="text/javascript">
window.onload = function() {
monkeyPatchAutocomplete();
replaceSAXExpression('<%=solrPath %>', 10,'tfield1', 'tquery1');
replaceSAXExpression('<%=solrPath %>', 10,'tfield2', 'tquery2');
replaceSAXExpression('<%=solrPath %>', 10,'tfield3', 'tquery3');
}
</script>
...
<!-- Add onchange event call for every search combobox -->
<select name="field1" id="tfield1" onchange="JavaScript:replaceSAXExpression('<%=solrPath%>', 10 ,'tfield1', 'tquery1');">
<select name="field2" id="tfield2" onchange="JavaScript:replaceSAXExpression('<%=solrPath%>', 10 ,'tfield2', 'tquery2');">
<select name="field3" id="tfield3" onchange="JavaScript:replaceSAXExpression('<%=solrPath%>', 10 ,'tfield3', 'tquery3');">
It's time to create a javascript library file that will contain a pair of functions. First one will patch autocomplete default behavior and will bold every part of result text that match our typed words. Second one will replace SAX expression used in autocomplete textbox everytime we change select value (we must search by other index). It will reside in [dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/static/js/autocomplete.js
/**
* Functions used on advanced search page
*/
function monkeyPatchAutocomplete() {
// don't really need this, but in case I did, I could store it and chain
var oldFn = $.ui.autocomplete.prototype._renderItem;
$.ui.autocomplete.prototype._renderItem = function( ul, item){
var term = this.term.split(' ').join('|');
var re = new RegExp("(" + term + ")", "gi") ;
var t = item.label.replace(re,"<b>$1</b>");
return $( "<li></li>" )
.data( "item.autocomplete", item )
.append( "<a>" + t + "</a>" )
.appendTo( ul );
};
}
function replaceSAXExpression(baseUrl , limit ,fieldName, fieldText){
var field = $('#'+ fieldName).val();
$('#' + fieldText).val('');
$('#' + fieldText).autocomplete({
source: function( request, response ) {
$.ajax({
url: baseUrl +"/terms?terms=true&terms.limit=" + limit + "&terms.sort=count&terms.regex.flag=case_insensitive&terms.fl=" + field + "&terms.regex=.*" + request.term + ".*&wt=json",
dataType: "json",
data: {
style: "full",
maxRows: 5,
name_startsWith: request.term,
},
success: function( data ) {
response( $.map( data.terms[field], function( item ) {
if(!$.isNumeric(item)){
return{
label: item,
value: "\"" + item + "\"",
}
}else {
return null;
}
}
));}
});
},
minLength: 1,
select: function( event, ui ) {
},
open: function() {
$( this ).removeClass( "ui-corner-all" ).addClass( "ui-corner-top" );
},
close: function() {
$( this ).removeClass( "ui-corner-top" ).addClass( "ui-corner-all" );
}
});
}
Now we must create a new SOLR core inside [dspace-source]/solr directory. To do so we'll use one of the existing cores and copy it's structure with this command:
bash$ cp -R [dspace-source]/dspace/solr/search [dspace-source]/dspace/solr/searchJSPUI
bash$ mkdir [dspace-source]/dspace/solr/searchJSPUI/data
bash$ mkdir [dspace-source]/dspace/solr/searchJSPUI/data/index
Edit [dspace-source]/dspace/solr/solr.xml and add one extra line with our new search core.
<cores adminPath="/admin/cores">
<core name="search" instanceDir="search" />
<core name="statistics" instanceDir="statistics" />
<core name="searchJSPUI" instanceDir="searchJSPUI" /> <!-- Add this line-->
</cores>
Overwrite new solr core configuration file [dspace-source]/dspace/solr/searchJSPUI/conf/schema.xml to handle lucene generated indexes.
If no modification to search indexes has been done on dspace.cfg file this configuration file will be enough:
schema.xml
Then we'll edit [dspace-source]/dspace/config/dspace.cfg and modify default JSPUI search index folder:
# Where to put search index files
search.dir = ${dspace.dir}/solr/searchJSPUI/data/index
So now our SOLR core is set up, new uploaded items indexes will be stored inside it's data folder. Proceed with next step:
Now we'll create the servlet responsible of reading autocomplete petitions and reponse with solr JSON structures. We could ommit this intermediate class but solr it's blocked from external querys due to security concerns, so this inner class will act as a bridge from client browser querys and internal solr server.
We'll create [dspace-source]/dspace-jspui/dspace-jspui-api/src/main/java/org/dspace/app/webui/servlet/SearchJSONServlet.java class and then will fill it's content with this code:
package org.dspace.app.webui.servlet;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLDecoder;
import java.sql.SQLException;
import java.util.Map;
import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.stream.StreamSource;
import org.apache.log4j.Logger;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Term;
import org.apache.lucene.queryParser.ParseException;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.queryParser.TokenMgrError;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.Sort;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.search.TopScoreDocCollector;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.Version;
import org.dspace.app.webui.util.JSPManager;
import org.dspace.authorize.AuthorizeException;
import org.dspace.content.Bitstream;
import org.dspace.content.Bundle;
import org.dspace.content.DCValue;
import org.dspace.content.Item;
import org.dspace.core.ConfigurationManager;
import org.dspace.core.Constants;
import org.dspace.core.Context;
import org.dspace.core.LogManager;
import org.dspace.handle.HandleManager;
import org.dspace.search.CustomQueryParser;
import org.dspace.search.DSIndexer;
import org.dspace.search.DSQuery;
import org.dspace.usage.UsageEvent;
import org.dspace.utils.DSpace;
public class SearchJSONServlet extends DSpaceServlet
{
/** log4j category */
private static Logger log = Logger.getLogger(SearchJSONServlet.class);
/* Create an DSpaceServlet Servlet */
public SearchJSONServlet()
{
super();
}
protected void doDSPost(Context context, HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException,
SQLException, AuthorizeException
{
doDSGet(context, request, response);
}
protected void doDSGet(Context context, HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException,
SQLException, AuthorizeException
{
String pathString = request.getPathInfo();
String queryString = request.getQueryString();
//Build localhost solr query path
String scheme = request.getScheme(); // http
String serverName = "127.0.0.1"; // Solr can only be accessed from localhost
int serverPort = request.getServerPort(); // 80
String parentContextPath = ""; // Change this value if dspace webpages are placed in a different folder other than html service root folder
String contextPath = "/solr";
String servletPath = "/searchJSPUI";
StringBuffer url = new StringBuffer();
url.append(scheme).append("://").append(serverName);
if (serverPort != 80)
url.append(":").append(serverPort);
url.append(parentContextPath).append(contextPath).append(servletPath);
url.append(pathString + "?");
url.append(queryString);
String solrPath = url.toString();
//Get response from solr server and send it as servlet reponse
StringBuilder jsonResponse = new StringBuilder();
URL sorlURL = new URL(solrPath);
URLConnection solrConnection = sorlURL.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(solrConnection.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
jsonResponse.append(inputLine + "\n");
in.close();
StreamResult result = new StreamResult(response.getOutputStream());
response.setHeader("Content-Length", String.valueOf(jsonResponse.length()));
response.getOutputStream().flush();
byte[] buf = jsonResponse.toString().getBytes();
response.setContentLength(buf.length);
ServletOutputStream servletOut = response.getOutputStream();
servletOut.write(buf);
}
}
This code captures input query string and builds a conformant SOLR query string that is send to SOLR server, it's reponse it's directly returned via result variable to reponse output stream.
Now we'll link our new servlet with it's appropriate url, to do so we we'll add this lines in [dspace-source]/dspace-jspui/dspace-jspui-webapp/src/main/webapp/WEB-INF/web.xml
<servlet>
<servlet-name>searchJSON</servlet-name>
<servlet-class>org.dspace.app.webui.servlet.SearchJSONServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>searchJSON</servlet-name>
<url-pattern>/searchJSON/*</url-pattern>
</servlet-mapping>