2013-01-16 - VIVO Search Strategy Mtg

Attendees

Jonathan Markow
Andrew Woods
Jonathan Corson-Rikert
Brian Caruso
Brian Lowe
Jim Blake

Agenda

1. Understand how we move forward
   • Important for new sponsorship
2. Options
   • Are we interested in hosting?
   • Are we interested in development?

Discussion

Existing tooling

1. LinkedIndexBuilder
   • Crawls data over VIVO instances

DuraSpace

1. We see cool opportunities for VIVOSearch
2. Division of labor
   • Existing VIVO Search team do development VIVO side
   • Duraspace will host and do system adminstration
3. Longer term
   • Interested in greater inter-operation
4. View collaboration as team effort

VIVOSearch technology

1. Built software to create a Solr index
2. Built webapp over the index
3. Harvesting was never brought to production state
4. Initial plans for Hadoop
   • They did not have the Hadoop experience
5. Went was Scala
   • Have some regrets with Scala implementation
   • If could do it again, would go in the direction of Hadoop
6. Development options
   • Try to continue with the Scala-Actor code (not recommended)
   • Or, take the useful parts and move towards Hadoop

Approach

1. Take a high-level, broad view
2. Goal: have a VIVOSearch app for range of institutions
3. Come up with comprehensive list of questions
   • What is the optimal platform?
   • What should the production app should do?
   • How many institutions?
   • What do we need to run it?
   • What do we need to support it?
4. Once we start to answer questions
   • We can start to come up with tasks

System parts

1. Drupal website
2. Solr index
   • Vanilla Tomcat with Solr
3. Solr is populated with Scala-Actor code
4. Components interact via HTTP
5. Drupal frontend is a bit unknown
   • Others worked on it
   • Was considered beta
   • Probably in Drupal 6
   • Can not stretch it to 200 schools
6. Earlier conversations around using a js-solr library
   • Do not recall the name of the library
7. Question: is current state in production state or beta?
   • Mostly beta, except Solr index
   • No reason to stick with Drupal, except that it exists
   • Indexing backend is considered a deadend
8. RDF cataloging project
   • Take MARC records in catalog, running them through RDF
9. Should we be considering Nutch?
10. We need to consider how much load we put on institution VIVOs in indexing

Scenarios

1. Replicate VIVOSearch.org in DuraSpace infrastructure
2. Some institutions would like to pick up app an run locally
3. Key selling point: agnostic to ??
4. Would like to validate user RDF in the future

Demo

1. Performed brief demo
2. UI has gone through fair amount of rigor

Pilot group

1. CPSA
   • They could serve as a good pilot group
2. Existing partners
   • UF
   • Cornell
   • Colorado
   • Duke
   • Brown
3. CPSA could specify requirements/needs
   • Could turn into a bureaucratic/political log-jam
4. It may be more practical working with institutions with which we have existing relationships

Key to success: no client effort

1. How much work needs to happen on the client's side
   • Indexing is via linked-data requests to VIVO sites
   • There is no client work required
2. Valid RDF?
   • VIVO sites do not always provide valid RDF
   • Potentially some parts of the graph can be ignored
   • Client may or may not need to clean up their RDF
3. Goal: require no work on client side
4. Some schools do not have VIVO, but want to create "RDF export tool"
   • 1 or 2 schools fall in this category
   • Toronto
5. Running "locally"
   • May mean running in AWS for a specific institution
   • May mean running on local servers

Business proposition

1. Bringing in more institutions to increase researcher visibility
2. Make app/search internationally available

Scala

1. Only one developer who knows it
2. Actor library is no fun
3. Code that does RDF to Solr document is reused Java code
4. Scala was supposed to help with multi-threaded processing
   • Not a lot of cooperation between processes
5. Errors tend to be cryptic
6. Existing code was developed quickly, not for quality

Next Steps

1. Several moving pieces
2. Document questions that need to be answered
3. Document issues
4. Document activities
5. Together, this will give a clearer idea of the scope of the project
6. Need to also determine how to turn it into a viable "business"
7. Some discussions are already happening in the wiki
   • Suggestion to put documents in wiki
8. Leverage JIRA?
9. Create wiki space for VIVOSearch
   • Use VIVO crowd permissions
   • Grant admin rights to:? j2blake
10. Next call: week after next
11. Need to get a rough notion of timeframe and institution cost
12. Would like to have something at VIVO conference (Aug)
13. 12-1pm Tues Jan 29th
    • Jon to send out Web-Ex invite