Purpose

The VIVO architectural fly-in was focused on bringing an architecturally-minded team together who individually represent distinct VIVO stakeholder constituencies for the purpose of developing architectural approaches required to address the direction of the project. The primary goal of the two-day face-to-face meeting was to assess and document a plan for improving the VIVO application architecture towards enabling and realizing the technical efforts defined in the "Statement on VIVO's Product Direction for 2019".

Participants

1. Huda Khan
2. Andrew Woods
3. Brian Lowe
4. Justin Littman
5. Ralph O'Flinn
6. Benjamin Gross
7. Jim Blake
8. Richard Outten
9. Alex Viggio

Leading up to the face-to-face meeting, the team convened six conference calls over two months with the goals of establishing a common understanding of:

• the purpose of the effort as well as 
• the perspectives held by each team member. 

In addition to collecting requirements, assessing existing features, sharing documentation resources, and drilling into areas of exploration, each team member provided hypothetical architectural diagrams which informed the face-to-face discussions.

Goals

At the beginning of the fly-in, the team defined and agreed on the following goals:

1. Translating the Statement of VIVO's Product Direction for 2019 into an actionable architecture
2. Engaging and incorporating technical efforts from parts of the VIVO community that have emerged adjacent to the core application
3. Evolving VIVO into a more modular, cloud-ready application. This includes defining modules that can be deployed as separate services (containerized) as well as enabling the replacement of those modules with standard cloud services (AWS Neptune, Azure Cosmos DB, etc)
4. Ensuring continuity for the community in order to facilitate incremental, manageable change.

Logistics

Great effort was put towards making sure that the discussions were open and supportive, and that there was space for the voices of all team members. Ground rules were established around:

• maintaining topic-focused discussions
• time-boxing sessions
• establishing goals for each session, and
• reviewing actions/takeaways from each session

The fly-in took place over the course of two days, starting at 9am and concluding at 5:30pm, followed by team dinners.

Topics

Ingest

Requirements

1. Ingest must support varied and currently unknown data sources
2. Ingest must be scalable: the use of different backend triplestores or datastores must be allowed to support site-specific scaling requirements
3. Ingest must support both JSON and RDF (note: JSON content must have a known mapping to RDF model)
4. Content must be validated before being ingested. Example validation mechanisms: SHACL, ShEx, JSON Schema
5. Ingest must expect models/shapes that are expected and able to be validated (note: initial models can be derived from what is in use in Freemarker UI)
6. Ingest tooling must support two modes of operation: hands-free (automated) and curated
7. Ingest tooling must support curation of data prior to ingest: disambiguation and reconciliation of entities
8. Ingest will be entity-centric vs triple-centric. Example entities: Person, Grant, Publication, Authorship
9. Ingest tooling must not require the use of a specific programming language
10. Ingest must support realtime, incremental updates

Out of scope

1. Extraction of data from data sources
2. Transformation of data from data sources

Below: DS1-3: Data sources, read in and validated based on “shapes” (or patterns or sets of triples representing entities), with entity resolution/URI creation as required, leading to a set of triples that can be read into VIVO. The arrow on the right side of the picture continues to VIVO in the next picture.

Below: The previous picture represents a view of the combine/ingest process that leads to VIVO in this picture (the munging/disambiguation box was supposed to be moved to the ingest process instead). Triples can then be requested for different entities which provide a UI. The portion on the bottom right identifies how the current VIVO system uses SPARQL queries defined in list views which together define what is expected to be displayed for specific entity types.

UI

Requirements

1. Current Freemarker UI will stay in-place for the scope of this plan (although deprecated)
2. UI must provide read access to VIVO data
3. UI must minimally be informed by the Production Evolution effort
4. UI must be based on data coming from a JSON endpoint
5. UI should render data served by a GraphQL server
6. GraphQL server should be configured with the same models used by ingest tooling (note: DocumentModifier.java may be updated to populate search index with these models)
7. UI must support accessibility
8. UI must support internationalization (i18n)
9. UI should avoid querying the triplestore when rendering

Decoupling VIVO

Requirements

Architectural concerns

1. From an architectural perspective, having a triplestore at the core of the application brings significant limitations
2. As we decouple components, we must ensure that we also decouple logic expectations between the components

...and more