Developers / DCAT Meeting before OR16 on Mon, June 13, 2016

Who is invited?

• All Committers,
• All DSpace Community Advisory Team (DCAT) members,
• Any other interested DSpace developers or technology-savvy individuals

If you don't fall into one of the above categories, you are still welcome to attend. However, be warned that discussion will likely get very technical at times (which is why we recommend you be a developer or have a technology background).

Logistics

• Day: Monday, June 13th, 2016 (Workshop / Tutorial day)
• Time: 9:00am - 12:30pm (with a break at 11am)
  
  • NOTE: This working meeting is considered a "workshop", so it is listed as part of the Workshop/Tutorial day schedule.
• Where: Lloyd A in Lloyd Institute on campus
  • OR2016 venues map: https://drive.google.com/open?id=1K1FspyeZLbl7ty1-5vK1q1h6DMI&usp=sharing
  • Campus map: https://www.tcd.ie/Maps/assets/pdf/tcd-main-campus.pdf

Agenda

9:00am : Introductions

• Name, Institution, How are you using DSpace?, How have you customized DSpace (or anything you'd be willing to contribute back)?
• Assign a note taker!

9:15 - 11:00am :  New UI Initiative , Decision Making / Collaboration

• DSpace New UI Initiative and Roadmap to DSpace 7 (Presented Slides at http://www.slideshare.net/tdonohue/new-ui-update-or16-developer-dcat-meeting)
  
  • How we got here (DSpace UI Prototype Challenge , also see YouTube playlist of video updates)
  • Why Angular 2? (Risks vs opportunities)
  • Proof of concept: https://github.com/DSpace-Labs/angular2-ui-prototype
  • DSpace 7 proposed architecture (overview), including necessary REST API enhancements
  • How you can help / How would you like to help
    • How can DCAT support this initiative / provide feature suggestions/expertise?  How can other developers / institutions get involved?
• DCAT, Committers and Steering/Leadership: How we collaborate and make decisions together  (NOTE: Unfortunately, we ran out of time and never got to this topic)
  
  • Streamlining the ways to get new features approved and into DSpace
    • Which features should be pushed up for DCAT or Steering approval first?
  • Moving the code through technical approval process. Are there ways to streamline?
  • Streamlining the ways we share expertise / contributions between and within our groups?

11:00 - 11:15am : Coffee Break

11:15am - 12:30pm: DSpace-CRIS (and other topics)

• DSpace-CRIS discussion (led by Michele Mennielli and Andrea Bollini of Cineca)
  • Why the Steering Group sees this topic as important (Jonathan Markow)
  • Demo / Overview of DSpace-CRIS functionality
  • Brief overview of DSpace-CRIS data model / technical differences with out-of-the-box DSpace
  • Discussion: Where does the repository end and the CRIS begin?
    
    • What features of DSpace-CRIS are of high interest to DSpace users in general?  Author Profiles?
    • Are there parts we feel should be include in "out of the box" DSpace?
    • Are there parts we should encourage distributing separately as "extensions" or "add-ons"?
    • How can we work together to move this effort forward?
• Time permitting, we may also select one or more topics from the list of "Additional Topics of Interest" below.

Additional Topics of Interest (time permitting)

What would YOU like to hear about or discuss?  Please add your ideas to this list, and if we have extra time during our meeting, we will pull topics from this list to discuss.

• Creating a community for repository managers/super users to share expertise.
  • Our developer meetings and DCAT discussions are often task-oriented and future-focused (next software release).  
  • We do not currently have a forum for exploring existing system features (facets, workflow, permissions, etc) and discussing best practices around those features.
    • Comment (from Tim): There may be direct opportunities to start some of this in the design / implementation of the new User Interface. I would love to have feedback on feature best practices and UI design. This is a great opportunity to build the UI we need/want, and not just recreate what we already have.
  • A forum such as this could be a useful way to nurture future project contributors.
• The DSpace 6 Service API - What is now possible
  • The new service API introduced hibernate into DSpace.  What capabilities does this infrastructure enable (performance tuning, diagnostics, etc)?
• Metadata Authority
  • The metadatavalue table contains 2 columns: authority and confidence.  
  • What is the intended behavior of these fields?  Are these fields behaving as expected? (note: the intended behaviour is documented in the Data Model section here: Authority Control of Metadata Values#DataModel)
  • There has been some recent discussion of text vs URL's in metadata.  Could these fields provide a solution to the issues being discussed?
• Usage Statistics - how to deal with inaccuracies such as bots.
  • NOTE: For folks interested in this topic, there are a few OR16 sessions related to improving DSpace statistics:
    • Tues @ 11am: Developer Track 1: Platforms and Analytics (see specifically "I can haz robot" by Joseph Greene)
    • Thurs @ 11am: Panel 5: Usage Statistics (Includes comparisons/discussions of DSpace, Eprints and Minho statistics engines)
• Overview of the process of how new code gets into DSpace; not just the GitHub-specific process, but what discussions take place, the whole workflow, and where people can help to move code through that process.
• GPG-Key-Signing: We sign our releases but we didn't build a chain of trust yet. Let's cross sign our keys to catch up. More info here (you'll need to print something before-hand if you actually want people to be able to sign your key).

Sign Up to Attend!

If you're planning to attend this meeting, please add your name to the sign up sheet below. This will help us better cater to content to the audience type. In addition, don't forget to officially REGISTER for this workshop from the OR2016 site (as that is the official registration form)

Sign Up Sheet - Will Be Attending  (If you are only able to attend part of the meeting, please note your approximate arrival)

1. Tim Donohue
2. Maureen Walsh
3. Jonathan Markow
4. Pascal-Nicolas Becker
5. Mariya Maistrovskaya
6. Terrence W Brady
7. Susan Borda
8. Hardy Potting
9. Graham Triggs
10. Roeland Dillen
11. Andrea Schweer
12. Ivan Masár
13. Ignace Deroost
14. Bram Luyten (Atmire)
15. Andrea Bollini (4Science)
16. Michele Mennielli
17. Margaret Vail

Possibly Attending / Unsure

• (add your name here)

Meeting Notes

DSpace new UI Discussion

(Thanks to Ivan Masar, Pascal Becker and Mark Wood for taking notes. What follows is a combination of their notes from this session)

Presentation of Angular 2 UI proof-of-concept

• Slides available at http://www.slideshare.net/tdonohue/new-ui-update-or16-developer-dcat-meeting
  • How we got here is described elsewhere at: DSpace UI Prototype Challenge and YouTube playlist of video updates
• Public demo at: http://ui-prototype.atmire.com/
  • Proof of concept UI code: https://github.com/DSpace-Labs/angular2-ui-prototype
• Proof from Google Scholar that the Angular 2 UI is completely SEO compatible
• Different possibilities to render the JS on server-side. Currently using node.js, but (in the future) rendering JS via Java is promised to be an option.
• TypeScript is a typed superset of Javascript. Any Javascript is valid Typescript. Typescript gets "compiled" into Javascript so that it can run in all browsers.

Q&A on Angular 2 UI proof-of-concept

Q: Can you talk more about the backend? (Ivan)

A: Regular Angular 2 backend.

Q: Server-side Performance (Pascal)

A: While full server-side load needs to be tested, it's worth noting that server-side rendering is only used for clients with JavaScript turned off (e.g. Google Scholar, explicitly in browsers)

Q  Will this UI work with NoScript extension? (Mark)

A: Roeland uses it, works, requires JavaScript turned on. Does more than just turning off JS per domain, like XSS protection. (Roeland)  NOTE: Since OR16, Tim has tested http://ui-prototype.atmire.com/ with the Firefox NoScript extension installed. It works at a basic level, though some Bootstrap images/icons are not currently displayed.

Q: REST API completeness, missing features

A: UI developers are keeping a list. We’ll discuss whether current REST is adequate or needs a rehaul.

Q: Road map for the UI? (Monika)

A: Will be covered later in the presentation (see slides 20-23)

Q: Theming support (Ivan)

A: Bootstrap theming (there was a successful prototype of changing Bootstrap variables from admin UI). CSS (SASS). Very HTML-like templates. (Tim, Art).  In the Angular 1 protototype from the DSpace UI Prototype Challenge, dynamic theme changes were shown to be possible. This has not yet been "ported" to this Angular 2 proof-of-concept, but seems plausible to achieve.

Q: Note on Performance (Kim)

A: The frontend (Angular server-side rendering) can run on a separate machine. Possibly multiple different frontends in front of a single DSpace instance.  Even in the demo at OR16, Tim was running the Angular 2 UI off of a Windows box that had a Linux VM running DSpace 5.x REST API.

Q: Theme per collection/community/site – we should support out of the box. (Ivan)
A: Agreed.

Q: Angular components (?)

A: In Angular 2, each part of a webpage is a component (see slides 14-17). Can be moved around easily, HTML-like snippets. (Tim) Creating new components is simple (Art).  Components can also implement interfaces or extend other Components. We created our own set of Components to implement the Angular 2 proof-of-concept.

Q: Architecture diagram? What components are being used? (Pascal)

A: Tomcat, Node.js, don’t need to be on the same server.  Demo showed by Tim had an out-of-the-box (uncustomized) DSpace 5 instance running on a Linux VM (using vagrant-dspace).  It provided the REST API.  The Angular 2 app was running on the Windows host machine (i.e. a separate server), where Node.js was installed and running.

Q: What other components are there in the UI? E.g. SASS compiler?

A: Two compilation processes take place in order to create valid CSS / Javascript for all web browsers. (1) SASS->CSS, (2) TypeScript->JS, NPM (Node.js Package Manager) does that. We are using a Node implementation of SASS compiler.

Q: Next directions for JS? Is there any risk that SEO support leaves JS? (Terry)

A: The idea of Universal (aka "isomorphic") Javascript, where JS runs on both server and client,  is a very popular one in other frameworks as well. E.g. React. (Tim) First page rendering on the server really speeds up the user experience. (Art). Seems unlikely that this ability to support SEO via server-side rendering will go away anytime soon. It's one of the main advertised features on https://angular.io/features.html

Q: Browser compatibility? In relation to server-side rendering. (Kim)

A: Serverside is just rendering HTML and CSS (no JS).

Q: Backward compatibility (e.g. input form declaration file, input-form.xml) (Terry)

A: Migration or backward compatibility will be considered. To be decided, still just a POC. (Tim) New formats are an opportunity to solve some current issues. Important that we tested that we can use JSON to dynamically adapt submission form. Opens customizability options. (Andrea)  May also be possible to translate current input forms into JSON in order to allow for migration forward.

Q: Would be nice to have separation of visualization vs. validation in input forms. User can play with it since it’s on the client side. (Roeland)

Q: Angular 2 Services, DI - are they related to Java? (Ivan)

A: Services are an Angular 2 concept, underlying components, conceptually map well to Java. In Angular 2, Services are classes that "deal with data".  So, Services often take care of communication via REST, and then components render the data provided via Services. Services vs. modules allows for an abstraction layer – e.g. only services could be adapted to DSpace 7 vs. 8 REST API.

Q: Considering using other Spring components (security, auth)? (?)

A: Yes. In general, we seem to be gradually moving more toward Spring. Needs volunteers to help re-implement or migrate other parts of DSpace into Spring tools (e.g. migrating DSpace authN/Z into Spring Security).

Q: We’re behind with DSpace 6, took year and a half, what about DSpace 7? (Pascal)

A: We’ll also likely need to take a year and a half for DSpace 7, as this is a large amount of effort to rebuild the UI on Angular 2 (with an enhanced REST API). Then hopefully get back to yearly releases.

Q: List of missing features so that devs can implement them? (Mark) List of desired features? (Monika)

A: Currently have list of use cases on Wiki, will transform to list of features.  We need more lists: feature lists for UI / UX designers, lists for Java developers where the REST API needs to be extended, etc.

Q: Compatibility of the UI with older version of DSpace (5, 6) while developing it for DSpace 7

A: Will be considered, we’ll see how hard it will be as we’re adding new features.  (Tim)  We can have different builds of the UI for DSpace versions, the architecture of Angular makes it easier (Art).  However, the reality is that the new UI will depend heavily on the REST API, so backporting it would either require backporting REST API changes or backporting the UI with functionality limited to what was available in DSpace 5 or 6 REST API.

Q: Security support for DSpace going forward (Pascal)

A: Still as declared in DSpace Software Support Policy, we’ll support JSPUI and XMLUI for 3 years as declared. We'll deprecate JSPUI and XMUI in DSpace 6, remove in 7.

Q: REST development vs. UI development? (Terry)

A: Opportunity to work in parallel.

Q:  Why do people not upgrade? (Monika)

A: DSpace 5 moves the data for you (even skipping DSpace versions), huge step in ease of upgrade for instance which don’t have many code customizations. DSpace 6  has new configurations system, will allow to build migration of config options. Pascal talks about potential upgrade problems (configuration changes, Java version requirements).

A: If we say 6.0 is last stop for XMLUI and JSPUI, what if @mire wants to add new features to XMLUI? (Bram)

Q: XMLUI may be refactored as a separate project out of DSpace and available separately as an addon, it just won’t out of the box.  If others want to continue to release features for these old UIs, they can, but the older XMLUI and JSPUI will be unsupported as soon as the Angular 2 UI is released in DSpace 7.

Q: We’re deprecating old UIs now, but we don’t have the new UI ready yet. Is this what we want to do? (Graham)

A:  The goal is to go from 3 UIs to 1, less effort to support them (Mark). Cocoon is holding us back from moving the underlying Spring technologies forward. Cocoon has a max. Spring version, you can’t upgrade Cocoon beyond that. (Art). We will not be releasing any version with 3 UIs. Instead, XMLUI and JSPUI will go away in favor of the Angular 2 UI. (Tim)  Perhaps note that the XMLUI and JSPUI will be removed "when the new UI is introduced", as that leaves us more flexibility in case the development of the new UI is delayed. (Graham)

Q: Will you have to go to 6 in order to upgrade to 7?

A: No. As of DSpace 5 you can more easily "skip" releases, as the backend (database/data) will upgrade automatically.  So, you will be welcome to upgrade from DSpace 4, 5 (or older) directly to 7, as you see fit.

Q: How does the UI development work in terms of work organization (Sprints)? How much time will I need to request to take part in a sprint? (Monika)

A: There's two ways we can work in Sprints, we can chose one or both. (1) Can take Sprint as a unit of work. Can view it as task-oriented (develop a feature). (2) Or, Time-based (can dedicate a week). Multiple models possible, even in parallel.  So, may not be necessary to always request a unit of time (though you can if that works better for your work environment). Instead, you could claim a task and promise to complete it (whether that task takes a week, a few days, or just a few hours).

Q: Translation file format? (Ivan)

A: Custom JSON format in Angular 2. May have some editor for that specific format . UI devs are in touch with Angular devs, so may ask about that (ng2-translate group).

DSpace-CRIS discussion

Presentation from Andrea Bollini at Cineca

• DSpace-CRIS has attracted a lot of attention, especially in Europe
• You can always fall-back to a default DSpace installation by just turning "off" the CRIS extension
• The core of DSpace was not touched by this extension. It just adds more objects and functionality
• Began in 2009 with DSpace 1.5.  In 2012 it was open sourced (DSpace 1.8.2), and has stayed parallel to DSpace ever since
• Only supports JSPUI (but will be moving to the new UI alongside DSpace in version 7)
• Some features:
  
  • Deduplication feature coming in next version of DSpace-CRIS
  • "Most Cited" (in SCOPUS, WoS, etc). Just needs your Scopus/WoS key
  • Deep ORCID integration
  • New objects: Author, Organization, Project, etc

Q&A on DSpace-CRIS

Q: How modular is it? If I don’t want CRIS only ORCID integration (Monika)

A: Question of development priorities and cooperation. In case of ORCID, deep integration not possible because the plain DSpace data model doesn’t contain an "entity" for an Author.  DSpace does represent EPersons, but these are not the same as an Author.

Q: Is it possible to modularize DSpace-CRIS instead of having a separate software as it is now?

A: Yes, but to to so we need to agree on a data model.  We need to decide which "entities" make sense to represent for all DSpace sites (e.g. Authors) and which may only be necessary for extra CRIS-like functionality.  This may allow us to more easily switch on/off CRIS-like features as desired.

Q: Discussion point – do we want to move towards the multiple objects and relationships model in DSpace? Where does the repository stop and CRIS start? (Tim)

A: Andrea – the data model is not specific just to CRIS, any entities/relationships can be modelled.
A: Don't look for the data model, but look for the user perspective. The data model has to fit the desired use cases/features

Q: If I want to try DSpace-cris, is it possible to go back to DSpace?

A: DSpace-CRIS adds lot of tables to your DSpace database but don't alter any existent structure. The relation between DSpace items and CRIS entities are managed using the authority framework. This mean that if you switch back to the standard DSpace webapps you will be able to use all your data without need to make changes. Of course you will loose the specific functionalities provided by DSpace-CRIS.

Q: We just need an entity for a person – advisor (in order to list theses they advised) (Monika)

A: The model is very flexible (Susanne). Is it really necessary? (Monika) There could be various profiles with various entities (ETDs, scientific). You really need at least three entities even for ETDs (item, person, org. unit). These are increasingly useful also because of H2020, RIOXX requirements. (Andrea)

Q: What could be other possibility? ORCID, author profiles (Pascal)

A: What are the use-cases we need to support in dspace? Could be useful for some of the enhancements we want. If data model supports it, we could use it in a particular use case, not in another (Graham). We’ve been abusing different fields because we didn’t have a proper representation in the model. (Terry)

Q: CINECA has been involved in the new UI development, too (Tim)

A: REST is an opportunity. DSpace-CRIS 7 will definitely, absolutely not be on JSPUI. It will be on DSpace 7 UI. We want separation of layers. (Andrea)
A: New UI is a big opportunity to bring DSpace-CRIS and DSpace back together. We'll now have both building off of the same UI, etc.