2019-09-19 DSpace 7 Working Group Meeting

Date

19 Sep 2019 from 14:00-15:00 UTC

Location: https://lyrasis.zoom.us/my/dspace (Meeting ID: 502 527 3040).

• More connection options available at DSpace Meeting Room

Agenda

• (15 mins) Developer Stand Up - Developers give brief updates on their effort (or their team's effort).

  • Update/see "Current Work" section below based on your status. Please feel free to update prior to meeting.
  • Please highlight any new work (needing reviews/testing), any blockers (for you), and any discussion topics you may have.
• (30 mins) General Discussion Topics
  
  1. "Scripts & Processes" endpoint proposal / proof-of-concept (Continuation)
     1. Original REST Contract PR:  https://github.com/DSpace/Rest7Contract/pull/17
     2. Atmire implementation proposal (to refactor ~6 scripts initially for DSpace v7): https://gist.github.com/benbosman/b2388ef670ddf8b8819720b04bcebd59
• (15 mins) Planning for next week
  
  • Assigning PRs for Review
  • Next tasks from Development Planning Spreadsheet

Attendees

• Art Lowel (Atmire) 

• Andrea Bollini (4Science)  

• Tim Donohue (Out of office Sept 23-24 at 2019 DSpace North American User Group Meeting)

• Heather Greer Klein (Out of office Sept 23-24 at DSpace North American User Group)

• Lieven Droogmans 

• Mark H. Wood
• Giuseppe Digilio (4Science) 
• Ben Bosman
• Chris Wilper
• Terrence W Brady
• Paulo Graça 
• Dimitris Pierrakos 
• Julius Gruber
• Laura Henze

Current Work

Tickets / PRs In Progress

1. (Angular) Adding Accessibility via Travis CI  https://github.com/DSpace/dspace-angular/pull/356 (work in progress) (Lower priority)
2.  (Angular Bug) https://github.com/DSpace/dspace-angular/issues/368 ( Art Lowel (Atmire) )
3. (REST Contract) Edit Homepage news: https://github.com/DSpace/Rest7Contract/pull/45 (Ben Bosman  - has outstanding questions/comments) (Lower priority)
4. (REST) DS-4043: Revisit the security layer of the submission  (work in progress) Andrea Bollini (4Science)
5.  (REST Contract) Group and eperson management: https://github.com/DSpace/Rest7Contract/pull/41 (Waiting on updates fromBen Bosman )
6. (REST) Pagination bug with withdrawn items: https://github.com/DSpace/DSpace/pull/2406 (Dimitris Pierrakos , Ben Bosman - Feedback provided)

PRs Needing Review

1. (REST Contract) Scripts & Processes endpoint: https://github.com/DSpace/Rest7Contract/pull/17 Updated based on feedback during the meeting ( Andrea Bollini (4Science), Tim Donohue)
2. (REST) Issue when community has multiple dc.title values https://github.com/DSpace/DSpace/pull/2486 ( Tim Donohue , Andrea Bollini (4Science) - feedback provided. WAITING ON UPDATES FROM KEVIN)
3. (REST) Oai harvesting setup https://github.com/DSpace/DSpace/pull/2491 (Tim Donohue -  Minor improvements requested, Andrea Bollini (4Science))
4. (REST) Spring security for createAndReturn with parent id https://github.com/DSpace/DSpace/pull/2489 (Tim Donohue -  Minor improvements requested, NEEDS SECOND REVIEWER) 
5. (NEW)(REST) Endpoints to collect statistics https://github.com/DSpace/DSpace/pull/2502 (Ben Bosman , Mark H. Wood )
6. (Angular) Item-Collection Mapper:  https://github.com/DSpace/dspace-angular/pull/348 ( Tim Donohue -  REREVIEW, Art Lowel (Atmire))
7. (Angular) Shibboleth integration support: https://github.com/DSpace/dspace-angular/pull/429  (Giuseppe Digilio (4Science) reviewed again fixed error with yarn start, Fernando FCT/FCCN, Paulo Graça)
8. (Angular) Submission Miscellaneous fixes: https://github.com/DSpace/dspace-angular/pull/432 (Art Lowel (Atmire), Tim Donohue -  Noted performance issues)
9. (Angular) Redirecting user to same page after login https://github.com/DSpace/dspafixed error with yarn startce-angular/pull/467 (Art Lowel (Atmire),Giuseppe Digilio (4Science))
10. (Angular) forceBypassCache should be removed from the RequestService: https://github.com/DSpace/dspace-angular/pull/468 (Art Lowel (Atmire) - feedback provided, Giuseppe Digilio (4Science))
11. (Angular) Collection pages WIP: https://github.com/DSpace/dspace-angular/pull/472 (Art Lowel (Atmire) - feedback provided, Tim Donohue)

PRs Merged this week!

1. (Backend) Upgrade to Solr 7:  support sharded statistics https://github.com/DSpace/DSpace/pull/2495
2.  (Backend) Solr 7 fixes for upgrading to DSpace 7 https://github.com/DSpace/DSpace/pull/2393 
3. (Angular) Convert i18n files to JSON5 format https://github.com/DSpace/dspace-angular/pull/439 
4.  (Angular) Move Item Component: https://github.com/DSpace/dspace-angular/pull/335

Blocked

1. (Blocked PRs go here)

Delayed / Needs Discussion

1. (REST) Scripts & Processes endpoint: https://github.com/DSpace/Rest7Contract/pull/17
   1. How do we move this forward? In our Jan 24 meeting, Tim noted a risk of "scope creep" with this feature suggestion. This idea has been tabled since then.
   2. In July 25 meeting, Atmire said they'd come back with notes on the proposed backend implementation.
2. Managing Authorization info in Angular UI: How to pass Authorization rights (i.e. logged in user's access rights) from REST API to Angular?  See for example: https://github.com/DSpace/dspace-angular/issues/393
   1. Can this be achieved via passed HAL "_links" (e.g. the existence of an "edit" link in REST response means you must have Edit rights)?
   2. In July 25 meeting, we noted this probably cannot be resolved with just one simple solution. May need to look at different options for different scenarios
      1. Also likely to need to store/cache a user's Groups in UI layer, as some areas (e.g. Administrative) require knowledge of user group membership
3. REST API Projections:  Unable to locate Jira server for this macro. It may be due to Application Link configuration.
   1. Early work begun at https://github.com/DSpace/DSpace/pull/1847.  Discussed in more detail in our Aug 22 meeting.  Overall, this approach seems like a good direction, need volunteers to move it forward.
      1. https://github.com/DSpace/Rest7Contract/issues/2 (discussion resumed by Andrea Bollini (4Science) could be relevant for the projection)
4. Initial Performance Testing from Chris.
   1. https://cwilper.github.io/dspace-perftest/
5. (REST Contract) Edit Homepage News: https://github.com/DSpace/Rest7Contract/pull/45
   
   1. Delayed until after Preview release. General agreement (in meeting on March 21, 2019) that storing HTML in metadata fields is not really ideal behavior.  Metadata (from a librarian standpoint) tends to be free of format-related markup (as that allows for easier sharing, understanding of metadata.  Currently Community & Collection homepage information is HTML-based and is stored in metadata that is appropriate for a minor subset of information (like the title) but it is better to move large/rich text to bitstreams.  
   2. Proposal here is to consider storing HTML-based markup (for Site, Community & Collection homepages) in Bitstream(s) associated with the object in question.  May allow for more CMS-lite behavior in the future
   3. Timeline for this is uncertain.  Possibly in 7 or 8. May depend on how/whether it can be scoped.
6. Concurrency in DSpace 7 (or 8).  What do we want to do when multiple editors are editing the same object?  Needs further analysis regarding implementation details
   1. We've decided (in meeting on March 7, 2019) to use ETags to implement concurrency. REST Contract notes on ETags: https://github.com/DSpace/Rest7Contract#etags--conditional-headers
   2. ETags only update of the two fields match. If someone edits first, your edit would fail and you would get a fail response (422?)
   3. ETags seems to have broader support in other REST APIs.  Recommended also by both Art and Andrea.

7. Improve/Re-enable End To End (e2e) Testing. Could there be opportunities to use Travis CI + Docker Compose for testing of Angular?? https://github.com/DSpace/dspace-angular/issues/453#issuecomment-519672141

   • Docs: https://docs.travis-ci.com/user/docker/#using-docker-compose
   • Blogpost on how to do it: http://elliot.land/post/using-docker-compose-on-travis-ci
   • An example travis.yml file: https://github.com/Ortus-Solutions/docker-buildfiles/blob/master/.travis.example.yml

Notes

• Estimation Process Feedback: DSpace 7 Estimation Process
  • Reminder, these estimates are just to get to 7.0 BETA (not 7.0 Final).  Beta will be the first release that is "full featured" so we are essentially estimating how to make DSpace 7 "full featured".  Once we get to Beta, we will be able to estimate the timeline of Beta to Final, as it may depend on the Community Testathon (to be run once Beta is released) and the results/feedback from that.
  • The estimation spreadsheets don't include all the tasks in our Development Planning Spreadsheet
    • Tim: Correct, we are doing this estimation in stages/rounds.  This is just the first round of estimates (to get familiar with the process).  For the first round, we ONLY included tasks that were NOT flagged as "NEEDS MORE INFO" in the Planning Spreadsheet.  Essentially, we are trying to start with the tasks that seem to have the most "definition" / details.
  • How much time should we be spending to estimate each task (line in spreadsheet)?  For example, if one person spends an hour on each, they may get a more accurate / detailed estimate, but it also would take a large amount of time
    • Andrea said he's been spending about 10mins per task (on average)
    • Ben said he's been spending around the same amount of time
    • Decision: Expectation is that each task estimate takes around 5-15 mins (based on your familiarity with the task/feature).  It's possible some could take slightly longer if you need to dig into DSpace 6.x to see how the feature used to work, etc.  But, you should NOT be spending an hour or so.  You only need to write a few sentences (short paragraph) on the task, and not a very detailed design.
  • Certainty multipliers seem to act unexpectedly for a few people.  "Very Certain" adds in no buffer, and "Certain" adds in a large buffer (up to 2 times).  Should there be a level between them?
    • Reminder from Tim that this spreadsheet was not our design. It was borrowed directly from the Lullabot team (who used it for Drupal estimates), and they defined these multipliers.  Here's the article describing the spreadsheet: https://www.lullabot.com/articles/handling-uncertainty-when-estimating-software-projects
    • Tim not sure if these multipliers were created out of Lullabot team's experience, or if they came from the books that defined the "Wideband Delphi" estimation technique.  Will see if Heather knows or not.
    • We can look into whether these can be tweaked.
    • UPDATE: Mark Wood dug deeper here and noted this on Slack:
      • Re:  uncertainty factors in the Planning Spreadsheet:  to over-simplify, the folks at Lullabot adapted information from McConnell's Software Estimation, so the numbers can be traced back to actual measurements, though what happened to them after measurement is still not entirely clear.  But they weren't just snatched out of empty air.  And software development, being highly creative, is hugely uncertain at times.  As McConnell says, practical estimators are just trying to avoid being off by more than 100%.
  • Overall reminder: This is the first time we are using this process for a large open source project like DSpace.  We've only previously used these spreadsheets for grant estimations, and smaller projects.  There are opportunities to tweak these for our second round of estimates.
• Community/Collection Handles.  Do we display the Handles on the Community/Collection homepages?
  • See: https://github.com/DSpace/dspace-angular/issues/471
  • Main problem: While Communities/Collections have Handles assigned, they will not be "findable" by default in DSpace 7, as the DSpace 7 URL paths are now based on UUIDs.  So, we need a different way to let users know these handles exist
  • All seem to agree, these Handles should be listed as on Community & Collection Homepages
    • However, Initially, we may need to "hardcode" the "handle.canonical.prefix" until this configuration can be provided by the REST API: https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace/config/dspace.cfg#L227
• Revisiting Scripts & processes endpoint discussion (from last week's meeting)
  • One outstanding question, should we "merge" Curation Tasks and Scripts & Processes?  What is the difference
  • Tim: This seems out of scope for DSpace 7. May be a large task.  Also not clear if there is a complete overlap in use cases.
  • However, if we find that Curation Tasks do NOT need their own REST API endpoint, then we could simplify and let them be "kicked off" from the UI via the "Scripts & Processes" endpoint.
  • Andrea: Notes that Curation Tasks do have names & categories that are used at the UI layer to make it easier to select the one you want. 
  • Tim: True, that might imply they need their own endpoint.
  • Decision: Scripts & Processes should move forward as-is.  Need a future discussion around whether Curation Tasks need their own endpoint (or not)