Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info
titlePlanning Sprint #2 : April 20-April 24May 4-8

...

  • (15 mins) Developer Stand Up - Developers give brief updates on their effort (or their team's effort).

    • Update/see "Current Work" section below based on your status. Please feel free to update prior to meeting.
    • Please highlight any new work (needing reviews/testing), any blockers (for you), and any discussion topics you may have.
  • (25 30 mins) General Discussion Topics
    1. (REST) (beta 3) Subresources should obey access restrictions https://github.com/DSpace/DSpace/pull/2726 Restricted endpoints are sometimes the only HAL link path to public endpoints (10 mins) Tim summarizes next steps for BTE vs Live Import 
      Jira
      serverDuraSpace
      Jira
      serverDuraSpace JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyDS-4495
      )
      1. Some endpoints don't allow you to GET them, but do have publicly accessible child endpoints (e.g. registrations, resourcepolicies, …). To get to those child endpoints the UI can't discover them because the parent endpoint returns a 4xx status code. We should consider a way to prevent a regular response but allow retrieving the HAL links
    2. Finalize / approve the initial list of all authorization features which we should implement for the /api/authz/features REST endpoint.  This list of features should be limited to only features which are required to enable/disable User Interface functionality. (In other words, we can always add more features in the future.  We just need to approve the list necessary for 7.0)
      1. Review current spreadsheet (from Andrea Bollini (4Science) ) : https://docs.google.com/spreadsheets/d/1182LcD_WqIZRbUGWpLtBw0aOMR9jhbOVB7GZqtTpR9A/edit?usp=sharing 
    3. 4491

      1. BTE framework was deprecated in 6.0, with Live Import framework selected as the replacement. 
      2. As discussed previously, the "Metadata Suggestions" PR (https://github.com/DSpace/DSpace/pull/2712) includes useful enhancements to Live Import functionality (from how it existed in 6.x).  However, this doesn't quite replace existing BTE functionality.
      3. Tim Donohue's recommendation is to start from a PR that implements a more simplistic implementation of Live Import that can be used to swap out BTE more directly.  Would this be possible?
        1. The idea would be to start small here with a basic PR that disables/replaces BTE from REST API in favor of Live Import. The focus would be on replacing/refactoring the two `dspace-server-webapp` classes that touch BTE as documented in this JIRA comment
        2. We would then follow that up with additional PRs to add/enhance functionality of Live Import to better align with (other) existing features in BTE.  Including PR#2712, but also including support for additional formats/services and perhaps improved command-line support (if deemed necessary)
      4. Ben Bosman has suggested that we might consider creating a BTE "plugin" for Live Import to help with the transition.  This will allow us to build all DSpace 7 features using the Live Import framework, while (temporarily) allowing BTE features/functions to still be able to work via Live Import. (Tim notes that this should only be temporary to avoid short term feature loss.  The end goal should still be one framework.)
      5. If others feel necessary, a separate meeting specific to BTE & Live Import could be scheduled to talk through the details of this change.
    4. (10 mins) (REST) (beta 3) Subresources should obey access restrictions Topics waiting on more information (will not be discussed until additional information is available)
      (WAITING ON PROPOSAL FROM 4SCIENCE BEFORE CAN BE DISCUSSED) Revisiting "Collection dropdown in submission" issues (how should we address these?  Beta 3 has several other submission tickets, should we include work on these?)
    5. Known performance issues: https://github.com/DSpace/dspace-angular/issues/487 (NOTE: These are still reproducible on the demo site.  If you load a new submission, there's a 4-5 second pause before the page becomes usable)
    6. Changing Collection doesn't work when new collection has a different form definition: https://github.com/DSpace/dspace-angularDSpace/issuespull/2726 
      1. Tim Donohue added feedback to the PR based on testing/analysis of Spring Security annotations: https://github.com/DSpace/DSpace/pull/2726#issuecomment-622581818
    7. (10 mins) How to move forward on Controlled Vocabulary feature: https://github.com/DSpace/DSpace/pull/2743
      1. Ben Bosman added feedback on feature gaps between DSpace 6 and DSpace 7 with regards to Controlled Vocabulary. How can we work to address these gaps in upcoming PR(s)?
      2. Early notes on differences in behavior: Controlled Vocabularies and Authority Control in DSpace 7
      3. This topic may be short on time, however if someone(s) is willing to draft a proposal for addressing these gaps, we can revisit next week.
    8. Tabled Topics (will not be discussed unless time allows)
      1. Restricted endpoints are sometimes the only HAL link path to public endpoints (
        Jira
        621
        1. If I recall correctly, Collection item templates also are not working right in this scenario.
        2. And I don't recall whether previously entered metadata gets saved or cleared out when collections have different metadata requirements.
        3. Since it's possible to start a new submission via a default Collection, this increases the likelihood of encountering these issues (if the deposit starts in the wrong collection).
      2. somehow related to this discussion Creation of DSpace Entities
      3. (WAITING ON TIM TO FINALIZE DISCUSSION QUESTIONS) Replacing BTE with Live Import in 7.x:
        Jira
        serverDuraSpace JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyDS-4491
        1. BTE framework was deprecated in 6.0, with Live Import framework selected as the replacement.
        2. "Metadata Suggestions" PR (https://github.com/DSpace/DSpace/pull/2712) seems to extend Live Import functionality (from how it existed in 6.x). If this is accurate, we should track it separately so that it can have estimates & scheduling specific to it.
        3. Is there a more simplistic implementation of Live Import that can be used to swap out BTE more directly?  That would be the easiest route towards a quick swap, and then we could follow that up with additional PRs to add/enhance functionality of Live Import to better align with existing features in BTE.
    9. (Feel free to add other topics)
  • (20 mins) Planning for next week

Attendees

7.0 Release Goals

These resources define the prioritization and general schedule we are working towards

Current Work

...

titleLegend for status icons

(blue star) = Highest Priority tasks (please prioritize these reviews/tasks over others).

(error) = review done, changes were requested or bugs found.

(tick) = review done, approved.

(warning) = review done, merge conflict or other minor changes requests

      1. 4495
        )
        1. Some endpoints don't allow you to GET them, but do have publicly accessible child endpoints (e.g. registrations, resourcepolicies, …). To get to those child endpoints the UI can't discover them because the parent endpoint returns a 4xx status code. We should consider a way to prevent a regular response but allow retrieving the HAL links
      2. Finalize / approve the initial list of all authorization features which we should implement for the /api/authz/features REST endpoint.  This list of features should be limited to only features which are required to enable/disable User Interface functionality. (In other words, we can always add more features in the future.  We just need to approve the list necessary for 7.0)
        1. Review current spreadsheet (from Andrea Bollini (4Science) ) : https://docs.google.com/spreadsheets/d/1182LcD_WqIZRbUGWpLtBw0aOMR9jhbOVB7GZqtTpR9A/edit?usp=sharing 
          1. Art Lowel (Atmire) : I don't see any immediate issues with the current set of features, but I would prefer a consistent naming scheme. I'd use canDoSomething for everything
      3. PROPOSAL FROM 4SCIENCE https://wiki.lyrasis.org/x/jwUoCw
        1. Revisiting "Collection dropdown in submission" issues (how should we address these?  Beta 3 has several other submission tickets, should we include work on these?)
        2. Known performance issues: https://github.com/DSpace/dspace-angular/issues/487 (NOTE: These are still reproducible on the demo site.  If you load a new submission, there's a 4-5 second pause before the page becomes usable)
        3. Changing Collection doesn't work when new collection has a different form definition: https://github.com/DSpace/dspace-angular/issues/621
          1. If I recall correctly, Collection item templates also are not working right in this scenario.
          2. And I don't recall whether previously entered metadata gets saved or cleared out when collections have different metadata requirements.
          3. Since it's possible to start a new submission via a default Collection, this increases the likelihood of encountering these issues (if the deposit starts in the wrong collection).
        4. somehow related to this discussion Creation of DSpace Entities
  • (15 mins) Planning for next week


Attendees

7.0 Release Goals

These resources define the prioritization and general schedule we are working towards

Current Work

Panel
titleLegend for status icons

(blue star) = Highest Priority tasks (please prioritize these reviews/tasks over others).

(error) = review done, changes were requested or bugs found.

(tick) = review done, approved.

(warning) = review done, merge conflict or other minor changes requests

Status
colourBlue
title1 approval
 = pull request only requires a single approval to merge.  This is generally reserved for PRs which are either smaller, obvious, and/or bug fixes with tests to prove they work.  

Claim a Ticket!

If you do not have access in JIRA or GitHub to officially claim the ticket you wish to work on, contact Tim Donohue

PRs Needing Review

  1. (REST Contract) (low priority) related to the scripts & processes PR above (related to PR 2648 below) 

...

Claim a Ticket!

If you do not have access in JIRA or GitHub to officially claim the ticket you wish to work on, contact Tim Donohue

PRs Needing Review

  1. (REST Contract) (low priority) related to the scripts & processes PR above (related to PR 2648 below) https://github.com/DSpace/Rest7Contract/pull/99 ((warning)NEEDS UPDATES FROM Kevin Van de Velde (Atmire) ) (Andrea Bollini (4Science) - (error) REVIEWEDTim Donohue )
  2. (REST Contract) Language support on the backend (possible new cookie for Angular?https://github.com/DSpace/Rest7Contract/pull/122 (WAITING ON Andrea Bollini (4Science) ) (Tim DonohueBen Bosman, Art Lowel (Atmire) )
  3. (REST) (low priority) (beta4) Scripts & processes: importing and exporting csv's https://github.com/DSpace/DSpace/pull/2648 ((warning)WAITING ON PR UPDATES FROM Kevin Van de Velde (Atmire) ) (Andrea Bollini (4Science) - (error) REVIEWEDTim Donohue - added summary of way forward Mark H. Wood  )
  4. (REST) (tentative 7.1) [DS-4281]: Metadata suggestions in the live import https://github.com/DSpace/DSpace/pull/2712 (NEEDS TICKET / DISCUSSION) (Tim Donohue - (warning) I believe this is a new feature? We should add this to the spreadsheet, Andrea Bollini (4Science), agree with Tim it is a new feature so postpone compared to existing features)
  5. (REST) (beta 3) Subresources should obey access restrictions https://github.com/DSpace/DSpace/pull/2726 (Tim Donohue  -(warning)New feedback added, Andrea Bollini (4Science) (error) implementation approach need discussion, Craig Rosenbeck )
  6. (REST) (beta 3) Administer Workflow (Abort WorkflowItem, Delete WorkflowItem) https://github.com/DSpace/DSpaceRest7Contract/pull/272799 (Tim Donohue - REREVIEW(warning)NEEDS UPDATES FROM Kevin Van de Velde (Atmire) ) (Andrea Bollini (4Science) REREVIEW TODAY - (error) REVIEWEDTim Donohue )
  7. (REST ) Configurable whitelist for "Access-Control-Allow-Origin" header:Contract) Language support on the backend (possible new cookie for Angular?) https://github.com/DSpace/DSpaceRest7Contract/pull/2735122 (WAITING ON ON Andrea Bollini (4Science) ) (Tim Donohue) (Ben Bosman ,  Giuseppe Digilio (4Science Art Lowel (Atmire) )
  8. (REST) (beta 3) Account profile management https://low priority) (beta4) Scripts & processes: importing and exporting csv's https://github.com/DSpace/DSpace/pull/27472648 (Tim Donohue - REVIEW BY MAY 7Andrea Bollini (4Science) - REREVIEWTim Donohue - REVIEW BY MAY 7REREVIEW  )
  9. (REST) Assume login feature ) (tentative 7.1) [DS-4281]: Metadata suggestions in the live import https://github.com/DSpace/DSpace/pull/27402712 (Ben BosmanNEEDS TICKET / DISCUSSION) (Tim Donohue - REVIEW BY MAY 7 (warning) I believe this is a new feature? We should add this to the spreadsheetAndrea Bollini (4Science) - REVIEW BY MAY 7, agree with Tim it is a new feature so postpone compared to existing features)
  10. (REST) (beta 3) Controlled vocabulary (TALK NEXT WEEK ONCE WE HAVE A BUG TICKET FROM ATMIRE) Mykhaylo Boychuk  Subresources should obey access restrictions https://github.com/DSpace/DSpace/pull/27432726 (REST Contract #120) (Tim Donohue, Kevin Van de Velde (Atmire)   -(warning)New feedback added, Andrea Bollini (4Science) (error) implementation approach need discussion, Craig Rosenbeck - (warning) Reviewed and added feedback)
  11. (Angular) (beta 3) Edit resource policies REST) Configurable whitelist for "Access-Control-Allow-Origin" header: https://github.com/DSpace/dspace-angularDSpace/pull/6452735 (WAITING ON Tim DonohueArt Lowel (Atmire) -  (warning) Feedback added,  Julian Timal (eScire) (Ben Bosman , Giuseppe Digilio (4Science))
  12. (AngularREST) (beta 3) Administer Workflow  Account profile management https://github.com/DSpace/dspace-angularDSpace/pull/6502747 (Tim DonohueJulian Timal (eScire))
    1. Depends on REST PR #2727 (see above). 
    (Angular) (low priority) (beta4) Scripts & Processes Admin UI  - REREVIEW BY MAY 14 Andrea Bollini (4Science) - reviewed, looks ok suggested some cleanup)
  13. (REST) Assume login feature https://github.com/DSpace/dspace-angularDSpace/pull/6362740 (Ben BosmanTim Donohue, Giuseppe Digilio  - REREVIEW BY MAY 14,  Andrea Bollini (4Science)Craig Rosenbeck)
    1. Depends on REST PR #2648 (see above) ON HOLD
     - reviewed, looks ok suggested some cleanup)
  14. (REST) (beta 3) Controlled vocabularyMykhaylo Boychuk https:(blue star) (Angular) (EARLY beta3) Switch to Angular CLI https://github.com/DSpace/dspace-angularDSpace/pull/6252743 (REST Contract #120) (Tim Donohue - REVIEW BY MAY 7Giuseppe Digilio (4Science)(warning)REREVIEW BY MAY 7, Kevin Van de Velde (Atmire) )
  15. (Angular) Alternative links ) (beta 3) Edit resource policies https://github.com/DSpace/dspace-angular/pull/652645 (Giuseppe Digilio (4Science)Tim DonohueTim Donohue)Art Lowel (Atmire) -  (warning) Feedback added,  Julian Timal (eScire) )
  16. (Angular) Login as EPerson  (beta 3) Administer Workflow https://github.com/DSpace/dspace-angular/issuespull/653650 (Tim Donohue - REREVIEW BY MAY 7Giuseppe Digilio Julian Timal (4ScienceeScire) - REVIEW BY MAY 7 ,(warning)Julian Timal (eScire))
    1. Depends on REST PR #2740 (see above)
    (warning))
  17. (Angular) (low priority) (beta4) Scripts & Processes Admin UI (Backend) (low priority) DS-626 : Exchange usage data with IRUS https://github.com/DSpace/DSpacedspace-angular/pull/2664636 ((tick)Craig RosenbeckTim DonohueNEEDS SECOND REVIEWER)
    (Backend) (low priority) (tentative 7.2) DS-4440 GDPR - Anonymize Statistics FeatureGiuseppe Digilio (4Science)Craig Rosenbeck)
    1. Depends on REST PR #2648 (see above) ON HOLD
  18. (blue star) (Angular) (EARLY beta3) Switch to Angular CLI https://github.com/DSpace/DSpacedspace-angular/pull/2692625 (Andrea Bollini (4Science)Ben BosmanTim Donohue)
  19. (blue star) (Backend / Security) (EARLY beta3) Upgrade Spring Boot, Spring & Spring HATEOAS: https://github.com/DSpace/DSpace/pull/2720 ((tick)Andrea Bollini (4Science) ,  (tick)Craig Rosenbeck, (tick) Kevin Van de Velde (Atmire) )

PRs Coming Soon / On Hold

  1. (beta 3) REST Language Support on the backend Mykhaylo Boychuk ETA 17 or 20 April
  2. (PR from Kevin coming?) We do not fully support Community or Collection Admins adding/managing Community/Collection Groups. Related to 
    Jira
    serverDuraSpace JIRA
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
    keyDS-4486
     and this brainstorm REST Contract PR

PRs Merged this week!

  1. (tick) (first PR merged goes here)

Blocked

  1. (Blocked PRs go here)

Delayed / Needs Discussion

  1. Initial Performance Testing from Chris.  Needs revisiting / retesting prior to 7.0. 
    1. https://cwilper.github.io/dspace-perftest/
    2. These performance tests were run prior to the work on "projections" (to limit the data returned by the REST API).  Therefore, it is likely performance is much improved, but needs verification testing.
  2. (REST Contract) Edit Homepage News: https://github.com/DSpace/Rest7Contract/pull/45
    1. Delayed. General agreement (in meeting on March 21, 2019) that storing HTML in metadata fields is not really ideal behavior.  Metadata (from a librarian standpoint) tends to be free of format-related markup (as that allows for easier sharing, understanding of metadata.  Currently Community & Collection homepage information is HTML-based and is stored in metadata that is appropriate for a minor subset of information (like the title) but it is better to move large/rich text to bitstreams.  
    2. Proposal here is to consider storing HTML-based markup (for Site, Community & Collection homepages) in Bitstream(s) associated with the object in question.  May allow for more CMS-lite behavior in the future
    3. Timeline for this is uncertain.  Possibly in 7 or 8. May depend on how/whether it can be scoped.
  1. (tick) Tim DonohueGiuseppe Digilio (4Science)REREVIEW BY MAY 14)
  2. (Angular) Alternative links https://github.com/DSpace/dspace-angular/pull/652 (Giuseppe Digilio (4Science) REVIEW BY MAY 14Tim Donohue - REVIEW BY MAY 14)
  3. (Angular) Login as EPerson  https://github.com/DSpace/dspace-angular/issues/653 (Tim Donohue - REREVIEW BY MAY 14Giuseppe Digilio (4Science) - REVIEW BY MAY 14 ,(warning)Julian Timal (eScire))
    1. Depends on REST PR #2740 (see above)
  4. (Backend) (low priority) DS-626 : Exchange usage data with IRUS https://github.com/DSpace/DSpace/pull/2664 ((tick)Craig RosenbeckTim Donohue - (warning) Added feedback)
  5. (Backend) (low priority) (tentative 7.2) DS-4440 GDPR - Anonymize Statistics Feature: https://github.com/DSpace/DSpace/pull/2692 (Andrea Bollini (4Science)Ben BosmanTim Donohue)
  6. (Backend) [DS-4149] porting XOAI additional indexer https://github.com/DSpace/DSpace/pull/2756 (Paulo Graça, Andrea Bollini (4Science) )

PRs Coming Soon / On Hold

  1. (beta 3) REST Language Support on the backend Mykhaylo Boychuk ETA 17 or 20 April
  2. (PR from Kevin coming?) Support Community or Collection Admins adding/managing Community/Collection Groups. See 
    Jira
    serverDuraSpace JIRA
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
    keyDS-4486
     and this early brainstorm REST Contract PR

PRs Merged this week!

  1. (tick) (blue star) (Backend / Security) Upgrade Spring Boot, Spring & Spring HATEOAS: https://github.com/DSpace/DSpace/pull/2720
  2. (tick) (REST) (beta 3) Administer Workflow (Abort WorkflowItem, Delete WorkflowItem) https://github.com/DSpace/DSpace/pull/2727

Blocked

  1. (Blocked PRs go here)

Delayed / Needs Discussion

  1. Initial Performance Testing from Chris.  Needs revisiting / retesting prior to 7.0. 
    1. https://cwilper.github.io/dspace-perftest/
    2. These performance tests were run prior to the work on "projections" (to limit the data returned by the REST API).  Therefore, it is likely performance is much improved, but needs verification testing.
  2. (REST Contract) Edit Homepage News: https://github.com/DSpace/Rest7Contract/pull/45
    1. Delayed. General agreement (in meeting on March 21, 2019) that storing HTML in metadata fields is not really ideal behavior.  Metadata (from a librarian standpoint) tends to be free of format-related markup (as that allows for easier sharing, understanding of metadata.  Currently Community & Collection homepage information is HTML-based and is stored in metadata that is appropriate for a minor subset of information (like the title) but it is better to move large/rich text to bitstreams.  
    2. Proposal here is to consider storing HTML-based markup (for Site, Community & Collection homepages) in Bitstream(s) associated with the object in question.  May allow for more CMS-lite behavior in the future
    3. Timeline for this is uncertain.  Possibly in 7 or 8. May depend on how/whether it can be scoped.

Notes

  • Discussion Topic #1: Transitioning from BTE to Live Import
    • Decision and reasons behind it are documented in 
      Jira
      serverDuraSpace JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyDS-4491
      .  We will remove BTE and transition to Live Import.
    • Tim notes that Ben shared an idea to help transition from BTE to Live Import by creating a Live Import "plugin" for BTE.  This allows us to build all tools using Live Import, but temporarily support BTE plugins as we move to migrate them.  Tim notes this is only temporary though. End goal is one framework still.
    • Andrea notes little movement in Live Import since BTE was deprecated.  Tim agreeds, but notes that it is evidence of how complicated it is for us to maintain two frameworks. We tend to not do that well.
    • Also noted that we could adopt BTE and fix security issues.  Tim notes we haven't had a good track record of doing that. See XOAI as an example. We've done little to help that project along in any way, and still haven't even upgraded DSpace to latest version of XOAI.
    • Pascal notes that we should have a plan detailing step by step which plugins will be migrated from BTE to Live Import, when.  When does final removal occur.  Tim agrees, this is not detailed out quite yet and we should do so.
    • ACTION: All agree on a separate meeting to discuss in more detail.  Tim created a Doodle poll for next week: https://doodle.com/poll/sbavqcudm2kzt29t
  • Discussion Topic #2: Moving along Subresource access restrictions
    • Tim added feedback that Andrea & Craig agree with: https://github.com/DSpace/DSpace/pull/2726#issuecomment-622581818
    • Ideally, first improvement in that feedback would occur in the PR, to ensure some default permissions set across all new endpoints
      • ACTION: Ben will look at this and update PR
    • Second improvement (looking at "PreAuthorize" annotations for exact rights instead of assuming READ privileges) could be in a separate PR
      • If separate, a ticket needs to be created to track this work. It would also require an estimate and be scheduled for a future beta (possibly beta 4?)
      • ACTION: Ben will analyze this and get back to us.
  • Discussion Topic #3: Controlled Vocabularies and Authority Control in DSpace 7
    • Summary is on that wiki page.
    • All agree summary is accurate. 
    • Andrea feels that the side effects could be fixed.  Might be able to simply not  store an "Authority ID" for value-pairs or controlled vocabulary.  So, they still may be wrapped in Authority control, but would not see the side effects of having an Authority ID.
    • Tim asks Andrea to document a proposal on how to address these side effects & what the estimate might look like.
      • ACTION: Andrea to create a proposal / estimate for resolving these, ideally shared by Tuesday end of day.
    • Andrea notes obviously if we decide to revert to DSpace 6 behavior we'd also need to estimate that & come up with a proposal
      • Tim agrees, but one step at a time.  Leans towards keeping the new behavior if we can address the side effects. But, if not, we will need to figure out a route towards reverting to DSpace 6 behaviors.

...