Versions Compared

Old Version 1

changes.mady.by.user Tim Donohue

Saved on

compared with

New Version Current

changes.mady.by.user Tim Donohue

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Current Work

...

  1. (REST Contract) Collecting statistics - https://github.com/DSpace/Rest7Contract/pull/63 (Dimitris Pierrakos, Andrea Bollini (4Science) (warning) Feedback provided, Ben Bosman , (tick) Tim Donohue )
  2. (REST Contract) Update bitstream properties and format https://github.com/DSpace/Rest7Contract/pull/68 ((tick)Tim Donohue- minor improvements noted, Andrea Bollini (4Science)(warning) Feedback provided)
  3. (NEW) (REST ) Pagination bug with withdrawn items: https:Contract) DS-4317 bundles in REST https://github.com/DSpace/DSpaceRest7Contract/pull/240672 (Dimitris Pierrakos , Ben Bosman - Feedback provided(tick) Tim Donohue - minor tweaks suggested, Andrea Bollini (4Science) )
  4. (NEW) (REST Contract)  implement upload bitstream to archived item item - update https://github.com/DSpace/DSpaceRest7Contract/pull/247371 ((tick) Tim Donohue, - (warning) Feedback provided,  Andrea Bollini (4Science)- (warning) Feedback provided )
  5. (Angular) (Entities) Deleting relationshipsREST) Pagination bug with withdrawn items: https://github.com/DSpace/dspace-angularDSpace/pull/4022406 (Paulo Graça - will test again, Tim Donohue Dimitris Pierrakos , Ben Bosman - Feedback provided)
  6. (Angular) Move Item Component: REST) implement upload bitstream to archived item https://github.com/DSpace/dspace-angularDSpace/pull/3352473 (Giuseppe Digilio (tick) Tim Donohue ,  Andrea Bollini (4Science) reviewed and provided feedback, Tim Donohue )(Angular) Item-Collection Mapper- (warning) Feedback provided)
    1. Questions on Bitstream Sequence ID. MAKE READ-ONLY
  7. (NEW) (REST) Issue when community has multiple dc.title values :  https://github.com/DSpace/dspace-angularDSpace/pull/3482486 (UNBLOCKED as PR#2282 was merged.(tick) Tim Donohue NEEDS SECOND REVIEWER , Andrea Bollini (4Science) )
  8. (NEW) (Angular) Shibboleth integration support: REST) Oai harvesting setup https://github.com/DSpace/dspace-angularDSpace/pull/429  (Julius running into an error with 'yarn start' only) (Giuseppe Digilio (4Science) feedback provided, Fernando FCT/FCCN, Paulo Graça 2491 (Tim Donohue, Andrea Bollini (4Science))
  9. (Angular) Submission Miscellaneous fixes(Entities) Deleting relationships: https://github.com/DSpace/dspace-angular/pull/432402 (Art Lowel (Atmire) will re-review, Julius Gruber Paulo Graça - will test again, Tim Donohue )
  10. (Angular)  Edit current item query Move Item Component: https://github.com/DSpace/dspace-angular/pull/440335 (Ben Bosman, Tim Donohue(Giuseppe Digilio (4Science) - REREVIEW, Tim Donohue - REREVIEW)
  11. (Angular) UI Language Cookie Item-Collection Mapper:  https://github.com/DSpace/dspace-angular/pull/443348 ( Tim Donohue, Paulo Graça)(Updated) (Angular) Convert i18n files to JSON5 format https://github.com/DSpace/dspace-angular/pull/439 (Tim Donohue - REREVIEW, NEEDS SECOND REVIEWER)
  12. (New) (Angular) flatten i18n files Shibboleth integration support: https://github.com/DSpace/dspace-angular/pull/457 ((tick) Tim Donohue429  (Julius running into an error with 'yarn start' only) (Giuseppe Digilio (4Science) feedback provided, Fernando FCT/FCCN, Paulo Graça )
  13. (New) (Angular) bitstream format registries Submission Miscellaneous fixes: https://github.com/DSpace/dspace-angular/pull/455432 (Tim Donohue, (tick)Art Lowel (Atmire), Julius Gruber )
  14. (New) (Angular) Disable e2e tests in travis UI Language Cookie https://github.com/DSpace/dspace-angular/pull/454443 (CAN BE MERGED)(tick) Tim Donohue, Paulo Graça)
  15. (Angular) Convert i18n files to JSON5 format Indirect entity refs during CSV import https://github.com/DSpace/DSpacedspace-angular/pull/2471439 (Ben Bosman, Tim Donohue, NEEDS SECOND REVIEWER)

PRs Merged this week!

  1. (tick) (Angular) Fix for failing travis tests Search Performance optimizations #2 https://github.com/DSpace/dspace-angular/pull/444(tick)437 (Angular)Tim Donohue, Giuseppe Digilio (4Science))
  2. (Angular) bitstream format registries  Replace model factories https://github.com/DSpace/dspace-angular/pull/434(tick) (455 (Tim Donohue , (tick) Art Lowel (Atmire))
  3. (New) (Angular) Fix the labels on edit collection and community pages REST Contract) OAI Harvesting configuration - https://github.com/DSpace/Rest7Contractdspace-angular/pull/66459 (Giuseppe Digilio (4Science) , Tim Donohue )
  4. Indirect entity refs during CSV import (tick) (REST) Item Mapper functionality: https://github.com/DSpace/DSpace/pull/22822471 (Ben Bosman - (warning) reviewed and provided feedback, Tim Donohue )
  5. (Backend) Solr 7 fixes for upgrading to DSpace 7(tick) Minor update to Contract per this PR: https://github.com/DSpace/Rest7ContractDSpace/pull/70(tick) (Backend) Travis CI - Continue to use Ubuntu Trusty 14.04: https://github.com/DSpace/DSpace/pull/24772393 (Chris Wilper(question) , NEEDS SECOND REVIEWER) 

PRs Merged this week!

  1. (First PR merged goes here)

Blocked

  1. (Blocked PRs go here)

...

  1. (REST) Scripts & Processes endpoint: https://github.com/DSpace/Rest7Contract/pull/17
    1. How do we move this forward? In our Jan 24 meeting, Tim noted a risk of "scope creep" with this feature suggestion. This idea has been tabled since then.
    2. In July 25 meeting, Atmire said they'd come back with notes on the proposed backend implementation.
  2. Managing Authorization info in Angular UI: How to pass Authorization rights (i.e. logged in user's access rights) from REST API to Angular?  See for example: https://github.com/DSpace/dspace-angular/issues/393
    1. Can this be achieved via passed HAL "_links" (e.g. the existence of an "edit" link in REST response means you must have Edit rights)?
    2. In July 25 meeting, we noted this probably cannot be resolved with just one simple solution. May need to look at different options for different scenarios
      1. Also likely to need to store/cache a user's Groups in UI layer, as some areas (e.g. Administrative) require knowledge of user group membership
  3. Initial Performance Testing from Chris.
    1. https://cwilper.github.io/dspace-perftest/
  4. (REST Contract) Edit Homepage News: https://github.com/DSpace/Rest7Contract/pull/45
    1. Delayed until after Preview release. General agreement (in meeting on March 21, 2019) that storing HTML in metadata fields is not really ideal behavior.  Metadata (from a librarian standpoint) tends to be free of format-related markup (as that allows for easier sharing, understanding of metadata.  Currently Community & Collection homepage information is HTML-based and is stored in metadata that is appropriate for a minor subset of information (like the title) but it is better to move large/rich text to bitstreams.  
    2. Proposal here is to consider storing HTML-based markup (for Site, Community & Collection homepages) in Bitstream(s) associated with the object in question.  May allow for more CMS-lite behavior in the future
    3. Timeline for this is uncertain.  Possibly in 7 or 8. May depend on how/whether it can be scoped.
  5. Concurrency in DSpace 7 (or 8).  What do we want to do when multiple editors are editing the same object?  Needs further analysis regarding implementation details
    1. We've decided (in meeting on March 7, 2019) to use ETags to implement concurrency. REST Contract notes on ETags: https://github.com/DSpace/Rest7Contract#etags--conditional-headers
    2. ETags only update of the two fields match. If someone edits first, your edit would fail and you would get a fail response (422?)
    3. ETags seems to have broader support in other REST APIs.  Recommended also by both Art and Andrea.

  6. Not discussed much, but could Improve/Re-enable End To End (e2e) Testing. Could there be opportunities to use Travis CI + Docker Compose for testing of Angular?? https://github.com/DSpace/dspace-angular/issues/453#issuecomment-519672141

Notes

  • Lots of individuals just returning from holidays.  Welcome back!
  • Quick update on Improved Estimation Strategy for Beta release.
    • Heather Greer Klein has been working on this while Tim was out on holiday. She has some initial spreadsheets nearly ready, and Tim will be reviewing today/tomorrow
    • Anticipate that estimation spreadsheets will be ready to share either on Aug 29 or Sept 5....once ready, we'll go through them in detail and get some estimators lined up, etc.
  • Discussion: Revisiting REST API Projections.
    • Relevant tickets: 
      • Jira
        serverDuraSpace JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyDS-3533
      • Jira
        serverDuraSpace JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyDS-4306
      • Jira
        serverDuraSpace JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyDS-4307
    • Main point is to improve performance by limiting the amount of information returned via the REST API.
      • Performance will improve because REST API is gathering less information (via DB & Hibernate), and UI is retrieving/parsing less information (smaller JSON)
    • Old PR (from Tom Desair, previously of Atmire) began some Projections work based on Spring Data REST Projections
      • https://github.com/DSpace/DSpace/pull/1847
      • Basic idea is that Spring Data REST provides a `@Projections` annotation that can be used to define named projections.  In requests to the REST API, by default you still get all data back, but you can pass in a "projections=" parameter to limit the information you need based on one (or more) defined projections.
    • Will require changes both to REST API and Angular UI
      • Art Lowel (Atmire) notes that UI's current caching system may need some rethinking.  All objects currently cached in UI layer.  If some requests are only returning partial objects (e.g. metadata only), then the UI may need to "know" when it only has a partial object – so that it can make a new request for the full object when necessary.  Ideally though, we obviously should figure out the REST API implementation first, then brainstorm how this affects the UI cache.
      • Andrea Bollini (4Science) notes that Tom's initial PR had a good approach of building off Spring Data REST Projections... however , as noted in this PR comment, Andrea feels that the Projection itself should be using the "*Rest" classes, like ItemRest.class, (instead of the Hibernate classes, like Item.class), as Projections are a REST API level concept...and should therefore be implemented at that level.  Currently, it seems odd that they are bypassing the "*Rest" classes to define projections on the Hibernate classes themselves
        • Tim notes point well taken. However, it sounds like this needs further investigation / thinking on how to change the implementation to use "*Rest" classes. Perhaps we can do this in two stages?  Start with Tom's current strategy just to get an initial implementation (that the UI can start to use & implement).  Then, we can refactor to use the "*Rest" Classes once we figure out a way to do so.
        • Andrea agrees.  We should do the REST Contract first, then an initial implementation to build from.  Will need to scope the idea of Projections to make this easier to approach.
    • Scoping Projections: What types of Projections do we need?
      • Andrea notes that Projections should be based on use cases, e.g. a "list" of objects (Items, Collections, Communities) should return less information about objects in that list than an Object page would return.
      • Tim suggests we start small.  One initial projection.
      • ACTION: Start with a "list" projection that ONLY returns:  all metadata and one thumbnail (if exists)
        • Others agree.  Andrea notes this also seems like it would resolve our two outstanding bugs related to projections (DS-4306 and DS-4307, see above for links)
    • Next Steps:
      • Need a REST Contract for Projections. This is already somewhat detailed in Tom's initial PR's description...just needs to be formalized into a REST Contract
      • Initial implementation will be limited to a projection(s) for the "list" use case (all metadata + thumbnail).  This might be one "list" projection, or maybe two projections: "metadata only" projection and "thumbnail" projection.
      • Keep an eye out for other possible Projection use cases / needs...this "list" projection though seems to be the biggest need (and may provide massive performance improvements)
      • May then need to investigate using the "*Rest" classes & refactor that initial implementation (see comments from Andrea above)
    • Who will start this work?
      • Ben notes that Chris Wilper might be interested.  Will check with him though
      • ACTION: Tim Donohue will update JIRA tickets with what we decided today & then we can look to assign the ticket(s).
  • Assigning PR Reviews
    • One discussion topic came up
    • Questions on Bitstream Sequence ID in two PRs: https://github.com/DSpace/Rest7Contract/pull/68 and https://github.com/DSpace/DSpace/pull/2473
    • Should Sequence ID still exist in DSpace 7?  It seemed to be used as an alternative URL for bitstream downloads (in DSpace 6).  Bitstreams could be downloaded via sequence ID (unique in an Item) or name, or both.
    • Sequence ID is misnamed. It's not defining a sequence (ordering) in terms of Bitstreams...it's also not managed by the database layer (not a database sequence).  It's an integer defined (by the UI in DSpace 6) for each bitstream, and it must be unique within an Item.
    • Andrea notes that Sequence ID had another purpose in DSpace 6....it was a "persistent" Identifier for Bitstreams, especially since a Bitstream's name could be changed
    • Ben notes that in DSpace 7, the Bitstream's "persistent" Identifier is its UUID
    • We need to keep Sequence IDs around in DSpace 7 to allow for redirects from old (DSpace 6 style) URLs to new URLs
    • However, the way they were managed in DSpace 6 was odd.  In DSpace 7, we'll make Sequence IDs READ-ONLY at the REST API layer.
    • Need a ticket to discuss whether Sequence IDs should remain a key feature (in which case they probably should be auto-assigned by the Database or Java API layer, they currently are not), or if they are only being kept for backwards compatible URLs (redirecting pre-DSpace 7 URLs to new URLs).  Tim Donohue will create this ticket
    • POST-MEETING UPDATE: After further analysis, Tim realized that Sequence IDs are being managed by ItemService.update().  Therefore they should not  be deprecated.  Instead, Tim created a small PR to better document sequence IDs in the code / javadocs: https://github.com/DSpace/DSpace/pull/2492/
  • Support for bundles in REST (DS-4317)
    • General agreement that we need to support Bundles in REST API
    • This also means that the "/item/{uuid}/bitstreams" endpoint should go away: https://github.com/DSpace/Rest7Contract/blob/master/items.md#bitstreams
    • Should be replaced by a "/item/{uuid}/bundles" endpoint.  May also want to (eventually) think about how to use Projections to limit the Bundles/Bitstreams embedded
    • Keep in mind, we should also implement the "primary" bitstream option (supported in DSpace v6).  This allows an easier way to select the main Bitstream (and Thumbnail) for display on Item Page (or in Item lists).
    • We recommend a staged approach here.  The "/item/{uuid}/bitstreams" endpoint may exist in parallel to the new bundles endpoint until everything (including Angular) moves over to Bundles...then it should be removed in a followup PR
  • End to end (e2e) tests in Angular UI
    • Should we disable immediately per this PR? https://github.com/DSpace/dspace-angular/pull/454  They are causing issues in Travis CI.  Travis servers are taking a very long time to run these tests against the demo REST API and often timeout
    • The tests themselves are very minimal right now.  Some are not useful. The only useful ones are the Search e2e tests.
    • We all agree that e2e tests are necessary, and we'd like to have more of them.  They will find bugs that are not evident in normal tests...if Angular only tests itself and REST only tests itself, then bugs can still exist if their tests/assumptions are not completely aligned.  We need e2e tests to test Angular against the current REST responses.
    • Next Steps
      • Disable e2e tests for now (all agree)
      • Leave this e2e ticket open: https://github.com/DSpace/dspace-angular/issues/453
      • Reach out to the Docker experts (Terrence W Brady and/or Pascal-Nicolas Becker) to see if there's any interest/ideas for potentially spinning up Docker instances in Travis to run the e2e tests.
      • If we can find a better strategy for running e2e tests without reliance on the demo REST site then we can create more e2e tests to more fully test the two applications working together.
  • Security discussion on newly proposed statistics endpoints.  See this PR https://github.com/DSpace/Rest7Contract/pull/63
    • Tim noted some possible concerns about whether this endpoint should "know" whether a request came from the UI or elsewhere: https://github.com/DSpace/Rest7Contract/pull/63#issuecomment-519182491
    • Great discussion here of what is really doable. 
    • In the end, we've realized this is really really complex.  There's no straightforward way to identify a UI request versus a different REST client.  So, honestly, there's nothing we can do in the REST Contract here.  In the implementation if we notice issues here we could potentially check the Referer header as a resource to understand how the request was generated. But, even headers can be "faked" by a bot.  So, a perfect filtering mechanism doesn't exist
    • Next steps: Move forward with this REST contract & implementation as described.  This is at least as trustworthy as any web based statistics generated in DSpace v6.  If we find/discover ways to improve it, we can do that later (possible even post DSpace v7)
  • Update on Estimation process
    • Tim did a thorough review of  Development Planning Spreadsheet 
    • First column now identifies tasks that are well defined and can be immediately estimated ("Estimation") versus tasks that are not well defined yet ("Needs More Info") versus tasks that have been implemented already ("Completed"). 
      • Please note that "Completed" doesn't necessary mean the feature is 100% perfect, just that the feature exists.  If further improvements are known to be necessary, they should be added as a separate task which can then be estimated (or discussed as needed).
    • In this review, three main "big topics" were pulled out that effect a lot of other estimates and/or scoping of tasks
      • REST API Projections - affects performance of REST API, and a few outstanding features need projections to ensure they can function without slowing down the entire system
      • Authorization in Angular - currently Angular UI has hardcoded some authorizations, as we never determined exactly how to pass authorization information from REST API to Angular
      • Scripts & Processes REST endpoint - an early idea which was tabled for some time. Many administrative tools could potentially use a generic "scripts & processes" endpoint if it exists. More discussion & scoping of this idea would better define whether it should exist & based on that how administrative tools (export/import/curation) should be scoped/estimated.
    • Tim will be bringing these topics forward in upcoming meetings for revisiting.
    • Tim will also be working with Heather to start estimation process for those features which are well defined...we can begin estimations with those, and do a second round of estimations for other features (once defined/scoped)
  • NEXT MEETING
    • There is a holiday in much of Europe on Aug 15. 
    • WE WILL NEXT MEET IN TWO WEEKS, on Aug 22.  However, we will update status next week (via Slack/Wiki) so that development can proceed as normal