2020-05-18 DSpace 7 Transition BTE to Live Import Mtg

Date

18 May 2020 from 14:00-15:00 UTC

Location: https://lyrasis.zoom.us/my/dspace (Meeting ID: 502 527 3040).

• More connection options available at DSpace Meeting Room

References

• As noted in 2020-05-07 DSpace 7 Working Group Meeting, BTE will be replaced by Live Import per the ticket at  Unable to locate Jira server for this macro. It may be due to Application Link configuration.
• Currently, the two frameworks support differing sets of formats:
  • Live Import in v7 currently supports LoC Names, Sherpa RoMEO, ORCID and PubMed (via "External Sources", see https://github.com/DSpace/DSpace/blob/master/dspace/config/spring/api/external-services.xml and https://github.com/DSpace/Rest7Contract/blob/master/external-authority-sources.md)
  • BTE in v6/7 current supports ArXiv, CiNii, CrossRef, PubMed (in DSpace codebase) and OAI-PMH, BibTeX, CSV, Endnote, Excel, and RIS (in BTE codebase)
    • Bolded items are not yet supported by Live Import

Agenda

1. Discuss Strategies for replacement.
   1. (Option 1) Quick Replacement
      1. Remove BTE entirely
      2. Build up Live Import
      3. Will cause loss of functionality for JSPUI users. As noted above, there are formats BTE supports that Live Import doesn't yet support (until each can be implemented)
      4. May be less development work overall, as it doesn't involve having to write a BTE plugin for Live Import (see next option)
   2. (Option 2) Transition by temporarily wrapping BTE
      1. Create a BTE plugin for Live Import.
      2. Remove BTE from usage(s) in REST API v7, replacing with Live Import.  Live Import can temporarily use BTE to process files/services that it doesn't yet support (no immediate loss of functionality)
      3. Transition BTE "data loaders" / services into Live Import "data providers".  Perhaps migrate one format as an initial example. That would provide a reference example to other developers to migrate other formats.
      4. Remove BTE entirely once "major" formats have been migrated to Live Import. (Need to determine which formats are considered "major".)
   3. (Option 3) CHOSEN SOLUTION
      1. Deprecate BTE (no further BTE development allowed)
      2. Build up Live Import to match BTE base features (An example file importer, and an example of migrating one importer from BTE to Live Import)
         1. First two would be funded. Others may need to be volunteer-based, depends on funding available.
      3. (Based on what we learn in migrating first two plugins, we'll decide whether to create a BTE plugin for Live Import: 1-2 days of work)
      4. Remove BTE entirely
   4. (Other options?)
2. Discuss timelines for replacement
   1. (Option 1) BTE is completely removed by 7.0 Final (ideally, but will need to be finalized based on what we learn migrating the first two plugins).  Ensure all/most formats previously supported by BTE are either supported by Live Import, or scheduled for 7.1 or 7.2 (if needed).
   2. (Option 2) BTE is completely removed by 7.1 or 7.2.  Would require disabling BTE by default and placing warnings about possible security issues (especially DOS/Out-of-memory attack which seems possible in the Excel data loader)
   3. (Other options?)

Attendees

• Tim Donohue
• Andrea Bollini (4Science)
• Lieven Droogmans
• Ben Bosman
• Craig Rosenbeck

Notes

• Discussed options listed above
• Based on discussions, a third replacement option was added.
• Chosen solutions are in bold above.  Timeline is still not quite certain. Tim favors fixing this in 7.0, but we need to determine whether Live Import can sufficiently replace BTE by 7.0
  • This need not mean that Live Import will implement all plugins  by 7.0.  It may only implement the first two, with additional plugin support coming in 7.1 or 7.2, etc.