Versions Compared

Old Version 1

changes.mady.by.user Danny Bernstein

Saved on

compared with

New Version Current

changes.mady.by.user Michael Ritter

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Danny Bernstein
  2. David Wilcox 
  3. Jared Whiklo 
  4. Peter Winckles 
  5. Daniel Lamb 
  6. Ben Pennell 
  7. Thomas Bernhart  (star)
  8. Calvin Xu Mike
  9. Ritter Michael Ritter (star)

Agenda

  1. Announcements
    1. 2021-04 Fedora 6 10 Camel Toolbox Sprint 
    2. North American User Group Debrief
  2. Pilots / Testing
  3. Performance issues needing attentionPre release short list of bug fixes and improvements
    1. Short-lived transactions related issues
      1. Jira
        serverLYRASIS JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyFCREPO-3697
      2. Jira
        serverLYRASIS JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyFCREPO-3695
      3. Jira
        serverLYRASIS JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyFCREPO-3696
    2. Migration Utils slow-down
      1. Do we know the problem is neither memory nor IO bound?
      2. Are there JVM tunings that we haven't tried?
      3. Is a heap dump likely to help us?
      4. Status of instrumenting migration-utils with micrometer:
      1. Jira
        serverLYRASIS JIRA
        serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
        keyFCREPO-
      3289
      1. 3692


  4. Pilots / Testing
  5. Pre release short list of bug fixes and improvements
    1. Jira
      serverLYRASIS JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyFCREPO-3620
    2. Jira
      serverLYRASIS JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyFCREPO-3561
    3. Jira
      serverLYRASIS JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyFCREPO-3638
    4. Jira
      serverLYRASIS JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyFCREPO-3672
    Any new tickets/issues to be considered for the release
    2. Jira
      serverLYRASIS JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyFCREPO-3690
       
    3. Jira
      serverLYRASIS JIRA
      serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
      keyFCREPO-3691
    4. others
  6. Marmotta is retired: fcrepo-camel-toolbox/fcrepo-ldpath depends on it.  What's our plan?
  7. Volunteers to be the maintainer on Your topicModeshape CI https://github.com/fcrepo-exts/modeshape/pull/3
  8. fcrepo-java-client question
    9. migration-utils
  9. Any other new tickets/issues to be considered for the release
  10. Your topicmigration util still have performance issue after tuning java heap size and turning off checksum validation

Tickets

  1. In Review

    Expand

    Jira
    serverDuraSpace JIRA
    jqlQueryfilter=13100
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  2. Please squash a bug!

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    maximumIssues20
    jqlQueryfilter=13122
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  3. Tickets resolved this week:

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    maximumIssues20
    jqlQueryfilter=13111
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  4. Tickets created this week:

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    maximumIssues20
    jqlQueryfilter=13029
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


Notes

  1. Announcements
  2. Pilots / Testing
    • Performance testing by Andrey, Peter and Jared
      • Andrey and Jared were seeing fairly bad performance
        • Andrey has performed the same tests on Fedora 4 and 5, and is seeing significantly worse performance on 6
        • Will check to see what the specs of the machine running the tests were
      • Peter and Danny have been unable to reproduce the poor performance
    • Including search index commits in same DB commit as the other db commits seems to maintain normal performance
    • Search queries are fast except when getting back RDF types. Danny is working on resolving this in a separate PR.
    • Comparing POSTs and PUTs - Jared reported that PUTs were a little bit slower
    • Single object vs Multi-object transactions
      • At the moment all transactions assume they are multi-object
      • It may reduce the database latency significantly if single object transaction commits were implemented, but at the cost of considerable complexity
      • what percentage of the latency is due to database? To determine how much of a concern this is.
    • Calvin testing migration-util with larger heap, checksum turned off
      • Initially faster, slowed back down to 25k per day after about 1.5 million objects
      • Previous runs used about 30% cpu, but not it is only using about 7%
      • It seems to still be processing image files, which are not particularly large
      • UVA originally ran their migration before alpha2, had significantly slower performance in their second migration after alpha2
    • Ben C created a ticket to look into slow resumption of migration-util, which iterates back through all the Fedora 3 objects.
  3. Pre release short list of bug fixes and improvements
  4. Any new tickets/issues to be considered for the release
  6. Performance Issues
    1. Short live transactions
      1. More of an impact on reads than writes, e.g. getting the root resource of your repository with 1m+ items
      2. Ordering adds more latency
      3. Ongoing work for short lived transactions
        1. Long lived transactions perform full join
        2. Short transactions perform a simpler query
        3. Could be expanded to writes
      4. Search Index
        1. Current work is for a synchronous update
        2. Question of how much of a performance impact there is when updating the index
        3. Possibility of adding config to update async through event bus from initial discussion
    2. Long running transactions in bad states
      1. If changes are already committed to the ocfl layer, changes can't be rolled back
      2. Might be able to attempt to rollback, then mark transaction as failed
      3. Can also prevent the transaction from being committed at all
    3. Migration utils
      1. For pid list: need to stop iterating the F3 repository once we've processed all items
      2. metrics: Try and capture bytes/sec
    Other topics
  7. fcrepo-java-client
    • Need to follow through on java 11 update and update of integration tests.
    • Thomas will take a look at the existing two tickets for updating it
  8. Modeshape
    • Danny will look at the gh actions PR
    • Will update Seth's PR to run gh actions to make sure tests pass
    • We will review the unreleased updates to modeshape since the last release was cut, to determine if we want to sync up our fork
    9. ocfl-java databasePeter will be checking on its usage of postgres for inventory caching and distributed object locking, as there are additional costs incurred by its use of separate processes per connection that he had not realized



Actions

  •  Look through migration util commits since December to see if there are any potential causes of performance drop off.