Time/Place
This meeting is a hybrid teleconference and slack chat. Anyone is welcome to join...here's the info:
- Time: 11:00am Eastern Daylight Time US (UTC-4)
- Audio/Video Conference Link: https://lyrasis.zoom.us/j/396037251
- Meeting ID: 396 037 251
- Find your local number: https://lyrasis.zoom.us/u/ad6Xb7q3ia
Join fedora-project.slack.com on the "tech" channel
- Self-register at: http://slack.fcrepo.org/
Attendees
- Danny Bernstein
- David Wilcox
- Jared Whiklo
- Peter Winckles
- Daniel Lamb
- Ben Pennell
- Thomas Bernhart
- Calvin Xu
- Michael Ritter
Agenda
- Announcements
- 2021-10 Camel Toolbox Sprint
- North American User Group Debrief
- Performance issues needing attention
- Short-lived transactions related issues
- Migration Utils slow-down
- Do we know the problem is neither memory nor IO bound?
- Are there JVM tunings that we haven't tried?
- Is a heap dump likely to help us?
- Status of instrumenting migration-utils with micrometer:
- Short-lived transactions related issues
- Pilots / Testing
- Pre release short list of bug fixes and improvements
- Marmotta is retired: fcrepo-camel-toolbox/fcrepo-ldpath depends on it. What's our plan?
- Volunteers to be the maintainer on https://github.com/fcrepo-exts/migration-utils
- Any other new tickets/issues to be considered for the release
- Your topic
Tickets
In Review
Please squash a bug!
Tickets resolved this week:
Tickets created this week:
Notes
- Announcements
- Performance Issues
- Short live transactions
- More of an impact on reads than writes, e.g. getting the root resource of your repository with 1m+ items
- Ordering adds more latency
- Ongoing work for short lived transactions
- Long lived transactions perform full join
- Short transactions perform a simpler query
- Could be expanded to writes
- Search Index
- Current work is for a synchronous update
- Question of how much of a performance impact there is when updating the index
- Possibility of adding config to update async through event bus from initial discussion
- Long running transactions in bad states
- If changes are already committed to the ocfl layer, changes can't be rolled back
- Might be able to attempt to rollback, then mark transaction as failed
- Can also prevent the transaction from being committed at all
- Migration utils
- For pid list: need to stop iterating the F3 repository once we've processed all items
- metrics: Try and capture bytes/sec
- Short live transactions
Actions
- Look through migration util commits since December to see if there are any potential causes of performance drop off.