- Interest in being able to handle content already in S3/Swift content
- DURACLOUD-1316
- Interest:
- Andy - definite interest, but it'll be challenging to ensure everything works.
- How do we prevent overwrites (or simultaneous writes)?
- Perhaps some type of locking mechanism?
- Could have a toggle that only allows writes via either S3 or DuraCloud API
- S3 does provide a way to send SNS notifications when content lands. Could use this to add a message to the SQS monitored by the Mill
- Fedora has a similar need to allow direct writes to OCFL then notifying Fedora
- Would need a way to ensure we get checksums properly and that the manifest/audit logs
- Top priorities
- Andy: 1258 is important, Mill using quite an old version. 1257 is essentially done.
- Danny: 1290 (docker/terraform), setting us up for long-term maintenance
- Nick: Snapshot totals (most questions), 1150, 1279, 1301
- Bill: Snapshot totals
- Plans for this week
- Andy: 1257, 1258, 1290
- Nick: 1069, 1268
- Danny: 1290
- Bill: 1292, 1278
- Github Actions
- In DuraCloud, we only merge to master on release
- The push to sonatype can push it all the way through
- Most projects tend to use tags
- Should we consider changing our branch structure? Have master being the leading edge?
- One option: develop → main and master → release
- It's helpful to have a branch where we're able to merge everything in and do testing
- Rename master to main?
- Develop: The integration point for development
- Master/Main: Merges from develop after testing (usually a release)
- Status
- Bill: 1316 and 1292
- Danny:
- Nick brought up issue on the Mill. Danny added fix PR, has been merged.
- Focused on Terraform/Docker for the Mill. It's coming along
- Plan to have one Dockerfile in the mill that will get built and pushed on release. Will include all the java jars related to the mill. A script will run the type of node that we want.
- A configuration setting (node type) will be passed in to say what that node will do
- Will also be an AWS bucket that includes all the mill configuration
- Outside of the docker container the mounting for EFS will be handled. Terraform will handle mounting EFS and set up the path mapping.
- There will be a script to push to dockerhub, creating the image with Github Actions
- Nick:
- Working on 1268, Retrieval Tool failing.
- Preferred solution: When dealing with files list, get the space manifest
- Andy
- Made a token start on updating the methods for the mill
- Topics
- Make the move from master to main - Action for Bill
- Check on whether there can be a symlink? Are these auto-forwarded?
- Github actions to push release artifacts should work off of tags (rather than merges to main)
- Status
- Danny:
- Have a docker container working for the workman piece of the Mill
- Structure in place for running the various node types
- Will continue to use cloud-init for launch
- The program will be invoked by terraform, passing in user parameters, which will generate the cloud-init scripts
- Terraform will generate the auto-scaling groups
- cloud-init has been pared down considerably (attach EFS, run docker)
- Each node type will run have a distinct cloud-init
- Within the docker container there will be scripts based on the node type
- Most of the work so far has been in the mill project, will push so Andy can take a look
- Should be able to run the Docker container independently (locally)
- Andy
- Crossed off 1257 and 1258
- Picking up 1304
- Can't make Thursday call
- Mohana focused elsewhere - not much traction on DuraCloud
- Nick
- PR for 1268 in progress (Retrieval Tool for list file)
- Probably worth adding a comment in the docs to say how it handles
- 1069 in process
- 1150 also in process - snapshot totals in the Bridge API
- Do we need to have a client parameter
- Bill
- Continued work on dependencies - 1292
- Indicates who will be taking minutes