No meeting with Stanford this week. Met with ShareVDE to talk about datastores and APIs. See Q&A doc for more info and followup questions. Primary take aways...
2 tenants:
PCC
Everything else – could perhaps use provenance to pull out info for one institution
3 datastores:
Postgres DB
primary datastore into which data is ingested synchronously
data includes CKB and (I think) institutional bibliographic data as (I think) BibFrame
Solr
derived from Postgres DB asynchronously
only enough data to fulfill full text searching (I'm pretty sure)
Stardog - derived from Postgres DB asynchronously
data includes CKB and institutional bibliographic data as BibFrame (I'm pretty sure)
3 APIs
GraphQL
search Postgres for keywords using boolean search; search solr for full text search
returned data is JSON with shape based on query
REST API
same datastores searched as GraphQL
returned data is JSON with predetermined shape based on entity
SPARQL endpoints (one for PCC, one for everything else)
search Stardog
returned data is RDF with shape determined by SPARQL query
Data planned to be released in 3 stages with Stage A including Stanford & LOC
Next steps: clarifications with SVDE team, discuss with Sinopia-QA group, develop plan
Consider moving live QA instance from EBS to container version? Need to consider update mechanisms CI/CD. Agree that this is a good direction and Greg/Lynette will discuss
2021-06-25
Greg made some progress on containerization. Has a Jenkins job that can pull from github and deploy to the running service. This is key to replacing the ElasticBeanstalk version. Can now design the update process and then think about replace the beanstalk version
Lynette will be working on this next week: firm up instructions for QA servers and getting containers for Dave's work
Also need to think about LD4 conference presentation, week of July 19
2021-07-02
Lynette has successfully manually deployed to dockerhub and AWS ECR. Now working on automated build from github commit using github actions
2021-07-09
Lynette has been working with Ci/CD to build using github actions. Not complete but looking good and will be much better than Jenkins if it works
2021-06-11 Tim and Huda had a meeting to discuss priorities and approach. Main concerns were showing user reps advancement in features and . Tim working on mockups on redesign; will not implement – but will present to user reps to decide which options. Not prioritizing anything functional at-present. Will make sure pages work well when there is not enough data on the page. Will put prototype in position that we feel more comfortable when people play with it. Concerned that if show same thing at user reps, will be unproductive... but question whether anyone will remember. With more robust prototype, hope will yield more of a decision. Tim does not have much time to work on this (2.5-3 weeks) – aiming for user rep meeting in August
2021-07-02 Huda and Tim discussed work to do on DASH before user reps meeting
2021-07-09 Huda did not DASH this week but will DASH next week
Expect to include Works. Need to do something beyond what we already have live from the OCLC concordance data.
Full OCLC concordance us 343M rows, and gzipped the file is 3.3GB
SVDE Works
2021-02-26 Have to develop SPARQL queries to pull out certain sorts of connected Work. Don't expect data to be very dense but do expect that we would get useful connections between print and electronic for example. We already have a link based on the OCLC concordance file from several years ago.
ACTION - Steven Folsom and Huda Khan to work on building an equivalent of the OCLC concordance file based on SVDE data and then do a comparison to see how they are similar and different
2021-04-02 Steven and Huda met to think about putting together queries to extract a similar dataset. (Document for recording queries). Open questions about the counts – got 16k works from one view, got about 8k where limited to case with at least one instance. These numbers are much much lower than expected
2021-04-16 Steven working with Dave on how to pull our SVDE data. Dave still working through some errors in ingest of SVDE data – this needs to be resolved before looking for concordance. Has asked Frances for 2015 concordance
2021-04-23 Waiting on indexing of PCC data, have learnt more about the basis for the old OCLC concordance file
2021-05-07 Steven didn't have much luck getting data from SVDE, learning GraphQL endpoint but also problems with timeouts there (HTTP 503)
2021-06-11: At impasse. new modeling is represented in GraphQL data but fuller data are in RDF. Need to talk to SVDE when have QA/Sinopia conversation. Asked for test data but unsure when we'll have it all. Could consider doing this via Stanford Institutional data - though not ideal. ACTION: Steven will ping Anna to inquire on existing thread
What is the space of Work ids that we might use and their affordances?
OCLC Work ids, SVDE Opus (Work), LC Hubs (more than Hubs), what else?
Connections to instances, how to query, number
Other SVDE entities
2021-05-07 ACTION - Huda will reach out to Jim Hahn about entities other than Works represented in SVDE - DONE
Summarized here: Jamboard link - U Penn Enriched Marc: Work Ids in 996 Field. 1.2 million with OCLC Work IDs in > 1 description. ~3.9 million with OCLC Work IDs in only one record.
Publisher authorities/ids
At Cornell we haven't tried to connect authorities with publishes
LC working on connecting to publisher identifiers - utility is things also published by a publisher
Also possible interest in series and awards
2021-04-23 Might be able to use LC publisher ids in BANG!, Steven will look at whether there is a dump available
2021-06-04
To plan BANG! we need to think about what can be done with the available data. Perhaps take some concrete examples to consider what LC and SVDE data might give us, no longer sure what we could do with current OCLC works data (hope that entity work will provide new data later)
What about providing users with better access using alternative labels etc. that might better match their expectations, including different languages via VIAF connections. Much of our catalog data around languages is very bad because we use roman transliterations based on LC rules that are not well sync'd with actual practices in other locales.
Other possible datasets? Wikidata information is quite sparse (see jamboard). We get Syndetics ToC data for the catalog now, are there other structured data sources for ToC? Perhaps also look at wikicite – could suggest articles even if we don't generally have article level data. ACTION - Huda to ask Jesse whether there are any open structured datasets for ToC, even if much smaller.
2021-06-11
Huda asked Jesse about open structured datasets.
Huda reached out to Filip Jakobsen from Samhaeng; asked whether anything we can learn about use cases around people wanting to search across institutions to see what works exist (in ReSHARE capacity); Filip made two points: people do not benefit from looking at separate pages for Works and Instances (e.g.: conceptual distinction is not useful for users); users do not want multiple pages per institutions that has that work. If 35 instances that are same across institutions, they don't care for them to be separated. Context here is ILL – and wonder whether that would be true in local library's catalog. Filip had diagram that showed mapping b/t hubs and opi (opuses). ACTION: look at what works are and how would we map concrete examples... can you walk thru end-to-end representation of information for a few concrete examples.
2021-06-25
Huda will ask Jesse again (but after or on July 1st) about other open structured datasets for table of contents information.
Steven will look at what works data we have in the last SVDE converted dataset in DAVE for Cornell
2021-07-02
No update
2021-07-09
Beginning to form some kind of plan regarding explorations:(BANG! Data Analysis)
Working through a concrete example for LOC Hubs, ShareVDE Super works/Opuses
Wikidata properties around works: supplement what is visible/possible
Visualize or query ShareVDE PCC/Cornell (from a few years ago) relationships around different identifiers: ISBNs/LCCNs/OCLC Work IDs. (Queries still show same numbers as before so may need to revisit)
Follow up regarding OCLC Entity Backbone new Work API
Collect user stories/related work around viewing works/instances (multiple related entities, including from different institutions): ShareVDE/ReShare work
Meeting with Jesse next Monday to go over Table of Contents sources
DAG Calls
2021-06-25 Discussion on outputs from DAG calls, hope to get KP white paper completed over the summer, and then the "lord of the rings" (to bind them all) spreadsheet
2021-07-06 What to do during the conference? 7/20 canceled (because during conference)
Developing Cornell's functional requirements in order to move toward linked data
Purpose: Vision for mid-term (3-5 years) transition to support linked-data at Cornell. May include things we don't yet have or cannot yet do, but not long-term vision of post-MARC environment
Important to understand sources of truth (primary data) and where there is derivative data
Imagine landscape with items described in multiple formats including at least MARC, BF, DC (eCommons), JSTOR
Imagine all items indexed and discoverable via D&A
2021-06-25 At the moment we are working toward the notion that we need BF editing in Sinopia with FOLIO so that is a focus. Perhaps pick this up again to explore more later
Other Topics
OCLC Linked Data / Entities Advisory Group
2021-06-25 OCLC have announced that API will be open in July
PCC
2021-06-25 Task Group on Non-RDA Entities headed by standing committee on standards will formally propose a list of non-RDA entity types. Steven will join and work picking up again
2021-06-25 Planning group for data exchange met with meeting planned for 9/10 September: day 1 - foundational agreements including vendors (actors/profiles); day 2 - identify partnerships and tests (extending what is already happening as part of grant); also profiles group will pick up in July
Hope to include URIs as part of Cornell FOLIO migration, possible LD4P work
2021-06-11: Devs in FOLIO are working on MARC authority storage and basic features for maintaining authorities. mock-ups provided and have asked for feedback and test cases (positive and negative)
2021-06-25 Waiting on CUL-IT capacity, Simeon had suggested August perhaps
Upcoming meetings
https://kula.uvic.ca/index.php/kula/announcement/view/1. Call for Proposals - Special Issue: "The Metadata Issue: Metadata as Knowledge". Due January 31, 2021 (abstract 300-500 words). Includes "The use of linked open data to facilitate the interaction between metadata and bodies of knowledge" and "Cultural heritage organization (libraries, archives, galleries, and museums) and academic projects that contribute to or leverage open knowledge platforms such as Wikidata"