Page History

...

• Qa Sinopia Collaboration – Support and evolve QA+cache instance for use with QA
  
  • 2021-03-05
    • Discussed the status of ShareVDE.  Dave is still working on cleaning up the data.  He has a subset that are in Fuseki for experimentation.  Action items (in order): Dave will continue to clean and index the data, Steven will compare PCC templates in Sinopia with PCC data in the cache, Steven will define the shape of data required for extended context and for a single URI dereference, Dave will create a query API based on the results of Steven's exploration, Lynette will create a QA config, Jeremy or Justin will connect the QA config to Siniopia.  At that point, it will be ready for exploring search and clone in Sinopia.
    • Dave is still working on resolving issues with the new indexing scheme.
    • Performance numbers reported in the UI are getting worse over time.  I believe this is related to timeouts of Dave's index.  I would like to revisit the statistic collection code and have it track timeouts separate so that the response stats do not include the timeouts.  This will add a column to count the number of timeouts as a separate analytic.
    • Causes of timeouts? 1) Index system down, 2) Maybe failure to find a result but not sure if Lucene still does this (was the case with SPARQL)
  • 2021-03-12: No meetings this week
• Search API Best Practices for Authoritative Data working group
  • 2021-03-12:
    • There were 53 responses.  If you only take into account the first choice, Change Management is the top choice.  If you take into account the top 2 choices, Linked Data Tooling takes the lead.  With the top 3, Language Processing just barely moves to the top.  With all 4 choices taken into account, all but moving to specifics from user stories are roughly equal.  I'm inclined to take on Change Management because it it the most straightforward and would be a quick win.  Then move to Linked Data Tooling for a 3rd charter.
    • 
• Cache Containerization Plan - Develop a sustainable solution that others can deploy
  
  • 2021-02-19 Greg completed CloudFormation template that allows someone to spin up a QA service in AWS easily. About 500 lines of template code that brings this very close to being a turnkey solution (in services-ci branch).Greg notes pre-reqs for spinning this up: S3 bucket for configs etc. which could be added to another template.
    • When complete Lynette will test, then ask Dave to test, then ask Stanford folks. Greg will also create a demo screencast.
    • What about replacing the current QA setup with this new approach? Would need to check authority configuration and correct setup for load. Lynette notes need to copy over the DB to retain history
    • Next steps
      • start to look at containerize Dave's setup. Two steps: 1) code to serve from cache, 2) indexing process
      • think about instructions for a vanilla linux server setup
  • 2021-02-26
    • Cache containerization discussion in QA-Sinopia meeting: We mostly talked about the next steps for the cache creating two containers: 1) container for API requests to retrieve cached data, 2) container to ingest data downloads and creation of the Lucene index.  This is fairly straight forward in the current approach of a full-data dump and ingest.  It is expected that there will be some complexities to resolve in how to update indices when change management techniques are deployed by authority providers that allow for incremental updates.  We punted that discussion until later when the format of change management streams is defined.  Stanford was asked their preferred deploy platform and they indicated that AWS was preferred.  
    • Greg will work with Dave when he starts work on containers and tester and sounding board
    • CloudFormation - Greg has written templates and Lynette is going to test these out (will document time taken). Hope to find anything missing in template or documentation, perhaps some permissions issues will be revealed too that will allow documentation of critical permissions
    • Next Greg will look at prerequisites that need to be set up and work to template these in a helper template
  • 2021-03-05
    • Completed prerequisites template which includes S3 bucket and EFS filesystem - next step is to document instructions and how then to move to next template
    • Greg/Lynette to coordinate Lynette's testing next week - use feedback to refine documentation
    • Then create demo screencast

...