Original Google Docs link
KPAOW Mockups Feedback - July 29, 2019
Overview
Participants:
Total 5: 3 undergraduate students, 2 graduate or post-doc
- Law Grad student. Research assistant/reference librarian for 5 years in different libraries
- Undergraduate student. Will be Sophomore. Biological sciences major (microbiology department)
- Undergraduate student. Will be Sophomore. Industrial and labor relations
- Undergraduate student. Will be Junior. Human Development.
- Post Doctorate student. Econ and nutrition.
Process
Participants were recruited from Mann library, with the reference desk librarian volunteering and then helping request additional volunteers from the people in the library. All sessions occurred on the same day.
After explaining the voluntary nature of the session, providing a brief background on the project, and reminding them that we were testing the system and not them, I walked through the mockups using the motivating tasks and examples for those workflows. The participants provided feedback for each of the tasks
Results Summary
High level takeaways
- Knowledge panels were generally found to be useful. Design should take into account whether the amount of information may be overwhelming as well as ensure that labeling, terminology, and sections are clearly understood.
- The integration of Digital Collections was considered useful. Some participants were unclear about what the Digital Collections source was and what content it included. Design work should consider ways of clarifying what is meant by “Digital Collections”.
- Transitions between search results or terms from external sources and catalog results appeared to make sense to participants.
Overview
- Generally, information in author or subject heading knowledge panels was found useful. One participant noted that there was too much information within the Abraham Lincoln subject heading knowledge panel and that made it hard to parse or understand the information.
- Participants who were asked whether the transition between links in knowledge panels to catalog search results made sense all answered in the affirmative. For example, the transition between Vinnie Ream as a listed Digital Collections artist and search results in the catalog made sense, as well as the transition between clicking on the narrative location subject heading (in the geographic knowledge panel) and the subject (region) facet search results in the catalog also made sense.
- Digital collections results were considered useful to integrate but not all participants understood what this source meant.
- All participants noted they found the presence of digital collections results useful or interesting when viewing the suggested search/highlighted collections screen
- Not all participants were aware of digital collections resources beforehand.
- One participant noted it wasn’t clear how digital collections was a different source from the catalog while another asked what digital collections were. A third participant indicated they had no training about how to access digital collections and there didn’t seem to links from the catalog to digital collections.
- Three participants noted some confusion with the understanding the connection between the digital collection results in the subject heading knowledge panel and the artists/contributors listed below them.
- Derived/related works are useful
- All participants mentioned some version of the inclusion of derived works as useful.
- Four participants liked the inclusion of derived works on the Tale of Two cities page that showed Wikidata derived works on the right side of the page. One of these participants mentioned it was useful to see if there was a movie version of the work which may help them in case they didn’t have time to read the book.
- One participant noted they thought it was useful that the film “Lincoln” was listed as a derived work the Team of Rivals page.
- Properties/information
- Authors who also won specific awards:
- Three participants appreciated the inclusion of “awards received” but two noted they were not interested in this information.
- Derived works: Useful to have a non-book version of a work (always good to have a movie!)
- Influence/Influenced by
- Three participants noted they liked the inclusion of this information. One said it was more “persuasively” useful than intellectually. One stated it was useful but vague. None of the participants objected to the inclusion of this information.
- Two participants asked questions around how this relationship is assessed or determined with one asking for an explanation and another for sources such as interviews.
- Narrative location
- Four participants indicated they found the inclusion of narrative location interesting or useful . One didn’t necessarily find it useful but thought its usefulness would depend on context (e.g. information would be useful for someone studying literature or a specific region or culture )
- Two participants mentioned combining this information with temporal information about the work. Finding works for Paris, France may not be as useful as finding works about Paris set in the same time period as the book.
- Suggestions: Famous events occurring at the same time, books describing the region’s culture at the time
- Artists/Contributors
- Two participants indicated displaying more frequent or popular artists/contributors under the digital collection results would make more sense instead of listing the artists/contributors tied to the displayed digital collection results.
- Labeling and terminology could be improved
- One participant was confused by the term “narrative location” and suggested the term “place where story is set”.
- One participant noted “See also” under References in the subject heading was unclear and asked if it indicated more information about the subject.
- The term “author” was suggested instead of artist/contributor and three participants noted some confusion regarding either the use of the term or the connection between the digital library collection results and the artists/contributors listed below them.
- Additional suggestions
- One participant suggested the inclusion of LibGuides references along with digital collection results.
- Three participants touched on extending the geographical facet result, either by allowing for additional facets or linking to more information relevant to that time period or culture at that time period when the book was set
- Instead of “depictions” as a keyword for searching for Lincoln depictions, three participants had suggestions for alternate search strings or terms: using the search string “how as Lincoln depicted” or using the terms “pictures” or “photos” to replace the word “depiction.
- One participant suggested the inclusion of information to help with citation. (They may have meant either generating a citation of the work they’re looking at or looking for additional citations to include.)
- Google knowledge panels and carousels
- Four participants liked at least some aspects of Google knowledge panels or stated they had used the panels. Two mentioned the “people also search for” link as useful while another two mentioned they would use the panel to go to the Wikipedia source. One of these two participants stated they would use the panel to find a Wikipedia source to then find non-Wikipedia sources for use as primary sources. One participant indicated they were familiar with Google knowledge panels but found them inconsistent and not necessarily useful sources of information.
- Two participants indicated that the carousel content or ordering was confusing, with one stating they were unsure whether the carousel contained a selection or full list of related works and the other saying there didn’t seem to be any logical order to the content (chronological or alphabetical or otherwise). Another participant said they did not like how the title text was cut off in the carousel.
Mockup slides references
The following lists the mockup pages or Google searches that are referenced in each of the sections below under each participant.
- Expanded subject hierarchy and search/connection with digital library collections
- Searching for Abraham Lincoln public opinion in the subject hierarchy
- Search results for public opinion along with “Highlighted Collections (related to your search)” section with RMC Gettysburg address link and results from Digital Collections
- Loathing Lincoln
- Depictions of Lincoln/Lincoln in Stafford/Vinnie Ream
- Lincoln in the bardo
- Team of Rivals
- Tale of two cities
- Google knowledge panels/carousel
- Knowledge panels:
- Google knowledge panel for Abraham Lincoln
- Screenshot from 9/13/19 for reference
- Carousel
- Google books carousel linked off Abraham Lincoln’s knowledge panel section
- Screenshot from 9/13/19 for reference
Individual Participant Notes
Participant 1
Law Grad student. Research assistant/reference librarian for 5 years in different libraries
- Expanded subject hierarchy and search/connection with digital library collections
- When using library catalog/resources to help patrons, unsure about how to access/find digital collections resources.
- Link to digital collections is cool because biggest blind spot. I can show people summons but don’t have training on accessing digital collections. I can get people access to journals and books. It is harder through catalog to access digital collections
- Patrons not always sure about how library catalog differs from Google and that they can’t just put in keywords “and pray”
- Is the highlighted collection connected to LibGuides? Lib guide from class about Abraham Lincoln would be super helpful. People don’t know how to find out lists of things they wouldn’t have known to get
- Loathing Lincoln
- Knowledge panels are “awesome”.
- Stratified subject headings are useful as sometimes too many subject headings in one row.
- Works by and works about useful
- Alternate forms useful (especially in biological text)
- Digital collection results wonderful. Indication of how many results is useful.
- Depictions of Lincoln/Lincoln in Stafford/Vinnie Ream
- Search wording often not “lincoln depictions” but rather “how was lincoln depicted”
- People are used to “listical” search: “reasons why this” or “how this is a thing” and more of a sentence framework instead of title matching framework
- Students search thinking what is the answer to the question I have
- Digital collections
- Most popular artists/contributors probably better to include than authors/contributors for the digital collection results displayed above
- Digital collection results are not broken out by format
- Picking facets on the catalog does not translate to selecting those same facets in digital collections. If collections does not have too many items, then probably better not to apply selected catalog facets.
- Lincoln in the bardo
- Appreciate description as well as occupation info (from authorized heading)
- Influenced by: “More persuasively useful rather than intellectually useful”
- Knowing occupation may or may not influence whether I want this particular work
- Instead of using this search method of finding information about occupation (such as geophysicist) within knowledge panel details, for certain kinds of research questions would want the ability to search for geophysicist authors instead of looking for occupation on individual results.
- Links for awards/authors good to have as some people only read books that have garnered awards
- Team of Rivals
- Occupations more useful for historical books than “funsy” books to provide more credibility for author
- Derived works useful to show that non-book formats or forms of works exist
- Tale of two cities
- Not clear that the derived works are in fact films. Would not read the text (that includes “film”) but would think these are copies of the same book
- Narrative locations: Super useful for works and translations. Some translations don’t translate name of place (dependent on wikidata)
- Interested in how to get results for that place during the time period for the book’s content (i.e. when the book is set and now just where)
- E.g. Some way to add temporal info? Genre filter? Paris at 18th century
- The right side of the page seems like it’s for suggesting items (e.g. related/derived works)
- Google knowledge panels/carousel
- Knowledge panels
- Burkean parlor: What people also search for is very useful. How can we introduce people to the attendants in the parlor?
- People have been chatting in this room for ages, and specific topics have bubbled up and specific people are talking about specific topics
- Like the stratification of brief summary, quick facts, sources about/from and then people search also for. If “search also for” was first, would probably not look at the other sections
- Carousel
- Have seen with movies but not with books before
- Don’t like how title is cut off b/c care less about cover than title (title text)
- Cover is ok (cool) but hard to read
- Stinks goes to ads about buying books instead of worldcat
- Also like the section for “people also ask”
- Everytime one clicks on a question it adds more questions, generating follow up questions under people also as
- Suggestions
- Connection to LibGuides useful (e.g. lib guide about Abraham Lincon)
Participant 2
Undergraduate student. Will be Sophomore. Biological sciences major (microbiology department)
- Expanded subject hierarchy and search/connection with digital library collections
- Like addition of categories on the right
- Confused by highlighted collections: is it because we have actual documents in library
- Appreciated the results/speed
- For the heading “Highlighted Collections (related to your search)” showing the RMC and digital collection results: Like term “highlighted” but found “related to your search” confusing. Perhaps use terminology “found/location” in library “collections
- The digital collections individual result link was going to the digital collections search and not the corresponding item. The participant noted they expected the link to go the item in digital collections.
- Like the inclusion of digital collections. Having some kind of link/dropdown that goes directly to the search result.
- Loathing Lincoln
- Font-size for hovering a bit small
- A lot of info in the knowledge panel. Not sure what to click on
- What does “see also” under the Reference Info title mean? Is it also info about the president? (These are related headings)
- Digital collections: ok to have there but lots of stuff to look at
- Like the breaking out of subject headings. Not sure how to use advanced search buttons. Then can go to this page and click public opinion and search digital collections
- Depictions of Lincoln/Lincoln in Stafford/Vinnie Ream
- Knowledge panel: still a lot going on
- Artists/contributors: appreciate since I’m looking for art. Would prefer that over the other content in the knowledge panel
- Potentially confusing:Ezra Cornell mentioned in digital collection result title “Letter from Ezra Cornell” as well as in the artists lists. Which one should I click on?
- Results from collections are general and could be any format
- On Vinnie Ream catalog search page linked from knowledge panel: Like ability to bring back actual art piece in digital collections (like the right hand side result) and would probably click on the bust example immediately
- Clicking on vinnie ream in knowledge panel: makes sense it links to vinnie ream in catalog results
- Different types of links so think that makes sense
- Lincoln in the bardo
- Like the knowledge panel. Doesn’t feel as text heavy as the one about Abraham Lincoln in the previous examples. It’s just about the author. Lincoln knowledge panel: everything that pertains/is about lincoln or by lincoln.
- Like “influenced by”
- When clicking on “influenced by John Updike”, expected explanation of the relationship (the reason why we think this influence relationship exists)
- Awards received by: that link would be useful
- Knowing what this award is and who else has won this award. Appreciated b/c if were investigating this award and could potentially learn about previous awardees
- HK: Mockup showed the Atwood name (with 1939) in the authorized heading and this confused them since they were expecting the award name after having selected that
- Team of Rivals
- Very similar knowledge panel
- No influenced by relationship
- Way better than the Lincoln one
- Wish subtitles were bold. Subheading looks shade lighter than content (focusing more on content than title). It is harder to parse pieces of info and it needs a better way to distinguish between the two
- Appropriate amount of info
- Tale of two cities
- Like related works section on the right
- Narrative location
- Initially confused by narrative locations (why there are two)
- Narrative location: the term was initially confusing (wasn’t clear what it actually meant)
- “The story takes place in” - perhaps something more explicit
- Geographical popup: Wish subheadings (like subject/other works) was bolder (different sections)
- More: if expanded on page, may prefer that (but not sure)
- Generally useful to have
- Really like the other works with other narrative location
- Clicking on narrative location: gives you items related to paris/france
- Could use other formatting filters
- Connection makes sense
- Google knowledge panels/carousel
- Knowledge panels
- Usually go into the wikipedia page and then find a reference that is not wikipedia from the wikipedia page (because you need primary sources)
- If actively searching, will click on wikipedia page or go to google scholar or library website and find sources (if was just procrastinating, would just look at panel and do something else)
- Carousel
- Used it for movies but not so much books
- Ok way to quickly look at books
- Google knowledge panel on the side
- Probably wouldn’t realize that it was there, but would scroll over it and discover it
- Fine to have panel hidden on library websites because on library catalog - not really thinking of it like wikipedia-place but just looking for sources to continue that research
- Having the info there is still helpful and may redirect my research and help me find more source
- General feedback:
- Really like the highlighted section (suggested search) (in the expanded search/highlighted results from digital collections section).
- Appreciated all the info and the new sections
Participant 3
Undergraduate student. Will be Sophomore. Industrial and labor relations
- Expanded subject hierarchy and search/connection with digital library collections
- The search string for this example for Abraham Lincoln was typed in subject area format (last name, first name). Would it work with first name last name?
- Highlighted collection link to gettysburg is good too
- Digital collections: results are great and would certainly be useful
- Loathing Lincoln
- Info box: Would be looking for things for citation
- Presumably some of the info would be there
- Wouldn’t know to hover over info box to get the info
- Good it’s there
- Not sure if digital collections results are useful to me but may be useful for citation
- Probably look for the info elsewhere on the page
- Digital collections results may be useful if looking for more info/resources
- Depictions of Lincoln/Lincoln in Stafford/Vinnie Ream
- Digital collection results
- Artists/Contributors could be useful. Good especially if maybe you’re doing research
- Honestly not sure what connection there is between authors and collections? Would want it spelled it more clearly
- Ok to read generally
- Working with small space
- Q: What is the digital collections?
- How would I cite this info?
- Useful to have that info on the right (digital collections result) - may be useful for someone
- Vinnie Ream: Don’t know who vinnie ream is
- Link in info box goes to results -that’s fine
- Lincoln in the bardo
- Info box uses a lot of synonyms: multiple versions of “teacher” useful when writing a paper
- Awards and influenced by: fills the quota
- Not sure awards received would be useful
- Influenced by is pretty cool
- You’d have to directly know the influences to know if it really helps
- Not sure if that is useful - is there a real possibility that this is miscellaneous?
- How do you determine what influenced by means?
- Navigation is ok
- Would click that only if struggling my resources (HK: Assume from notes and context, this means clicking on the names listed)
- “Won’t just start clicking on names randomly”
- “More info is always better than less, especially on a library catalog”
- Already established awards kind of useless but if it were useful then the link to books by authors who have won that award would make sense
- Team of Rivals
- For the author knowledge panel in this example, now sort of miss “influenced by”. Wouldn’t use it but appreciate it being there
- Prefer the previous knowledge panel example with George Saunders because it had more info
- Tale of two cities
- Narrative location: great to have!! Helps me!
- Some stories are ambiguous but this could save me like time
- Geography knowledge panel:
- Helps to provide context - no embarrassing contradictions
- Paris, France connection (link that searches catalog)
- Transitions well
- Goes out to broad scope
- Derived works:
- Analysis to reference
- Movies/books: useful info
- Related TO tale of two cities: just click related resources which we already have
- Google knowledge panels/carousel
- Knowledge panels
- Very inconsistent. Have seen this my entire life so am familiar with it. It’s great but some of this is pretty irrelevant.
- Don’t care about any of these books
- Means nothing to me. I would just click the wiki page itself (although know the info box is the information from the wiki page)
- If I wanted an age for celebrity, just type in Celebrity age
- Carousel
- Book carousel: Why THIS order? Some old ones first? Old then new?
- Did people only write about lincoln a long time ago and not anymore
- More recent selections ?
- HK: Participant showed me a carousel example on their phone for classic hip hop
- Classic hip hop: the results are presented in the completely wrong order
- Not chronological, not alphabetical - no logical order
- Assume that the books are going to be like that
Participant 4
Undergraduate student. Will be Junior. Human Development.
- Expanded subject hierarchy and search/connection with digital library collections
- Why is the suggested search section on the right instead of up top (like advanced search). I thought it was an ad on the right.
- Didn’t pay attention to it at first. I don’t generally pay attention to what is on the right side of the screen. The section is gray and that may also make it less noticeable.
- Normally would look under the top search or where the facets are.
- Loathing Lincoln
- The “i” would be sufficient to say something there to hover over
- Info in info box looks fine
- Digital collections results: clicking and going to that item does make sense
- Having those results is helpful
- Depictions of Lincoln/Lincoln in Stafford/Vinnie Ream
- Search term to use may be “photos of abraham lincoln”
- Perhaps looking at sculptures
- Hover pop up
- Artists/Contributors: What did they contribute to? To lincoln?
- Confusing to figure out the connection
- Make the connection clearer between digital collection results and artists/contributors
- Would be better to see more frequent authors/contributors
- Transition from info box to Vinnie Ream results is ok. Having the info about digital collections related to vinnie ream is also ok
- Not a fan of the gray section on the right (when looking at Vinnie Ream digital collection results after transitioning to catalog results for Vinnie Ream)
- Lincoln in the bardo
- Like George Saunders pop over
- Looks like a wikipedia page (that’s helpful)
- Would want to see more works by him in a similar way to how IMDB displays information such as “other things you may know him from”
- Really helpful breakdown
- Influenced by/Awards: both make sense
- Influenced by: is helpful
- How do you know that they are influenced by? Interviews?
- Perhaps include some info indicating there was an interview or something which lead to this relationship being defined
- Awards: good for credibility/credentials
- Team of Rivals
- Author info box cool but less informative than page
- Tale of two cities
- Related works: helpful
- Other places with the same narrative location: not helpful
- Like the info/context around region
- Maybe famous events that happened at that time? What culture of the region was at the time?
- Results for region: could know more about region - find more works - makes sense
- Google knowledge panels/carousel
- Knowledge panels
- Always use this. Awesome. Really like it
- Look up somebody, read the info/context, skim through books. If interested in them, just focus on them. Click on wikipedia link or go to other site
- The “people also search for” links are helpful to find related people
- Carousel
- Ok way to navigate? Probably not
- Never looked all through the books, but just skimmed
- If I wanted a particular book, I would just type in the name of book
Participant 5
Post Doctorate student. Economics and nutrition.
- Expanded subject hierarchy and search/connection with digital library collections
- Suggested search part was a little confusing. Should I click on the search area (i.e. the input box) or the hierarchy (of subject areas) that is below the search box? The interaction between search and drop down wasn’t clear.
- Overall concept makes sense
- It is ok that suggested search is on the right side
- After selection of subject area, facet confirmation on the top is useful
- Digital collection results: probably would be useful. I want to go through everything available
- Loathing Lincoln
- Info box
- If I were interested in book, I would just request the item
- If somehow a little lost, would maybe look at the info box
- Info in info box is useful
- If looking for other items, may click on “works by” and “works about”
- Wouldn’t directly understand what the difference would be between the digital collections section and “the works about”?
- If interested in finding specific information, if looking at a topic, having access to all sources is super useful
- Not clear how digital collections is a different source. I haven’t used it and do not know what it means
- Depictions of Lincoln/Lincoln in Stafford/Vinnie Ream
- Don’t generally look for pictures and search libraries for artists
- May not use the word “depiction” but “pictures” perhaps
- Subject info box
- Not super clear what the distinction between authors/contributors of these pictures or titles
- Also terminology - titles of books/articles
- If it was a book, you’d use the term “author”. The terms “artists/contributors” are strange
- Clicking on vinnie ream and seeing results: that transition does make sense
- Lincoln in the bardo
- Think the George Saunders info box is useful
- If interested in book, want a clear picture of book and how to get it
- Influenced by: John Updike
- Useful but a bit vague
- Transition is clear between terms returned to search results
- Important to have the info for what you searched for (e.g. john updike author facet)
- Info box popup clear (you can hover and see the information)
- Awards received: May not be interested in
- May be confusing if browsing since links look like they could it take you to different places, perhaps to wikidata.
- HK: Unclear what section this refers to
- Having just the author name makes things unclear.
- Team of Rivals
- Author info box is good
- Listing information about derived film is interesting
- Tale of two cities
- Related works: super useful
- May not have time to read the book and could then watch the movie
- Narrative location
- Seems like extra information but doesn’t bother me too much because not that much information
- If looking to read dickens, wouldn’t care about what novels are set in Paris or London
- Depends on what you are looking for
- For someone studying literature or studying specific region or culture, may be super useful
- Transition from subject to region search makes sense
- Perhaps more useful for more exotic locations
- Google knowledge panels/carousel
- Knowledge panels
- Use it and think it’s informative.
- If not sure what you’re looking up, knowledge panel gives a good high level overview
- Even if I’m not clicking on the knowledge panel, if don’t know much about the topic, it is good to have that summary
- Carousel
- The carousel list is confusing since unclear what the list contains?
- Is this a selection or full list?
- Is it most read? Most recent? Everything?
- If looking at a specific item, then it’s useful, but if looking at whole selection of items then I would need to look for another way to find that