SMASH! Usability Testing Results

Link to original doc

Huda Khan, Astrid Usong
March 2020

GOAL

Understand how suggestions in different formats (where they are or could use semantically modeled information) could be more useful to students conducting research.

METHODOLOGY

Huda sat in the Mann Library on the first floor with a sign offering protein and candy bars for 10-15 minutes of time and asked if students wished to participate. She also asked if student workers who are helping a colleague would be willing to spend 10-15 minutes in a usability study.   She interviewed 5 students over the course of 2 days.

PARTICIPANTS

5 users: 4 undergraduate students and 2 graduate students

SCHEDULE

March 4, 5

KEY LEARNINGS

• Auto-suggest is a useful feature which could benefit from additional support for misspellings.  Furthermore, the label “author” could be revisited to clarify what the list indicates (i.e. works by versus works about). Potential exploration could be undertaken around how to provide a mechanism to distinguish between results (especially if they have the same string value) to enable users to know which result to select.  
• All participants were able to identify subjects within auto-suggest.  
• The other auto-suggest task asked them to find people, and two participants needed clarification that the term “author” in the results still indicated people.  One participant said they were looking for books about a person and not by a person, so the term “author” was confusing for them.  
• When asked, three participants stated they would find auto-suggest useful.  One participant said they usually do not care who the authors are unless they happen to be professors.  They also said there wasn’t enough information in the list to know which one to pick. They also said subject suggestions would be useful if you knew which person you were looking for. 
• Related person and subject suggestions have the potential to be useful, but additional steps could be taken to clarify how these entities are related as well more clearly lay out the knowledge panel design to display supplemental information such as contemporaries for authors.  Furthermore, labeling could be improved for related concepts. For example, we could use the term “similar” instead of “close” for closely related concepts. Additionally, clarifying that the suggestions are clickable could be clearly indicated using a pointer above the item. 
• The task asked participants to find related subjects.  Three participants appeared to have trouble with the task wording and started to review search results.  For these participants, rewording the task to use the word “concept” instead of subject seemed to help. I had to tell one of these participants to look at the left (and they said they don’t normally look at the left sidebar section of the page where facets reside).  
• Three participants found broader/narrower subjects without additional help.  One needed clarification on the terms “broader” and “narrower” and could then find these related subjects, while another had to be told the broader and narrower subjects were within the subject knowledge panel.  Three participants noted that the labels for related subjects could be clearer. Three participants suggested the term “similar” be used instead of “close”. Suggestions for “close” also included “related” or “people also look for”. One participant suggested “similar” be used instead of “broader”.
• All participants found a related person. None of the participants saw the contemporaries listed within the main knowledge panel.  Four looked at the list of person suggestions under the knowledge panel to see if they could find contemporaries. Some participants saw the scrollbar while others did not, but none expected to find contemporaries listed within the knowledge panel.  One participant who saw the scroll bar stated they thought perhaps there was additional biographical information within the knowledge panel.  
• One participant noted that hovering over the suggestions list didn’t convert the arrow into a pointer which would have indicated they could click on the item. 
• The participants indicated they thought these features would be useful in some contexts or after adding some qualifications.  One participant noted that it was generally useful to find related concepts but finding related people (or perhaps contemporaries) was not useful as they don’t usually study time periods in their domain.  A different participant answered “maybe” when asked if they would find these features useful, noting they look for the most recently produced content in their domain. Another participant said they don’t generally use the catalog for open-ended search.  Another pointed out the need to identify and more clearly explain how these people or subjects were related. When asked if they would find these features useful if the results were relevant in the way they desired, they replied in the affirmative and said they don’t always know where to go after their initial search.  
• Most participants indicated zero-results suggestions were a useful feature.  Some participants indicated that the highlighted snippets provided useful context around how these suggestions related to the person’s query.  Additional labeling changes could be made to clarify that the subjects on the left are suggestions and that the results displayed are related searches.  
• All participants were able to find subjects on the page as well identify which search result came from HathiTrust.  
• Four participants noted that having zero results is better than having no results whatsoever or looking at a blank page.  
• Two mentioned Google features such as “did you mean this?” or “Searched instead for”.  One noted it would be useful to have help with typing errors or misspellings. Two participants noted the highlighted or bolded parts of the search result that showed snippets matching the user’s query were helpful.  
• One participant suggested using the term “subject suggestions” instead of “subject”.  One participant also suggested adding a heading that said the results were related searches instead of “zero results”.  

TASKS AND SCREENSHOT REFERENCES

This document outlines the tasks which are also repeated below along with relevant screenshots

.

Auto-suggest

• You are researching books and resources related to Albert Einstein.  You open up the catalog. Please search for “Albert Einstein”.
• Can you identify one person that matches your query?
• Can you identify a subject that matches your query?

Subject and person suggestions

• You are exploring Viennese architecture and are interested in seeing what topics and people may be related to this subject.  Please search for “Vienna architecture”.  
• Select a related subject.  
• For your selected subject, are there broader, narrower, or closely matching subjects? If so, could you please list them?
• 

• You are researching Charles Dickens.  Please search for “Charles dickens”.
• Select a related person. 
• Are there contemporary authors for this person? If so, please list one.

Zero-results page

• You are researching “Akestor, King of Paphos”.  Please search the catalog with this query.  
• Can you identify a subject related to this search?
• Can you select one of the results from HathiTrust?

Follow up questions (or similar) for each section

• Which of these tasks do you currently do when researching or looking for resources?  Which of these tasks would you not do? Would you please explain why?
• Would any of these features be useful to you in how you search or do research?  

Notes

Participant 1

Grad student, master of public administration 

Auto-suggest

• You are researching books and resources related to Albert Einstein.  You open up the catalog. Please search for “Albert Einstein”.
• Can you identify one person that matches your query?
• Can you identify a subject that matches your query?

Hit enter so had to pause (stop the results from loading)

Identified person and subject suggestions easily

Places that are named after Einstein: looked like clutter

But organized so that could easily tell what the person results were. 

Subject and person suggestions

• You are exploring Viennese architecture and are interested in seeing what topics and people may be related to this subject.  Please search for “Vienna architecture”.  
• Select a related subject.  
• Found related subject (city planning)
• For your selected subject, are there broader, narrower, or closely matching subjects? If so, could you please list them?
• Could see broader, narrower, close 
• Urban policies
• Did you like feature?
• Wouldn’t have thought to look at broader/narrower but saw “policy” listed so thought would click on urban policy
• Clicked 
• You are researching Charles Dickens.  Please search for “Charles dickens”.
• Find a related person suggestion.
• (Asked them to find the 1812 one and they did that pretty easily)  
• Found on the left
• Find one contemporary author for Charles Dickens.
• Looked in list to find contemporaries (i.e. other results)
• Didn’t see contemporaries in knowledge panel
• Would you find this interesting?
• For field, contemporaries less interesting b/c not studying a period
• Perhaps if studying literature
• Finding related people subject useful?
• Yes. 
• If literature you like and enjoy from a time period
• Generally useful to see related concepts

Zero-results page

• You are researching “Akestor, King of Paphos”.  Please search the catalog with this query.  
• Can you identify a subject related to this search?
• Yes
• Can you select one of the results from HathiTrust?
• Could see this cleary

Follow up questions (or similar) for each section

• Which of these tasks do you currently do when researching or looking for resources?  Which of these tasks would you not do? Would you please explain why?
• Usually go to databases (from library.cornell.edu)
• Probably never used newcatalog library catalog
• Example: euromonitor -> passport -> see catalog record for database
• Science director - go through cornell portal b/c don’t get full article even with cornell email
• In Google, rarely use info boxes 
• Would any of these features be useful to you in how you search or do research?  
• Often go to Google first
• When specific title, then would go to Cornell catalog
• If more open-ended, perhaps useful but very used to Google
• Have also used chat with librarian
• Do you have suggestions/recommendations for any different approaches?
• No, not really

Participant 2

Undergrad student, agricultural sciences, senior

Auto-suggest

• You are researching books and resources related to Albert Einstein.  You open up the catalog. Please search for “Albert Einstein”.
• Can you identify one person that matches your query?
• Yes
• Can you identify a subject that matches your query?
• Yes (had to scroll down that)
• Would this feature be useful?
• Don’t really know much about the author. Everyone knows Einstein.  Usually when looking for something, don’t really care about author.  
• If he or she is professor - would care
• Wouldn’t know which one is the right person to pick (just from names)
• Subject: if know person, “fiction” “influence” would be useful
• Looking at the numbers to see what might be relevant (the ones at the bottom have small numbers so wouldn’t want to use that)

Subject and person suggestions

• You are exploring Viennese architecture and are interested in seeing what topics and people may be related to this subject.  Please search for “Vienna architecture”.  
• Select a related subject.  
• Scrolled up and down a bit without seeing.  Saw subject facets below
• Didn’t see related subject searches headline at first
• For your selected subject, are there broader, narrower, or closely matching subjects? If so, could you please list them?
• For city planning, did see them (but needed to have the question redone)
• Not sure what close and broader
• “Similar” as label more useful
• You are researching Charles Dickens.  Please search for “Charles dickens”.
• Select a related person. 
• Did find it
• Are there contemporary authors for this person? If so, please list one.
• Looked in list below
• Had to be told to scroll down (didn’t really notice scroll bar before)
• Might be helpful if the box containing knowledge panel was bigger to allow for seeing contemporaries 
• Would this feature be generally useful for bringing in related subjects and people?
• Maybe.  Looking for newer data so don’t really look for

Zero-results page

• You are researching “Akestor, King of Paphos”.  Please search the catalog with this query.  
• Can you identify a subject related to this search?
• Saw related subjects
• Can you select one of the results from HathiTrust?
• Took a few seconds but identified
• Would something like this feature be helpful?
• Suggestions would be helpful
• Make a lot of typing mistakes
• Like Google’s “did you mean this”

Follow up questions (or similar) for each section

• Which of these tasks do you currently do when researching or looking for resources?  Which of these tasks would you not do? Would you please explain why?
• These are not the topics usually search for
• Prefer newer data
• Science changes
• Don’t really want something from 1812
• Example “plant breeding” is a term you would use
• If people and subjects were current, that would be useful
• Yes/maybe for the tasks
• Peer-reviewed articles
• Google scholar is main venue, sometimes use library resources
• Cornell library
• Would any of these features be useful to you in how you search or do research?  
• When translated to domain, it would be hard to tell if useful

Participant 3

Undergrad, applied economics and management, junior

Auto-suggest

• You are researching books and resources related to Albert Einstein.  You open up the catalog. Please search for “Albert Einstein”.
• Can you identify one person that matches your query?
• b/c it said author, wanted to go to subject instead since I asked for person
• Can you identify a subject that matches your query?
• Found subject
• Initially thought would go to autobiography page
• But number = pages about him
• Once saw the catalog search, could see why subject search
• Was thinking in terms of Google (section that describes the person)
• Would you find useful?
• MAkes it faster b/c can quickly see exactly what you’re looking for

Subject and person suggestions

• You are exploring Viennese architecture and are interested in seeing what topics and people may be related to this subject.  Please search for “Vienna architecture”.  
• Select a related subject.  
• Scrolled down first
• Past facets
• Saw items that seemed to be about the subject 
• After explaining concepts, could see related subjects on the left (wouldn’t have seen unless I said look on the left b/c usually doesn’t use facets, class assign specific readings, direct link, or buy it in harvard business course pack)
• Don’t use the catalog for more open-ended tasks
• [click arrow doesn’t turn into pointer so can’t tell if clickable)
• For your selected subject, are there broader, narrower, or closely matching subjects? If so, could you please list them?
• Broader, right there
• Close: some could be considered narrower 
• Easy to understand
• Would it be useful to have this feature?
• Yes, would be useful, it would help narrow down search - if blank page on essay page, type in phrase and then delve into related topics
• You are researching Charles Dickens.  Please search for “Charles dickens”.
• Select a related person. 
• Found the box and person
• Are there contemporary authors for this person? If so, please list one.
• Feel like it would say it in the box
• Did find a box 
• Didn’t even see the scroll bar to the right
• Generally, finding related people and subjects fairly useful
• Google: definitely use the info boxes 
• Would use info box on the right, sometimes have picture, and would use dictionary definition. But definitions tend to be vague so have to scroll down the page to get more context

Zero-results page

• You are researching “Akestor, King of Paphos”.  Please search the catalog with this query.  
• Can you identify a subject related to this search?
• Could identify - would have picked Greek  mythology based on the search results
• Can you select one of the results from HathiTrust?
• Can find that immediately
• Definitely consider having options useful (instead of no options at all)
• Anything else you might want to see on this page?
• On Google, may be incorrect or clear and it will say “didn’t search what you typed in but searched for this instead”
• Sometimes not clear when researching topics so if computer could help you clarify what topic or subject you’re looking for more exactly

Follow up questions (or similar) for each section

• Which of these tasks do you currently do when researching or looking for resources?  Which of these tasks would you not do? Would you please explain why?
• Mixture of first two activities
• Basic topic
• Looking for person - if you find people who come up with ideas, solidifies evidence
• Never thought of catalog having people
• Would any of these features be useful to you in how you search or do research?  
• If in catalog, all features would be useful
• Researching phase very long and strenuous and this could help cut that down and remove stress from doing research

Participant 4

Undergrad, senior, communications

Auto-suggest

• You are researching books and resources related to Albert Einstein.  You open up the catalog. Please search for “Albert Einstein”.
• Can you identify one person that matches your query?
• Yes, under authors.  And because dates assume birth and death dates
• Can you identify a subject that matches your query?
• Same person twice but different number of results
• 409 vs 393
• Would you find this feature useful?
• Like how separated by author and subject
• If need certain author, could just find them or if looking for a subject, could narrow it down
• Q: What do you think the numbers mean?
• Take me to results page and how many results listed = number of actual results

Subject and person suggestions

• You are exploring Viennese architecture and are interested in seeing what topics and people may be related to this subject.  Please search for “Vienna architecture”.  
• Select a related subject. 
• Particular time frame?
• If so going to keep searching since some of the results have different dates, first item is contemporary
• Did see the related subject box but didn’t think that was what I was asking for 
• [tester: when I said concepts instead of subjects , may have been closer)
• For your selected subject, are there broader, narrower, or closely matching subjects? If so, could you please list them?
• First went to search results
• Then started looking at list of subject searches
• Had to be told that the info was in the box
• Instead of “close” using “related” or “similar” or “people also looked at”
• Not as intuitive that these are related when looking at it the first time 
• Subject: Buildings, and Close: architecture
• “Broader” still make sense as a term
• You are researching Charles Dickens.  Please search for “Charles dickens”.
• Select a related person. 
• Saw Charles Dickens (first result)
• Are there contemporary authors for this person? If so, please list one.
• Looking at list of people in the suggestions (perhaps in the list there)
• Then found contemporaries list in the main knowledge panel
• Scrollbar was obvious in the knowledge panel but not clear what to expect (thought perhaps biography or something) [Really didn’t expect that info to be in the panel]
• The photo reminded of Wikipedia page with photo followed by descriptive info
• Is this feature useful?
• saves me time from having to look through the search results to pull out similar people or subject
• Or having to do a google search to look for similar subjects and authors

Zero-results page

• You are researching “Akestor, King of Paphos”.  Please search the catalog with this query.  
• Can you identify a subject related to this search?
• Want to say the subjects on the left could help narrow the search
• Can you select one of the results from HathiTrust?
• Found that pretty easily
• IS this feature useful?
• Yes, better than a blank page saying your search turned up nothing.
• Q: Would you like any other info to be on this page?
• Think this is fine. 
• Having the search words be highlighted in the bold in the search snippets is useful
• Instead of subjects on the left, have it say “subject suggestions”

Follow up questions (or similar) for each section

• Which of these tasks do you currently do when researching or looking for resources?  Which of these tasks would you not do? Would you please explain why?
• For some of my classes, do use the library catalog quite a bit because no textbook.   
• Given list of articles, research papers, book excerpts (set number of pages from book)
• When trying to find papers from conferences -> For conference, have to google conference terms - the broader concept - and then from there, can narrow it down
• Or use researcher’s name which helps find paper more quickly
• If need a book that don’t want to physically check out look in library to see if can be read online
• Would any of these features be useful to you in how you search or do research?  
• Yes. Like these ideas.
• Would save people time and frustration from having to comb through 100s of results

Participant 5

Undergrad, information science and fashion design, junior

Auto-suggest

• You are researching books and resources related to Albert Einstein.  You open up the catalog. Please search for “Albert Einstein”.
• Can you identify one person that matches your query?
• “What do you mean?” 
• Like author? Yes.
• Confusing that you were looking up a person but saw author, looking for book about author and not by them
• One of the results is a college, another can’t quite pronounce - they don’t like people, they look like an organization
• Can you identify a subject that matches your query?
• Find subjects in the list (yes)
• Q: subject make sense?
• Would guess Albert Einstein is subject of book
• The first two subjects are same, so confusing
• Numbers: how many search results
• Would feature be useful?
• Suggestions are useful.  Also be useful if misspelling name - when doing Google searches, the reason I want suggestions is for misspelling and not getting the wording right

Subject and person suggestions

• You are exploring Viennese architecture and are interested in seeing what topics and people may be related to this subject.  Please search for “Vienna architecture”.  
• Select a related subject.  
• Going to click on the first search result - it’s about Vienna and says contemporary 
• The question doesn’t make sense - assume subject IS vienna architecture - art is related
• When question clarified “related concepts”, then looked at the left
• For your selected subject, are there broader, narrower, or closely matching subjects? If so, could you please list them?
• Picked “city planning”
• Saw close: “art”, “urban renewal”
• Do the terms make sense, “close” and “broader”
• Instead of close, “similar” 
• Is this kind of feature useful?
• Not sure? Depends on kind of project whether or not it’s useful.  If wanted to expand beyond - if doing paper on Vienna architecture and want to know something else
• May also be interested in architecture but “Vienna” part very important
• Not sure if all the general concepts would be useful - except if they intersected with Vienna   
• You are researching Charles Dickens.  Please search for “Charles dickens”.
• Select a related person. 
• Saw related person box - charles dickens (first result)
• Are there contemporary authors for this person? If so, please list one.
• Looking in list of authors to see contemporaries
• She doesn’t know the other authors.
• Saw scroll bar but didn’t use it. 
• I’m just like, “eh.” I don’t know what makes these people related.
• That would be the info I would want.
• What do you mean related to these results?
• “Related”: not necessarily best word
• Authors from a similar era or authors who wrote similar things
• It wasn’t clear why these books came up (by Charles Dickens, about Charles Dickens, etc)
• I don’t know if ‘related’ is the right word. Need more specificity in the relationship.
• If people were relevant the way you wanted, would person/subject suggestions be useful?
• Not always sure where to go next after initial search so would be good to know what else to explore

Zero-results page

• You are researching “Akestor, King of Paphos”.  Please search the catalog with this query.  
• Can you identify a subject related to this search? 

Here are some clues, I don’t know why it wouldn’t come up in the search since it appears word for word. 

Those ones on the side. I was mostly looking at the results. Usually if there’s a list of stuff on the side, I don’t look at it. 

• Can you select one of the results from HathiTrust?
• Found the result
•  I don’t know if it’s related or not bc I don’t know enough about the topic.
• Having the bolded part is helpful
• But no bolded part in Hathi Trust selection
• Usually if I.. it’s more frustrating to get 0 results, than to get better, loosely related, and all of these have his name in them so they have info about him. 
• Have a heading that says these are related searches - it just says “zero results” and then a bunch of results.  Didn’t even see “zero results” at first.

Follow up questions (or similar) for each section

• Which of these tasks do you currently do when researching or looking for resources?  
• Haven’t done research like this in a while in any of my classes. There have been times...I think if I was trying to search something, I think this one would be the best, even if it was able to show the related subjects fields. That could be helpful.
• I usually just use Google. I don’t go to catalogs unless professor pushes us to do that. 
• Google has everything I need.
• Which of these tasks would you not do? Would you please explain why?
• Just seeing the related people. I wouldn’t know what that meant. I don’t know I would use that.
• Would any of these features be useful to you in how you search or do research?  
• Features would be useful, finding related searches, had struggles to find exactly what I was looking for so related would be helpful.
• IF it was very clear what exactly the related results were and giving results people were interested in. 
• I understood exactly what it was showing. For the authors, I didn’t understand where that was coming from.