Leafly Usability Study

Purpose: M.S. Human-Centered Design & Engineering Project

Collaborators: Michael Dougherty, Andrea Sequeira, Christopher Yang

Project Role: User research, usability study development, moderation, visual design

Project Overview

Leafly is an online information resource designed to help individuals discover, connect, and share cannabis-related information. Leafly provides industry news, crowdsourced product and dispensary reviews and information, and details about various strains of cannabis.

The recent and ongoing legalization of cannabis in the US provides an opportunity for Leafly to capture a large, untapped market share: novice cannabis users.

Our objective was to assess the overall usability of Leafly for key tasks for novice cannabis users: searching for information, resources, and products.

The study was designed to illuminate opportunities improve the first-time user experience and provide consistently delightful, productive, and satisfying interactions for our novice cannabis users.

We aimed to answer the following research questions:

• How easily and successfully do users find the relevant content to accomplish their information-seeking goals and make purchase decisions?

• What are the most significant usability issues preventing users from accomplishing their goals or resulting in inefficiencies?

• What parts of the product work well for users? What do users find frustrating?

• What are the most common errors or mistakes users are making?

Participants

The study included 6 participants who were all novice cannabis users new to Leafly. We recruited a mix of genders and ages. Our participants had the following characteristics, fitting our participant profile:

• Age 21-44

• Interested in recreational cannabis use

• Had used cannabis 10 times or less in the last six months

• Had owned an iPhone for at least six months

• Had not used Leafly (website or app)

• Had not worked for Leafly or competitors

Tasks

We gave our participants 5 tasks:

• Task 1: Find an article [for beginners]

• Task 2: Find a dispensary near [a location] and how to get there

• Task 3: Find a strain good for [insomnia]

• Task 4: Find a strain that [won’t cause hunger]

• Task 5: Find [cannabis-infused butter]

Methods & Procedures

Testing Logistics: We recruited 6 participants using an online screening questionnaire distributed via social network connections. All participants were scheduled for one-hour sessions over the course of one day. Testing was conducted in a dedicated testing environment with an iPhone.

Data Collection: The moderator administered the test seated next to the participant, while three observers took notes on customized data collection forms.

In addition to in-person observation and note-taking, we also recorded audio and visual data from the sessions with screen capture software and a video camera.

Our test included the following components:

• Standardized test script - Introduced participants to the study, set expectations, and ensured consistent administration by different moderators.

• Consent form - Obtained consent for participation in the study, as well as video and audio recording.

• Pre-study questionnaire - Collected basic demographic data.

• Discovery - Open-ended qualitative questions to better understand the context of each participant’s behavior and build rapport.

• Tasks - Five tasks structured into relatable scenarios for the participants, conducted with the Think Aloud Protocol. Each task concluded with a post-task ease-of-use Likert scale rating.

• System Usability Scale (SUS) survey - Measured people’s subjective perceptions of usability of a system for benchmarking future tests.

• Post-study questionnaire - A series of open-ended qualitative questions to follow up on additional points of interest.

• Honorarium - Participants received a $5 gift card and a thank-you note for their participation in the study.

Findings and Recommendations

The Strain Filter feature only appears on pages embedded deeper in the app architecture and is not easily discoverable. Despite multiple participants expressing the desire for this feature, none of our participants were able to find it.

• 0 of 6 participants found the Strain Filter feature.

• 5 of 6 participants expressed wanting a filter with this functionality.

  • “I want to filter - I might just end up going to Google.” (P2)

Recommendations:

• Increase visibility of Strain Filter feature by page placement. For example, place the filter on the Explore Strains page.

• Integrate first-time user assists (ex. icon bounces) on key features like the Strain Filter.

When typing in the Search Leafly search bar, the on-screen keyboard covers the lowest autofill results, blocking them from view. To compound the problem, autofill returns results different from executed search, so there is no workaround to access these listings.

• 3 of 6 participants were frustrated by their inability to access all the autofill results in Search Leafly.

  • “I hate this too, when you have to scroll to click the last thing in a list but the keyboard blocks it.” (P3)

Recommendations:

• Display lowest autofill search result above the keyboard

Product categories are unclear and the menu hierarchy is disorienting. All participants unsuccessfully looked for cannabis-infused butter by navigating the Products section. Some participants were unable to find the specific product within the catalog, while others were unable to even identify the correct sub-category.

• 5 of 6 participants initially attempted to find cannabis-infused butter by navigating the Products section, implying that it was easy to discover.

• 0 of 6 participants successfully completed the task.

• 5 of 6 participants verbalized wanting to use a search feature within Products.

  • “Products seems where I want to be, but it hasn’t been super fruitful.” (P2)

Recommendations:

• Add a search bar to the Products section. If further testing shows an information architecture problem, conduct a card-sorting study.

There are inconsistencies between search and the navigational structure of the app that poses a difficulty for users. The Products section lacks the option for local search present in other sections and executed Search Leafly results return headings that are different from what users see in the rest of the app.

• 5 of 6 participants resorted to using Search Leafly after struggling with the Products section.

• 5 of 6 participants entered task-irrelevant app sections through Search Leafly.

• 6 of 6 participants expressed confusion over Search Leafly results sometime during the test.

  • “It sucks that I can’t search inside products and brands instead of having to search the entire Leafly app.” (P2)

Recommendations:

• Make search and navigational structure support each other by adding search bars to all major content sections for consistency, and change Search Leafly headings to reflect the structure of the app.

When searching Leafly, there is a profound difference between the autofill and executed search results. Autofill results are categorized into some combination of Strain, Dispensary/Store, Articles, Doctor, Brand, Product. Executed results are always categorized into: All, Strains, Locations, Articles, and Reviews.

• 0 of 6 participants found expected content from Leafly search results.

• 5 of 6 participants verbalized wanting to search within Products.

Recommendations:

• Make search results consistent with categories appearing on both search options, preventing drop-off.

• Conduct further usability testing to ensure search hierarchy is consistent and delivers desired content

Articles for novice cannabis users are difficult to discover. Participants did not expect to find this type of information hidden within News.

• 3 of 6 participants incorrectly went to Help when looking for an information source for novice cannabis users.

• 3 of 6 participants entered News, but didn’t find anything relevant on the landing screen and exited the section.

  • “Because I’ve only ever done edibles, I would want an FAQ type thing… I don’t have that information yet and I don’t know where to find it.” (P4)

Recommendations:

• List category options prominently on the landing screen of News Home.

• Feature a Cannabis 101 article.

On the Map View screen, users were often distracted by issues with map icon sizing and spacing.

• 3 of 6 participants expressed difficulty or confusion with reading the icons based on their relative size and position.

• 2 of 6 participants questioned Leafly potential profit motives behind icon size hierarchy.

  • “It’s hard to get to the other one on the map. UI issue. I can’t get to the other one behind [this dispensary].” (P6)

Recommendations:

• Maintain dynamic sizing of icons as the user zooms to prevent icon stacking and hiding some locations from view.

Quantitative Data Summary (by task)

System Usability Scale

The System Usability Scale (SUS) is a widely accepted measure of people’s subjective perceptions of system usability. SUS Scores range from 0 to 100, but the score is not a percentage. SUS scores are a percentile ranking with a mean of 68.

Leafly scored 56, a score that has low marginal acceptability and is slightly above “OK” in terms of adjective ratings.

This score is not diagnostic and has limited usefulness alone. SUS is best used as a benchmark to quantify usability in comparison with future test results. We recommend continuing to evaluate this metric with future iterations of the app in order to track overall usability improvements.

Next Steps & Conclusion

We recommend that Leafly prioritize addressing the high severity usability issues immediately. Following that, we recommend reviewing the other proposed recommendations and design changes to ensure that they are aligned with both novice user goals and Leafly business goals - then make those changes as appropriate.

This usability study shed light on many actionable ways to improve the app for first-time Leafly users who are also novice cannabis users.

We believe that there is a big opportunity in this market space at this time. As legalization leads to more dispensaries in other states, the user base will continue to grow - we believe that now is a key time to capitalize on this space.

With the power of good UX, there is an opportunity to unlock massive potential and create the leading online cannabis experience.

Final Documentation

The full usability report can be found here.