Prioritizing Research Studies:Scaling the Impact of Your Research Team

Presented by Jeanette Fucella
Director, User Research & Insights, Pendo
This is an abridgment; view the full video presentation here.
February 2, 2023

Modern product teams are simultaneously balancing speed to delivery of new features and the need to test hypotheses, uncover unmet needs, and gain new insights about their users. So many user research questions; so little time! ​Given these competing pressures, how do you determine which studies to prioritize? How should a product team determine how to invest their limited time as well as their customers’ time? When is it worth investing in time-consuming primary research, and when is it ok to “ship it and measure it”?

Jeanette Fucella, Director, User Research and Insights at Pendo, answers these questions and many others in her recent session with the UX Researchers’ Guild.

Scaling the Impact of a Research Team with a Prioritization Framework

When Jeanette first put together this prioritization framework, she thought it might generate a few likes when she posted it online. But then, as she describes it,  “the thing went berserk” and has since been translated into multiple languages. This simple 2×2 matrix has helped Jeanette keep her sanity, and she encourages others to modify it to use in their unique work environments. 

Ratio of Researchers to Designers to Engineers/Developers

According to a study published by the Nielsen/Norman Group some years ago, the ratio of researchers to designers to developers or product can be skewed. This is where the prioritization framework stems from. According to these findings, the most common extended ratio is one researcher to five designers to 50 developers. And these ratios are probably changing with recent layoffs occurring in industry. Whereas in the past, these ratios may have been livable, now they are totally out of proportion. 

With this ratio, it is imperative to prioritize your time because people requesting research, whether designers, engineers, or product managers, outnumber those trying to provide the research. 

In the UX industry, there is a lot of talk about democratization. Whether you’re against or for it, is an irrelevant conversation. Researchers can’t survive without delegating a portion of their research activities and tasks.

This is where the framework comes into play. To use this effectively, however, there must be a change in the way researchers think about and approach their work.

The Role of a UX Researcher – Myth versus Reality

To those who know nothing about a researcher’s role, it might look like a slew of experiments, making amazing things happen while navigating successes and failures. And that would be true.

But what if we reframe how a researcher looks at their role? Instead of coming in to save the day, researchers are frequently putting out the fires to ensure that things don’t metaphorically go up in flames.

As an example, researchers want product managers to do the most effective and rigorous research possible. But what if the product manager is the problem? Then researchers need to explore a solution in the most effective, rigorous way possible. This results in a reframing of who you are as a researcher, and your role in relation to your colleagues, and your work.

The User Research Prioritization Framework

The framework came about as Jeanette discussed with her boss how to decide what to work on personally versus what to delegate to other people. That’s when he suggested a 2×2 matrix. Once she put her ideas into this format, she realized that it resonated with her colleagues and it became a meaningful and effective tool.

The framework begins with a vertical axis representing “Problem Clarity” (with a range from high at the top to low at the bottom). This first axis centers on problem clarity, which researchers deal with regularly.

This problem space is specific to customer problems, as opposed to business and product problems. This then begs the question: how well do you understand the specific concerns of your customers that you as a researcher are trying to solve or address? It might be high; it might be low. These customer problems, often referred to as Jobs to Be Done, are the issues customers are talking about and trying to achieve with or without our product. That’s different from product problems which show up as feedback requests. Product problems exist because a product exists and business problems exist because of the business. 

While there are multiple problems, stakeholders must focus on the customers’ problems within this framework. These other problems do exist and should not be minimized, but where innovation occurs is at the intersection of the three.

All design processes, including the double-diamond shown below, begin with identifying the customer problem which shows WHAT researchers are trying to solve for the customer. But it’s also to identify and mitigate risks as early as possible. Without a clear understanding of the problem, it’s difficult to know what risks might lie ahead. 

(For a more in-depth look at dealing with problems as a researcher, see Jeanette’s article, “The Problem with Problems,” https://dovetailapp.com/outlier/the-problem-with-problems.)

The horizontal axis represents the “Risk of Getting it Wrong” (also from high to low) or in other words, how bad is the risk if you get things wrong? Will customers bail right and left, or will they hardly notice?

The four quadrants created by the two axes, and recommended research methods, are as follows:

1. Research Heavy – Low problem clarity, high risk of getting it wrong. For this quadrant, Jeanette recommends doing Generative Research: JTBD interviews, Contextual inquiry, and Diary studies
2. Design Heavy – High problem clarity, high risk of getting it wrong. Recommendations: Iterative design and Evaluative research: Prototype evaluations, Click tests, and Comprehension tests
3. Research Light – Low problem clarity; low risk of getting it wrong. Recommendations: Generative research: Semi-structured interviews
4. Ship It and Measure – HIgh problem clarity, low risk of getting it wrong. Recommendations: Experimental/evaluative research: A/B testing, Behavioral analysis.

Variations on the Prioritization Framework

The great thing about this quadrant is that researchers can make it their own to fit any given circumstance. The following articles, and modified framework below, demonstrate variations on this framework and how individual researchers have leveraged it to meet individual needs. 

https://uxdesign.cc/building-a-framework-for-prioritizing-user-research-ed46622ead99
https://www.notably.ai/blog/how-to-choose-the-right-qualitative-research-methods
https://dovetailapp.com/blog/research-method-discovery-product-management/

Collaborating with Cross-functional Partners

On the issue of how to collaborate with cross-functional partners, there is no one right answer. It’s very organizationally dependent. Jeanette shared that she has data analysts and scientists that work right around the corner from her who organizationally report differently. Oftentimes, data scientists, especially in an early-stage startup, are focused more on revenue modeling. So that depends on what their remit is. But if you have people for whom that is part of their remit, those are great people to partner with. Part of the challenge is once you’ve set the KPIs or the measurements, remembering to monitor and check it, and then report back to the team.

A researcher’s role is to define what those measurements are and be accurate about what behavioral change accurately measures the success of the new functionality or solution you’re trying to implement.

Thoughts on the Prioritization Framework

Some people may be surprised that they hadn’t seen this framework before. But most would agree that its simplified nature works very well. One use of this matrix might be to plot things on the framework as a group to facilitate a good conversation with stakeholders. Another aspect to consider might revolve around how to layer in the dimension of time or window of opportunity in which research can make a difference, even complicate the prioritization of the work.

Research Impact

Some questions to consider alongside this prioritization framework might include: How impactful will this research project be? How does it align with the business priorities and the business objectives? In our current economic times, the more tightly you can connect the work you do to the priorities of the business, the better. Knowing what the priorities of the business are, and using those to help also prioritize your work, is vital. 

About Jeanette: Jeanette Fuccella started her career at IBM, as a human factors engineer, designer, researcher, strategist, and evangelist. After 16 years in this area, she moved to LexisNexis where she led design ops and research ops initiatives as well as a large global research team. She is now the Director of User Research and Insights at Pendo.