Using Advanced Statistics in UX Research

Moderated by Michele L. Oliver, Ph.D., UXr Guild Board President, with Guest Speaker Reem Ayouby
This is an abridgment; view the full video presentation here.
Session 3 – November 10, 2022

In session two of “The Q&As of Quantitative UX Research Methods,” Michele Oliver identified various tests and when to use them. The first section here is a continuation of this previous discussion.

How to Do a Chi Square Test in Excel

For A/B Testing, we’re going to do a Chi Square test. An example of using this test was when users were asked to update financial aid holds, and they were in one of two conditions. In one case, the null hypothesis is that there was no difference. Before doing the test, there were a few calculations we needed to do. This would be easier in Spss, but if you don’t have access to that, here are the steps using Excel:

1. Set up your table with frequency counts. Here you will see the raw data. Then convert the raw data into frequency counts to see how many were successful in each group and how many failed per group.
2. Calculate the totals for the rows and columns.
3. Calculate what is called Expected Values in a new table. In a Chi Square test, the observed values are what you get in the test itself, and the expected values are the calculations that were made. We then need to compare these two values. Do this in a new table to clearly see the data. The expected value is equal to the row total multiplied by the column and then divided by the overall total. Exp = (Row total x Column total) / Overall totals.
4. Calculate the actual Chi Square value. To find this, take the observed value and subtract it from the expected value. Once you do that, you square it. Then divide that by the expected value. X 2 = ∑(O – E)2 /E. This is easier to do in Excel because you just build the formula, and then you can do this with all your values separately. Then add them all up to find that Chi Square value.
5. Calculate what is called Degrees of Freedom (df). Use a 2×2 matrix with two columns and two rows. To calculate this, it is Row minus one, multiplied by the Column minus one. df = (r-1) x (c-1).
6. Then put this calculation into the formula bar in Excel to calculate the p-value.
7. Summarize your results by comparing the two groups.

Using Advanced Statistics for UX Research

(The following sections were presented by Reem Ayouby.)

While basic statistics are sufficient for many studies, advanced tools result in a more in-depth analysis. Some researchers might shy away from complex mathematical equations, but it is possible to do this without knowing all the formulas. But first, let’s set the stage.

The Difference Between Function and Feature

A function is what needs to be done. This can be related to the Jobs to be Done framework. A feature is how the app enables the user to get that job done. These definitions are important because they can be used interchangeably. An example is the A/B Test where the same function can be performed in two different ways and one may work better. The function is the same; there are just two ways to implement it through features.

The Importance of Data Visualization

Data visualization allows users and stakeholders to see, understand, and interact with the information you are presenting.

Visualization tools show how different factors will change; how two variables behave together. If you add a third variable, you can observe any interactions among them. This is helpful when presenting information and helping others see the ramifications of adding other variables into the equation. And if you add an interaction term, it can get even more difficult. Exploring these tools can help identify which research questions these can answer for you.

Can we use correlation? (IV and DV) – the correlation visualization. 

http://statkat.com/visualization-regression-one-predictor.php

Can we use regression? (2 IV and DV) – the linear correlation visualization

http://statkat.com/visualization-regression-two-predictors.php

What if there is interaction effect visualization? – the interaction effect visualization

http://statkat.com/visualization-regression-moderation.php

What is Correlation?

Correlation tells us when two variables are moving together, and one is higher, the other one will also be higher, and if one drops, the other will drop also. That’s the correlation, but sometimes there is more than one variable. For example, perhaps you want to know which products an individual will use as they grow older. Instead of just looking at someone’s age, you may want to add a variable such as a personality characteristic. If both go up, then a certain dependent variable will change similarly. 

If two independent variables may influence an outcome, and then a third is added, it starts to get complex in terms of how we can visualize this and how well we can make sense of what’s going on with the data.

What is Quantitative UX Research?

To answer this, a group of research scientists at Facebook came up with the following definition:

“Quantitative UX research delivers insights about people. UX researchers often approach research projects with questions such as: What are the human motivations for using these products? How do people perceive and use the product? How do they react emotionally and physically to it? What do they like and dislike about specific features? What role does the  product play in their daily life?” Mary Nguyen and colleagues, 2017 

By removing human motivations, perceived use, reactions, likes, and dislikes, you can often discern some of these through your qualitative research. So, why use quantitative research to revisit these ideas?

Qualitative research usually involves small sample sizes, not interviews with hundreds or thousands of people. With this small sample size,  research results may or may not generalize to the wider population you hope will purchase your product. Quantitative research can give a sense of the magnitude and strength of the relationship and how to generalize these findings to this wider population.

An example of a framework that is used in UX research is the Perceived Ease of Use and Perceived Usefulness. As researchers, you may frequently focus on Perceived Ease of Use to see how easy a product is for people to use. If they are not using it, it might be too difficult. You will then need to determine how to make the process easier and more intuitive. 

But then there’s also the Perceived Usefulness. Getting the Perceived Ease of Use right will remove hurdles. But it still has to be useful, which means it needs to be perceived as having useful functionality. With that, you can go into a lot of depth.

This is a typical model often used in user research in academia, but it also introduces the idea of causal inference. The idea is that when the perceived functionalities are in place, that leads to perceived usefulness, which leads to use, which then would lead to an increase in subscription level. It may even lead to customer satisfaction and loyalty. 

The Dichotomy of Qualitative and Quantitative Research

The following breakdown of research approaches moves from qualitative methods through mixed methods to quantitative methods is from a presentation by Chris Chapman at the Quantitative UX Research Conference held in June 2022.¹

Example: Social Ethnography:

• Field Work
• Ethnography
• Interviews
• Discussion groups

Example: Digital pen

• Usability assessment
• Beta/trusted testers
• International studies
• Prototype testing

Example: Profile CBC

• Qual/Quant groups
• Satisfaction & CSat
• User profiles
• Survey research

Example: Sentiment overview

• Conjoint analysis
• Sentiment analysis
• Competitive analysis
• Psychometrics

Example: Feature demand

• Regression models
• Multivariate stats
• Casual modeling
• Market estimation

When looking at regression models, you are trying to understand how various factors affect your variable of interest, often referred to as a dependent variable in multivariate statistics. But, you cannot consider that many variables by using linear regression. This is because the relationship between these variables may become complex. The complexity must be captured by the statistical model for useful conclusions to be drawn from the analysis. 

A solution to this is to use structural equation modeling. A friendly and more forgiving variant of structural equation modeling is partial least squares structural equation modeling (PLS-SEM). It is friendly because it is visually appealing. And it is forgiving because the statistical assumptions which must be respected are less demanding than for other statistical methods with similar capabilities.

An example of where using structural equation models was helpful in a recent study was exploring the impact of social media use on well-being. This model helped address questions of the influence of motives and perceived social media functionality on social media use; how social media use affects an individual’s well-being; and how emotion regulation is involved in the outcome.

This scientific research revealed that although there is a risk for negative impacts, it is possible for social media use to have positive impacts on individuals’ well-being. With this knowledge, researchers could then help companies research and design electronic social networks that would have the desired impact on users and society.

The full presentation offers an in-depth analysis of the use of structural equation models in this study.

About the speakers:

Michele Oliver has a Ph.D. in Experimental Psychology with an emphasis on Psychophysiology, Statistics, and Research Methods. She has been a Senior Lecturer and Adjunct faculty member. She is currently a Principal UX Researcher at Ellucian, a provider of SAAS solutions for higher education. Contact Michele at michele@uxrguildorg.flywheelstaging.com or through the UXr Guild Slack Channel.

Reem Ayouby, Ph.D., is a UX researcher and data scientist who leverages a combination of qualitative and quantitative approaches in her research. Her doctoral research focused on the motives, perceptions, use, and impacts of social media platforms such as Facebook. Reem holds a doctorate in Business Technology Management, a master’s degree in Management Information Systems, and a bachelor’s degree in Commerce with a major in Management Information Systems from the John Molson School of Business, Concordia University.

---

¹ CN Chapman (2022). Two UX Research Cases. In CN Chapman, KZ Xu, M Callegaro, F Gao, and M Cipollone, eds. (2022). Proceedings of the 2022 Quantitative User Experience Conference (Quant UX Con). June 2022, Sunnyvale, CA.