The Measuring Experience Field Guide
TYPE OF STUDIES FOR MEASURING USER EXPERIENCE
There are many ways to measure success as there are to measure experiences. But it is the efficient triangulation of 3 types of UX metrics that can bring us a complete understanding of how the user behaves and perceive the quality of a product or service. This section will help walk you through the two types of studies and show you how to use them to produce reliable, actionable results without breaking your budget.
“
The first step to choosing the right UX metrics is understanding your organization's objectives for your digital solution.
How good is my UX?
Designers have the power to use UX metrics to communicate to businesses how good their UX is or aren't. They can assess it on 4 levels:
Marketing-Opportunity Fit
Measuring Brand Attitude Perception, Appeal, and Convenience
Problem-Solution Fit
A Measuring Error Rates, Reliability, and Completion Rates
Product-Market Fit
Usage Rates, Usefulness, and Ease-of-use rating
Business-Market Fit
Renewal Rates, Sense of Novelty, and Innovation
Measuring Brand Attitude Perception, Appeal, and Convenience
Problem-Solution Fit
A Measuring Error Rates, Reliability, and Completion Rates
Product-Market Fit
Usage Rates, Usefulness, and Ease-of-use rating
Business-Market Fit
Renewal Rates, Sense of Novelty, and Innovation
How to choose the right UX Metric?
Metrics are the signals that show whether our UX strategy is working. Using metrics is key to tracking changes over time, benchmarking against iterations of our own site or application or those of competitors, and setting targets. Most metrics are marketing-oriented, not experience-oriented. The below metrics are a scheme to counter this trend.
As mentioned in my previous post How to Measure Experiences, I organized UX metrics according to Hedonic and Pragmatic properties, or both. But usability is not a single, one-dimensional property. If we want to measure usability we have to measure both.
Usability is attitude (hedonic) plus action (pragmatic).
Ease of use is usually one of the biggest differentiators in the customer experience: If people don't find your product easy to use, they aren't going to be very satisfied or loyal.
Usability is attitude (hedonic) plus action (pragmatic).
Ease of use is usually one of the biggest differentiators in the customer experience: If people don't find your product easy to use, they aren't going to be very satisfied or loyal.
Usability is the extent to which a product can be used with effectiveness, efficiency, and satisfaction in a specified context of use.
Type of Methods
There are many ways to measure success as there are to measure experiences. But it is the efficient triangulation of 3 types of UX metrics that can bring us a complete understanding of how the user behaves and perceive the quality of a product or service. Those 3 types are:
Issue-based Metrics
UX Performance Metrics
Self-reported UX Metrics
The first two generate behavioral data from how the user interacts with your digital solution. The last one generates the perceived data, meaning, how the user perceived the quality of your digital solution.
Issue-based and UX performance metrics will be a better instrument within this ocean of data if you connect them to how we arrived at a certain service quality that can be reported by self-report UX metrics.
For example, Issue-based metrics and performance metrics can measure the number of first-time signups and give us an idea about the effort required to reach out to new users, but they wouldn't give us any insight into the perceived quality of a new sign-up. Self-reported UX metrics are our final layer.
Although, performance and issue-based UX metrics, like task completion time or error rates, have been commonly discussed in our field. I feel that self-reported metrics haven't had the same spotlight. And its neglect can affect the effective triangulation of these 3 forms of UX metrics.
Issue-based Metrics
UX Performance Metrics
Self-reported UX Metrics
The first two generate behavioral data from how the user interacts with your digital solution. The last one generates the perceived data, meaning, how the user perceived the quality of your digital solution.
Issue-based and UX performance metrics will be a better instrument within this ocean of data if you connect them to how we arrived at a certain service quality that can be reported by self-report UX metrics.
For example, Issue-based metrics and performance metrics can measure the number of first-time signups and give us an idea about the effort required to reach out to new users, but they wouldn't give us any insight into the perceived quality of a new sign-up. Self-reported UX metrics are our final layer.
Although, performance and issue-based UX metrics, like task completion time or error rates, have been commonly discussed in our field. I feel that self-reported metrics haven't had the same spotlight. And its neglect can affect the effective triangulation of these 3 forms of UX metrics.
Issue-based Metrics
Track the frequency and severity of issues that users encounter while using a product or service.
Subjective & Attitudes Metrics
UX Performance Metrics
Self-reported UX Metrics
Metrics that rely on users' subjective feedback about their experience using a product or service. Provide valuable insights into users' perceptions and attitudes towards a product or service.
Behavioural & Performance Metrics
Provide insight into how users are interacting with a product or service and can help identify areas where the usability and ease of use can be improved.
Self-reported, UX performance and issue-based UX metrics are different types of UX metrics used to measure different aspects of the user experience.
Self-reported metrics are based on users' subjective feedback and opinions about their experience using a product or service. This includes metrics such as user satisfaction, ease of use, and perceived value.
UX performance metrics are based on users' objective performance using a product or service. This includes metrics such as task completion rates, time on task, and error rates.
Issue-based UX metrics are based on users' interactions with a product or service, and identify specific issues or problems that users encounter while using it. This includes metrics such as the number of errors or issues encountered, the severity of those issues, and the frequency with which they occur.
By measuring each of these different types of UX metrics, product teams can gain a more comprehensive understanding of the user experience and identify areas for improvement. They can then use this data to make data-driven decisions to optimize the user experience and ensure that it meets and exceeds user expectations.
Self-reported metrics are based on users' subjective feedback and opinions about their experience using a product or service. This includes metrics such as user satisfaction, ease of use, and perceived value.
UX performance metrics are based on users' objective performance using a product or service. This includes metrics such as task completion rates, time on task, and error rates.
Issue-based UX metrics are based on users' interactions with a product or service, and identify specific issues or problems that users encounter while using it. This includes metrics such as the number of errors or issues encountered, the severity of those issues, and the frequency with which they occur.
By measuring each of these different types of UX metrics, product teams can gain a more comprehensive understanding of the user experience and identify areas for improvement. They can then use this data to make data-driven decisions to optimize the user experience and ensure that it meets and exceeds user expectations.
Task-based vs Study-based Study
Task-based and study-based studies are used in UX research for different purposes, depending on the research goals and the type of insights needed to improve the user experience.
If you want to collect objective data about your solution, feature, or task effectiveness and efficiency you can move through a task-based study.
But if you want to collect subjective reactions on the overall satisfaction of your solution, feature, or task, you can move through a study-base study.
If you want to collect objective data about your solution, feature, or task effectiveness and efficiency you can move through a task-based study.
But if you want to collect subjective reactions on the overall satisfaction of your solution, feature, or task, you can move through a study-base study.
Effectiveness and efficiency are best measured by so-called objective data, which means behaviour data such as error rates or the time needed to complete a task.
you can choose to determine areas of improvement for your product or service.
ISO 9241
Task-based Evaluation
Study-based Evaluation
With this evaluation, you can choose to continuously improve your product or service by measuring the user experience of a new version or improvement. Or test the current quality of your user experience.
Task-based Studies
They are typically used to evaluate the usability of a product or service and to identify specific areas for improvement. These studies are often conducted during the design and development process to identify and fix usability issues before a product or service is released to the public. Task-based studies can also be conducted after a product or service has been released to gather feedback on its usability and identify opportunities for improvement.
Study-based Studies
They are used to gain a deeper understanding of users' needs, preferences, and behaviors when interacting with a product or service. These studies are often conducted during the early stages of the design process to inform product development and ensure that the user experience meets user needs and expectations. Study-based studies can also be conducted after a product or service has been released to gather feedback on the user experience and identify opportunities for improvement.
What to Measure?
Compare your UX to competitors and industry
Test if a product has sufficient user experience quality standards
Monitor the overall level of satisfaction and identify potential areas for enhancement
Continuous improvement by measuring the current or new version of the user experience
Study-based Study
Task-based Study
This decision tree provides you with the help you need to choose the right UX metric. I guide you through the decision process so that you may find the right metrics for your situation. Any feedback is welcome!
Task-based Study
Within a Task-based study, you can choose to continuously improve your product or service by measuring the user experience of a new version or improvement.
As I mentioned in my post Why Measuring Experiences you can measure it after the use of the new version or improvement or over time to understand if this new version has an impact on retention levels between new and old users, for example.
To do so, you can choose between some of these UX measurements tools:
As I mentioned in my post Why Measuring Experiences you can measure it after the use of the new version or improvement or over time to understand if this new version has an impact on retention levels between new and old users, for example.
To do so, you can choose between some of these UX measurements tools:
Within a Task-based study, you can also plan to test the current quality of your user experience. You can plan UX metrics to measure the whole product or service or just a new feature for example.
To do so, you can choose between some of these UX measurements tools:
To do so, you can choose between some of these UX measurements tools:
Study-based Study
Within a Task-based study, you can choose to determine areas of improvement for your product or service.
To do so, you can choose between some of these UX measurements tools:
To do so, you can choose between some of these UX measurements tools:
Last but not least, within a study-based study, you can also plan to run a comparison test to see where you stand regarding your competitors and market.
To do so, you can choose between some of these UX measurements tools:
To do so, you can choose between some of these UX measurements tools:
