UX Study Design & Analysis

Don’t allow a poor user experience to affect the value in your service or product.  

Harlen O’Keeffe is here to discover your UX problems.

 Using smart study-designs and expert statistical analysis; we create a perfect balance between low-cost testing and effective problem-discovery.

What Is UX Analysis?


What Is UX Analysis?


User Experience (UX) refers to all aspects of an end-user’s interaction with a company, its services and products. UX practitioners focus on making sure these interactions are enjoyable, useful, easy to understand.

Fundamentally, UX should aim to extract all the value in your offering and ensure it is delivered to the end user. A great UX will even create intangible (or perceived) value in your product which wasn’t there to begin with!

So how does UX impact on your business? In short, a lot. Good UX is great for your business and a bad UX can be devastating. Here are some stats that prove the point:

  • $547 million dollars in added earnings estimated for retail sector in 2014 due to good UX
  • $150 billion dollars in lost annual earnings for projects that delivered a poor UX
  • 45% of users will abandon a registration process if it is too hard.
  • $339 AUD lost per customer on average due to poor UX.

So how can your business create a great UX? Well, there are two key elements; data and design. Most UX practitioners are focused on the design and implementation process, but how do they know what designs will work?

That’s where we come in! At Harlen O’Keeffe we focus on the UX data. How to collect it, and how to analyse it. Our services help you quantify your business’s current UX, compare it against benchmarks and even design studies to find how many problems an old or new UX is likely to have. All of this information is then used to make sure you can make awesome, well-informed (and cost-saving!) UX design choices for your business.




Imagine you have a new product and you want to ensure your product’s UX is up to par.

Firstly, it is important to know how prevalent your UX goal conversions and/or problems are amongst the population of people that will use your product. For instance, if 10/30 participants in a study designed to test your product run into the same UX problem, does this mean that a third of all your potential users will also encounter this problem?

Well, maybe…

This hypothetical study only used 30 people. This might not seem like enough people to jump to any conclusions about ALL your customers. However, even with 30 people we can often make some useful ESTIMATES about everyone who will use your product. In this example, using some fancy statistics, we can be confident that the true rate of this problem occurring for all people using your product will be between 20%-50%. I.e. even though the study found a problem rate of 33%, we can generalise this result and say with a high level of confidence that the overall rate for EVERYBODY using your product is going to be somewhere between 20%-50%.

So, using 30 people, you found that a particular UX problem in your product could potentially be encountered by 20% (best case scenario) to 50% (worst case scenario) of your users. If this study was also looking at how many of the participants run into other UX problems with your product, you would be able to build up a picture of which problems are going to be experienced most frequently by people using your product. This is especially useful when you have a limited budget for fixing issues, and need to prioritise which UX design flaws you should tackle.

Q: Hang on! 20%-50% seems like a pretty wide estimate don’t you think? Why can’t you just give me an exact percentage?

A: Unfortunately, whenever we use a sample from the overall population (in this case, a sample from the population of people who will use your product) we loose the ability to give exact values for things. The best solution is to do a study and then estimate the value by providing an interval in which we can be very confident the “TRUE” value lies.

Q: What if I want an interval smaller than 20%-50%?

A: That’s certainly possible. Unfortunately there is a trade-off. In order to make the interval smaller, we would have to increase the number of people in the study. For instance, if we went from 30 people to 60 people, the interval would shrink to 22%-45%. If we had 120 people, the interval would shrink to 25%-42%. With 1000 people the interval shrinks to 30-36%.

Obviously 1000 people in a study is infeasible due to cost, but that is the amount of people you would need to create a 30%-36% interval (i.e. 3% either side of 33%).

What this hypothetical demonstrates is the need for estimates that are “useful” to your company, without blowing your budget on a 1000-person study.

Enter Harlen O’Keeffe!  We work with you to create UX studies that will.

  • Give you the best chance of accurately detecting UX problems and goal conversions.
  • Provide useful estimates to the values of UX problems and/or goal conversions amongst all people using your product.
  • Accurately compare your level of UX problems and/or goal conversions against benchmarks
  • Ensure the cost of the study remains within an affordable range for your company





“In God we trust,

all others bring data.”


W. Edwards Deming