There are many different types of user research methodologies out there, and, sometimes, the same method can be referred to with different names, such as generative and exploratory research, or they can be similar with slightly different nuances like usability testing and task analysis. Usually, as a user researcher, you specialize in different techniques, such as generative research, but can comfortably and confidently conduct other methods. However, for those who are interested in user research, but aren’t in a role dedicated to understanding the whole field, it can be difficult to know which method to pick.
Picking the “wrong” research method for the problem you are trying to solve can end up being a huge waste of time and effort, in the end. Certain research methods lend themselves to better answering certain types of problems, and when used incorrectly, can really skew results to either be completely unusable, or they can lead you in the wrong direction.
It is important to choose the most optimized method for what you are trying to understand and accomplish. By choosing the right research method, you can ensure the results of your research are, both, relevant and informative. The right research method will allow you to uncover insights to move your product/service forward, in the right direction.
What are the different methods?
First, it is important to have a basic understanding of what some of the most common research methods are, and what they aim to accomplish. I have listed the most common methods I have seen in the field. This is not an extensive list, but covers most of the reasons people want to do research. As this is an overview of the methods, I highly encourage you to look into each method specifically, especially those you decide to use.
I will start with more exploratory research types, and then dive into evaluative research. However, before that, let me explain the overall difference between the two:
Generative research allows you to use open-ended conversation to get a user to tell stories about experiences. From there, you foster and develop a deep understanding of that person’s overall motivations, goals, needs and pain points—both inside and outside the context of your product. By understanding a person’s underlying thought processes while they are considering or using your product, you can go far beyond improving your current offerings.
Evaluative research allows you to test your existing solution, or a prototype of the solution, to see if it meets user’s mental models, is easy to use, and is enjoyable. Getting designs in front of users as soon as possible ensures the experience will be shaped and refined to meet customer needs and expectations.
Generative research methods:
- One-on-one user interviews: These interviews are 60-90 minutes long, and involves having a conversation with a user on their needs, goals, tasks, pain points and emotions. This gives a very in-depth understanding about how users think, and what problems they have with your product. It can be used to shape the future of a product.
- Diary studies: Diary studies are longitudinal projects that can provide valuable insight into how users interact or use your product/service in everyday life. This helps determine both product requirements, improvements and innovation
- Mental models: Mental model interviews include asking users about their thought processes regarding how they imagine a product/service should work and how they expect to achieve certain goals through a series of actions/tasks. This leads to prioritization, innovation and improvement
- Journey maps: Journey map interviews are where you ask the user their about the experience outside of your product/service and the touchpoints/relationship they have with your product/service. This leads to prioritization, innovation and improvement
- Contextual inquiry: This is a semi-structured interview in which you primarily are observing the user interact with your product/service in their natural environment, giving you context you might not glean from an interview
- Jobs To Be Done: A one-on-one interview style that helps you understand the reasons behind purchasing behavior, and what ideal goals users have in mind when purchasing a product
Evaluative research methods:
- Surveys: Surveys are tools that can measure large sample sizes for understanding how user’s engagement with and perception of your product. It can also help determine user’s preference and needs at a large scale
- Card sorting: Card sorting helps you determine how information should be structured in a website or app by allowing users to sort information on cards in the way they would expect to find it
- Usability testing: Usability testing is testing your product (either live or prototypes_ with users by asking them to complete important tasks. This ensures your product/service is aligned with how users would expect to solve problems using your product. In addition, this can also help you find bugs in your product
- Competitive analysis: Competitive analysis is done by listing out the different weaknesses or strengths of competitor’s products. This provides insight into what your competitors might be doing well and where the market opportunities lie
- Benchmarking studies: Benchmarking gives you an understanding of how users are currently completing tasks on your website, and how long it takes them to do so. This gives you an understanding on how people are currently using your product, and also if there are any hacks being used to accomplish tasks
How do I do choose?
As you can see, there are a lot of different methodologies to choose from, all with their particular strengths and weaknesses. So, how do you even know which research technique you should choose for your project. I have defined a 3-step process to go to, which will enable you to choose the best method for your project.
Step 1: What stage is your product in?
By answering what stage your product is in, you can more easily decide a high-level approach by understanding if the current state of your product is better suited for generative research or evaluative research.
Generative research is best used when your product and company are in the following situations:
- Before or at the beginning of developing a company or building a product as generative research helps you understand who your users would be, what problems you would be helping them solve and how they expect to solve those problems
- When you have a fully built product and are trying to figure out the next steps for the product and the company. Generative research especially helps with the following:
- Determining upcoming strategy by enabling you understand the most pressing issues your users are facing, in order to pivot or build features aligned with user’s expectations
- Understanding how to innovate by asking users what is currently missing from your product/service, or what others are doing better than you
- Trying to develop visual deliverables, such as journey maps or personas. You need a deep understanding of your users in order to create impactful deliverables, and that requires a lot of one-on-one conversations
Evaluative research is most effective in the following scenarios:
- When you are testing prototypes of a new concept. This is the number one usage of evaluative research, and a really great way to get designs in front of users early, in order to understand if they make sense to your audience.
- When you are testing the current flow of a product. Evaluative research helps you understand if what you currently have makes sense to the user. This is also helpful in uncovering hacks and poor user experience
- When you are trying to do fast and iterative changes on a product that doesn’t require a deep understanding of behavioral triggers. Small changes can easily be tested through evaluative measures, and don’t necessarily require extensive understanding
To boil this down:
- A concept or idea: we need to understand the process people are currently going through (generative research)
- Innovation/strategy: we need to figure out what we will do next, and what else people would want from our product/service (generative research)
- A prototype: we need to uncover what people think about the prototype and how they expect to use it (evaluative research)
- Live code: we need to evaluate the performance of the product and what people think about it (evaluative research)
Step 2: What questions do you want answered by the end of the project?
Once you determine generative versus evaluative research, you get to decide on your actual methodology, which is the real fun part. You can often end up with more than one methodology to use, and that is perfectly fine to.
In this step, I write down the different questions I want answered by the end of the project, since that is the most important part. I ask myself, ‘what am I trying to learn?’ and ‘what much this research achieve?’ I call these the objectives of the project and they really help me define the technique I will use to illicit the answers I need from participants. If my objectives are more about understanding thought processes or the solutions people currently use, I will lean more towards generative research methods. If they are towards understanding how something works, and how someone will use something, I will use evaluative research methods.
Below are some sample objectives, and the different methods I would use. There may be overlap across a few methods and objectives, such as one-on-one interviews, as you can focus the subject of the interview to be directly about what you need to learn.
Sample objective 1:
Understand the end-to-end process of how participants are currently making decisions
Since you are trying to understand the process people are going through, generative research would be the best idea for this kind of project. The following specific methods are great for answering this objective:
- One-on-one interviews
- Mental model interviews
- Contextual inquiry
Sample objective 2:
Learn about any improvements participants might make to their current decision-making process
With this objective, you are trying to gauge what could potentially come next, and how you can make your current solution even better. Generative research is ideal for determining strategy and innovation. The following techniques can answer this objective:
- Journey map interviews
- One-on-one interviews
- Jobs to be done interviews
Sample objective 3:
Evaluate the current user experience/usability of a product/service, and any pain points users are running into while using the product
This objective is all about evaluating performance, which makes evaluative research the perfect avenue to take. These following techniques can help during this stage:
- Usability testing
- Benchmarking studies
- Card sorting
Sample objective 4:
Identify how users currently feel about a product/service, and how they use it on a day-to-day basis
I had to save the most difficult for last, as this could be either generative research or evaluative research. You could use the following methods to understand this objective:
- Diary studies (generative) to understand how people are thinking over a longer period of time
- Surveys (evaluative) to discover, on a large scale, how people are thinking about your product/service
Step 3: What types of participants do you have access to?
While this is probably the least important part of the process, it is still helpful to think about, and might help you choose between a few different methodologies.
I generally break users into three different categories:
- People who currently use your product/service
- People who do not currently use your product/service (and may use competitors)
- People who have stopped using your product/service
I usually do my best to get a mixture of types of participants during a study, but sometimes you need certain participants in order to answer certain questions. If you are trying to understand the problems users are facing with your product, it is certainly more helpful to talk to people who have been using your product. If you don’t have access to any current users, you could also try with people who use competitors, to see how your solution stacks up. Similarly, if you are trying to innovate to find a new market or space to grow into, it can be more helpful to talk to non-users in order to see what their current solution is, and how you could fix any problems they are having with their current solution.
Overall, I would always recommend trying to do some form of user research, even if it isn’t perfect, however, be thoughtful about why and how you are conducting your research to get the best results possible.
