The hidden costs of building AI products, and how UX research finds them early

Henrik Mattsson · 00:09
Hi everyone, and welcome to another episode of the UX Research Rundown. This season we’re talking about UX research and AI. Of course, everyone has to talk about AI right now. And I’ve had the pleasure to talk to a lot of very knowledgeable and cool guests, and today is no exception. With me, I have Jenny Lo, who is a UX researcher with a very cool background that she’s going to talk about herself in just a little bit, but she’s been at places like Grammarly and Uber.

And we are going to talk about some very cool stuff that I think is super relevant in this day and age, where people are just pasting AI everywhere, which is how UX research can help with lowering the costs of building these things. That’s kind of a summary. It’s going to be very interesting. So welcome to the show. Thanks for being here, Jenny.

Jenny Lo · 01:10
Thank you so much, Henrik, for having me.

Jenny’s path from market research to Uber and Grammarly

Henrik Mattsson · 01:13
Awesome. So, as per ancient tradition in this three-season-long podcast series, I always ask my guests to start by giving us their origin story. So if you don’t mind: how did you get into research in general, and into AI stuff in particular? What’s your way into research, and how did we end up here?

Jenny Lo · 01:37
Thank you so much. I think I first started when I was working in marketing research agencies. At that point, a lot of companies would build products and really try to figure out their messaging to land with their target customers. Only then did they actually learn: who are they actually looking to build this for? This doesn’t really make sense. Or there are all these issues and problems with it. And I think it was from there that I started to really hope for more companies to build with their user needs and problems in mind.

So I went and did my master’s degree, a Master of Information Management, at the School of Information at UC Berkeley. I also grew up in Hong Kong, and I was very fascinated by developing technologies in more resource-constrained environments. So then I joined Uber, being the first on their China team, working to scale products from 0 to 1 globally, and I was very fascinated by that space. From there, I went on to manage some of Uber’s more technically complex teams, such as marketplace, responsible for driver earnings and rider pricing.

Really the heart of Uber’s technology. And I think this started merging the user problem with the business problem and the technology problem, all together. My last role at Uber was head of research for the rides business, which is their rideshare, bikes, scooters and their B2B offering. And then I moved on to Grammarly, where I was leading the research team, responsible for user research, market research and research operations. Grammarly is actually where more of my background in doing AI with research came about. And it’s fascinating, because Grammarly is and has been an AI communications assistant tool, but in a way that not many people saw as AI in the current form.

People think of AI as generative AI, and that definitely has been the trend people are most familiar with, given the rise of ChatGPT. So at Grammarly it was definitely: how do we rebrand ourselves? How might we also incorporate this new technology that is possible today? And coming with it is a whole host of cost issues, along with understanding user value. I think a lot of companies are facing that challenge today, maybe in ways that they have not anticipated until it’s too late, until they’ve built with it.

So that’s a little bit of my journey, and I’m very excited to talk about that topic.

Why companies skip research when building with AI

Henrik Mattsson · 05:09
Awesome. Yeah, super excited to have you here. So let’s jump straight into it, because, running a team that builds with AI myself, it’s obvious to me that this kind of problem that you’re talking about is real. You can very easily just get excited and try to cut corners and skip the research, and then you end up with this monster of extremely expensive, what do you call it, consequences. Because it is quite expensive to run some of these things. So let’s jump straight into that. In your experience, how do companies get into that?

Because now we’ve already given the answer: you should do research and check, because this can become very expensive. But in what you’ve seen, how do people run into this trap, so to speak?

Jenny Lo · 06:16
Yeah, I think from my observation, there’s a lot of excitement with this new technology. There’s still a lot of lack of clarity as to where and how this could actually solve a user problem. But the excitement is so substantial, and the fear of being left behind is so prevalent. And I think many companies want to be labeled as an AI company, or have that capability, almost like a checkmark. And so from that perspective, a lot of people are already skipping the question of what are we building, what are we serving the customer, and going to the latter question of how do we incorporate AI.

And I think it’s the wrong question to start with, because it misses the entire journey of trying to create product-market fit. It just jumps straight into: how do we brand ourselves as AI, to be relevant in the conversation, to be attractive, so that when people look at us, they think of us as innovative? They think of us as a company that has state-of-the-art new technology. In a way, it’s much more of an engineering, technology-first conversation. And in my experience, how this came to me and our research is when engineering leaders or members might say: why do we need to do user research?

Customers, and including ourselves, may not even know what AI is capable of doing. So if we were to ask users, what are they going to offer us, when we don’t even know? We are better off creating something and then demonstrating the value to them, and then they’ll be impressed and something will stick. I think that narrative is starting to become very strong, particularly with the excitement and the discovery of what this technology can bring. But unfortunately, this is a classic type of problem. The value is not about pure technology.

It’s really about the experience that you’re offering and solving the specific problem that is actually very painful to the customer. So I think the first step is that when a new technology comes around, the narrative within the company shifts. So then how might user research stay relevant? Will the product development cycle get completely flipped and just focus on how we are going to incorporate the technology? I see this at some companies, definitely. Very true.

Data users won’t share and solutions too expensive to run

Henrik Mattsson · 09:32
Yeah, it’s so interesting, because, as you say, none of those things are really new per se. It’s more intense now because of this technology, because the excitement about the new thing is probably bigger than what it has been for other new things in the past. So it’s enhancing that. And this narrative of “well, users don’t even know what this new thing is, so we can’t ask them about it” sounds a lot to me like the old classic misconception about what user research is. It’s that kind of “if Henry Ford had asked people, they would have said faster horses” thing, right?

But it’s on steroids with this type of technology. So I find it super interesting, and I wonder: what’s the antidote, would you say, to these tendencies that we now see on steroids, but that have been around for a long time?

Jenny Lo · 10:44
Yeah, I think that is a very central question, because it does entail that user research needs to offer a response to that change in narrative. And the research toolkit also needs to adapt to the fact that it is going to be more expensive to build a prototype. It’s going to be much more important to offer a stronger confidence signal of product-market fit, or user value assessments. So that starts to gear towards some of the methodological toolkit development, in assessing products or assessing what type of problem is adequate for AI to support versus not.

I think one of the main things that research can really offer in this trajectory, and where I often see it overlooked, is that deep down with AI products, there are a lot of assumptions baked in as to what users are willing to trade off. And the very beginning would be the type of data. Some types of data are just completely impossible to share, and the value that users perceive they get back in return doesn’t outweigh the security risk that they might incur. This was definitely one of the lessons that we also learned at Grammarly.

There was a product that we’d been thinking about and built with certain assumptions as to whether users would share a type of data. It wasn’t until we did certain studies, and also got some data from customers, that we learned it’s just not possible. Reflecting back, the sequence should be making sure we’re doing more of that conversation and assessment upfront, rather than having built and then starting to learn, when the cost is already incurred. And it is labor-intensive, because this technology is very new: you have to overspend on more people to develop this type of technology.

So there needs to be more understanding around the data privacy angle. One of the painful things we learned is that people were not willing to share that type of data. It was just not going to be something they’re comfortable with. And so that was a very big learning. Another is that even with our classic user research focus on the user problem, where all else will follow, there are problems there too. For example, Grammarly learned that users really want to make sure their generative output is not as robotic.

Sometimes it’s very inauthentic: “I want it to sound closer to how I write, my voice or my style.” And it’s a very true problem. It’s also partially caused by the technology itself, but the problem is there. And you can spend many engineering cycles working on it, only to realize it’s very expensive to maintain and to launch. So this is a different situation for user research, whose classic approach is: focus on the user problem and then all else is fine. Actually, no, because now we also need to look deeply at the solution.

When it’s too expensive for the company, the solution is not sustainable. So even if you pick the right problem, and the problem is substantial enough that you should solve it, the solution also needs to be studied more. So that’s also one of the impetuses for this conversation. We may have classically looked at the problem, but now it’s about: is it painful enough? What’s the magnitude of that pain? How are we thinking about the different types of solutions? Do we really need AI? What level of AI do we need?

Hidden costs: reputation and willingness to pay

Henrik Mattsson · 15:30
Yeah, it’s almost like the classic thing when you hear your sales team come back and say, “Hey, customer X would love this.” And you’re like, “Oh, great, let’s understand that problem and build it.” And then it turns out, yeah, they would love it, but they don’t want to pay for it, because they don’t love it that much. So it’s kind of that classic problem. I love these two examples, by the way, of the cost for the company. One, you develop something that then turns out to be impossible because someone doesn’t want to share data.

And the other one is: well, we really understood the problem and we really understood how to build a solution, but it just became too expensive to run the solution. When we say it can become costly, are there other types of costs to consider here? Or are those the main two?

Jenny Lo · 16:27
I think that costs in the solution, or costs in the… yeah. What area of cost?

Henrik Mattsson · 16:40
I’m thinking, if I would categorize this, I would say that the first cost that you mentioned, if I understand correctly, is that we get excited as a team about building a solution for customers, and it turns out that because of their view on what data they can share, the whole thing just can’t be done. We’re not allowed to share that data, so we can’t do it, right? And then the cost is that we’ve lost all of this development time and all of these development resources.

We built something that can’t even work. That’s one type, right? And then you have this thing where you build something that actually does work, and that customers actually do want, and they’re like, “Oh, wow, that’s super cool.” But you can’t afford to run it, because it’s too expensive to pay the API bill to OpenAI or something, right? So those are two, and I don’t know if there are more, but I was wondering if you’ve seen other types of hidden or unforeseen costs that could become foreseeable if we just take one step away from the excitement and think it through a little bit.

Jenny Lo · 17:49
Yeah, I think those are definitely the more substantial ones. I could also imagine that some companies may not as easily understand reputational cost. For example, it’s the initial hype that pulls someone to label themselves with AI, but it does come with a reputational cost downstream: the types of risks that AI comes with. For example, with Grammarly, it’s really important to cater to the student population, for plagiarism. So how are you evaluating the potential upside, but also the downside if you don’t do that? Are you also at reputational risk for being on this AI trend, and do you need to make sure that you are serving your customers, delivering, and mitigating the risks and harm that AI could pose to someone?

So it’s a reputational risk, and something that companies really need to understand with their customers and mitigate in order to pursue it. I think there’s also an invisible cost of proselytizing, of getting excitement or getting customers, while being unclear on how much customers are willing to pay. I think that is the more grand cost, what we would classically think of: am I going to get money? How are people going to pay at the end of it? And I think that is very, very challenging, particularly when the entire industry is being flooded with funding for these AI capabilities.

Things are at a cost now, but not at the true cost. We sometimes saw this at Uber, for example. In the early days, Uber grew with a lot of promotions and a lot of driver incentives, really to build up the network. And once you have the scale and you’ve developed the behavior, then you actually start to introduce the true cost. Here, I can also see that this might really be happening. At some moment, you’re going to have to show the true cost of this technology, and the customer will then say: am I out, because it’s not serving me?

Or: yes, I’m still going to pay for this Uber ride, because it is truly solving an important problem for me, or else I can’t get to destination B. So I think that is also really big, the reputational cost as well as the actual cost, and people are having much more of a challenge identifying it.

How research can find the costs before you build

Henrik Mattsson · 21:23
Yeah, very cool. I could not have figured out those. That was great. It’s so interesting to hear you talk about this, because it just goes to show what happens if you think a little bit about these things. What I like about research is that it challenges these assumptions that we really want to believe, right? We’re excited about this thing, and just talking to users can help us reality-check a lot of our excitement. And the discussion we just had, I think, shows how that can also be done without even doing new research, just by reusing research, or just applying research thinking to it, almost.

That sounds a little bit like a paradox, not a contradiction, right? Because research is something you do with customers; it’s not a theoretical exercise. But I was wondering: what are some of the ways that research could help understand some of these costs upfront and mitigate some of these risks? If you listen to this and you feel like, yeah, this makes a lot of sense, my team fits this description perfectly right now, they’re super excited and they don’t want to hear from me: what’s your advice to researchers who are in that situation?

How do they go about inserting their research wisdom into this and trying to help their team?

Jenny Lo · 23:10
Yeah, that’s a great question. I think that is where the discipline of research offers some of its value, as a response to the narrative shift about why we need to do user research if people might not even know what AI technology is capable of. I think the value comes upfront, in really focusing on where the customer problem is, and dissecting the problem further. How painful is it? Is it a problem that has existing solutions? How cumbersome are those solutions to use? Are people willing to pay for a better solution to actually solve this problem?

Then it starts to unpack the problem more: how bad are the existing solutions, and how painful is it? The benefit of some of this AI hype is that you actually have a lot of analogous products. There are a lot of well-funded people out there, and everyone’s trying out different solutions. So it’s likely that there are startups or companies that have a similar prototype or product at some stage. So study: are people actually paying for that type of product? If it’s solving a similar or tangential problem, they’ve already spent the costs upfront.

So how are people thinking about paying for that product? Those are some of the classic kinds of research: using our toolkit to unpack the problem, understand the magnitude of the pain and the willingness to pay, as well as more competitive research on how people experience shopping for those products and how they think about the purchase. Because at the end of the day, you’re trying to get at the trade-off question: how are people thinking about the value that they’re going to get for the price that they have to pay?

How are they doing that calculation? Research is very good at unpacking mental models and how people do this kind of calculation in their own minds. So those are some things research can do. And then moving on from the problem to the solution: being part of solution development is also very helpful, because you can tell the company that this problem may have a lower magnitude of pain, such that we probably want to be much more careful about how deep our investments in AI go in this particular problem space.

In the discussion about finding a problem and using AI, you want that matching. Maybe it’s not as painful for a certain subgroup, but for this population it’s extremely painful. Then it starts to help craft what the level of investment in AI should be. Being part of that solution development, and assessing that, is going to be very helpful. And for the researcher, it means: how are you involved? Not just with design. Be present with your engineering partners, hear a little bit more about the challenges they’re coming up against.

I think it’s a very exciting opportunity to co-create in a space that we may not have been as familiar with before. Working closely with engineering to understand, hey, if we keep asking the user for different prompts, or to refine their prompts, this raises our costs, because each prompt is very expensive. One of the things that we considered at Grammarly was an improve button that combines several tasks into one. It improves the user experience, making it much easier. And it could also reduce the amount of back and forth that the customer needs to do with the chat.

So this could be a way to improve the user experience and also find ways to reduce the cost of running it. So there are a lot of new ways to explore doing research in this solution development phase as well.

In Lookback Show participants a competitor's product or your prototype: in a moderated Lookback session, push a link that opens in their browser. Learn more

Re-prompting and cost as a research question

Henrik Mattsson · 28:33
Yeah, absolutely. And this also makes me think that very often the cost of not doing research is dumped on the user, and only eventually on the company, because, theoretically, the users leave the company if there are better options, right? And in this case, with the re-prompting: if you have an experience that ends up in a bad AI output, then you’re going to have to re-prompt. Not getting that right, and having an unsatisfied user with a bunch of opportunities to retry, retry, retry, retry: usually that cost would just be on the user.

It’s their time, all the retries. But now it’s so clearly dumped back on the company. So I wonder if this is almost a good lesson. Companies that don’t get this right deserve to pay that cost a little bit, to learn from it. Perhaps it’s a healthy thing in the long run. Perhaps that’s too harsh, but…

Jenny Lo · 29:41
That’s so funny that you say that. That could be a very fascinating signal and opportunity. Like what you’re mentioning: where in the experience is there pain, and could one be cost? It would be very fascinating. Traditionally, user research may have shied away from a lot of this cost conversation, really focusing on delight and making the user experience very intuitive. That’s still true. But in the equation of how to be a very strategic researcher, in a conversation that is increasingly removing user research, I think cost is an important way to put yourself back in that room, into that conversation about why user research is going to be very valuable.

And the most valuable it could be is that you actually help the business make money, reduce their costs and make the right investments. To the point about bad output: it is so painful. When I watch videos and observe how people are using AI: “Hey, I’ve spent so much time, about ten minutes, trying to figure out how to talk to the prompt in order to get the output. I want to write this email. It’s almost better if I just write this email.” I thought that was such a fascinating equation of time and effort.

Are we actually making a solution that is helping people, or making things much more complex as a result?

Will AI replace UX researchers?

Henrik Mattsson · 31:39
Yeah, so interesting. So if I interpret you correctly, I would say that you are not one of those who worry that AI will replace UX researchers. It’s rather that we’re going to need more UX research. Is that a correct interpretation?

Jenny Lo · 31:56
Yeah, I think so. My belief here is that the narrative and conversation in the industry has not reached what we’re talking about right now. And the conversation also hasn’t realized that this is a potential ability that research has. More importantly, I think that worry about costs is going to come, and it’s going to happen. Right now it’s just about who has the deeper pockets, and maybe they care less if they’re well funded. So I think that’s definitely where the conversation is going to shift, and it’s a skill set that user researchers should be more prepared and ready for.

Ready to tackle understanding AI privacy and trust. How do we deal with that? And investigate: how are we able to offer better perspectives on the upside and user value, but also the costs to the company? How do we actually work closer with our design and engineering counterparts, again, to prepare us so we’re not replaced? I think, with AI or not, it’s very hard to just build products that have product-market fit and really solve a problem for the user that is very painful, that they’re ready to pay for.

And those things are always going to be true. And I think AI can help you in many ways, maybe in understanding, from your problem discovery, the micro tasks within it. But at the end of the day, I think it’s very hard to say that, as a complete function, we’re going to be replaced by AI. Our function also needs to do more, but in what direction will it evolve? I’m also looking very closely at companies like OpenAI. Are they going to build a user research function?

If so, what would that look like? I think there are a lot of different signals we can look at for how the discipline might evolve with this new trend going forward.

Why continuous research helps when new technology arrives

Henrik Mattsson · 34:39
Yeah, definitely. And it’s interesting. I want to try a thought I just had on you, to see if this is accurate. It was very fascinating to see, in our own experience with all of this stuff, that we got a little bit lucky. We also got very excited about AI, and I was definitely one of those people who wanted to make sure that when our customers and new prospects checked in to see what we were about and asked themselves, “Is this going to be an AI company?”

, they would think, “Yeah, of course. Lookback, yeah, this is great.” So I was definitely in that risk category. I got very excited, and I wanted to really push on this and everything. And I think to some extent we got a little bit lucky. My CTO is like, “Oh, it’s just hype.” He’s very cynical about these things, so he didn’t get us excited. But I realize also that one thing that really saved us was that we’ve done so much research over the years. It wasn’t a new research question for us.

We’ve done so much research on what the core problems are. So when a new technology comes along, it’s not a new research project for us. We’ve already done the research over time to understand the well-known problems. What are the things on that list that this technology now offers us a way to solve, that weren’t possible before? So it’s also a great reminder, I think. But I’d love to hear your thoughts on it. Perhaps I’m a little bit biased, and I want research to be super important here.

But it seems to me that the more these technology landscapes change, the more new ways of doing things there are, and AI is a great example of that, the more important it is that you’ve done your research continuously, so that you know what the key things are, right? And then it’s not a problem that you’re excited about it, because you don’t need to do new research. You can just think it through from that perspective. So I don’t know, am I biased and exaggerating this, or what are your thoughts on that?

Jenny Lo · 37:10
I absolutely agree. I think that is the best position, because what this means is that you have invested over time to know that this is the tried and true, most important and painful problem. Oftentimes these startups now are doing the flip side: I want to incorporate AI, then figure out the problem. Whereas now it’s: we know from investments over time and doing research that these are the most painful problems users have. And now you can go one step further: what are some of the capabilities that I can offer to be helpful in solving this?

Are we strategically unique in offering this, if we want to incorporate it? That was also an exercise we did at Grammarly when ChatGPT came about. We reviewed all the previous research, the critical user problems that we had, and what was likely to be solved by ChatGPT, what Grammarly is uniquely capable of doing, what is relevant for the category, and what AI would potentially be useful to solve. And we also got a very interesting dimension here: what did users feel like they wanted to keep doing themselves?

What do they enjoy doing, and what are the tasks they were quite clear they just don’t want to do? “I don’t want to do formatting” and certain things. That was very clear. And that was also a very helpful angle for moving beyond it. There are people who are having that flipped conversation and decision making, versus people who have done the research work, so they can move much further along and make the right types of investments for the problems that most people needed solved.

In Lookback Search across the transcripts, notes and highlights from all your team's past research. Learn more

AI risks that companies evaluate too late

Henrik Mattsson · 39:24
Yeah. Awesome. So now we have reached that point in the podcast that I warn all my guests about, where we could talk about this for hours, but we’re running out of time. But I do have one more question here. I love that you’re bringing this perspective. There’s so much we could talk about, but I think this is something that people forget about in the excitement. So I think it was great that we had this conversation on the costs that research can help discover and mitigate the risk of.

When you listen to conversations about AI and UX research, and AI in general, incorporating it into all of our products, because that’s what everyone is trying to do now: are there things you have been thinking about that you see as obvious risks, that we’re not really talking about? Other things of a similar nature that you recommend people reflect on a little more? Big question.

Jenny Lo · 40:25
I think there’s a lot of potential harm that AI is capable of doing. And it’s about how the company is positioning itself on investments in transparency and in safeguarding the user’s interests, and not overlooking their data privacy and trust. The obvious risk that I do worry about is that there is excitement. There is such an assumed belief that the value is so substantial that the risks people might impose would be overcome. And that equation, that journey people have, means that the risks for the customer are always evaluated too late.

Obviously, it is a new technology. But the way we want to make sure of it is: how do we bring that more into the product development cycle? How do we bring it into the conversation? I think user research is very effective at bringing in the users: what are they going to be excited about, where is the value, or what are the risks and downsides? My biggest worry is that it’s now put downstream in the product development. How do you bring it upfront, much more front and center?

So I think it’s really about asking a lot of companies where to sequence the types of investigation, and how to bring much more forethought into the decision making process. Too often someone says, “Oh, it’s such a rare case that this might happen.” But you do not want to be the one, because we all know, and have seen it all over the news: when those risks happen, it’s very damaging to a company. And at that point, that’s the reputational risk. You jumped into it, and you didn’t figure out the right response from the very beginning for how to safeguard your customers.

Henrik Mattsson · 43:00
Very, very wise. I think we’ll let the conversation end on that very relevant point. Thank you so much for being with us today. I’ve learned a lot, and I’m sure that our listeners are going to love this episode. So thank you very much. And to everyone out there, happy researching. It’s exciting technology. We shouldn’t be afraid of it, but we should use our powers to do the right stuff in the right order and actually build some great experiences, instead of taking the wrong risks and messing things up.