Generative AI and UX research: why asking a model can't replace asking users
Listen on Apple Podcasts, YouTube or your podcast app.
In the season 3 opener of UX Research Rundown, a season about AI and UX research, Lookback CEO Henrik Mattsson talks with Jess Holbrook, who leads UX research for generative AI at Meta and co-founded People + AI Research at Google. They discuss what LLMs change in a researcher's work, why asking a model is no substitute for asking users, whether LLMs blur the line between qual and quant, and why UX researchers should stop acting like underdogs.
Key takeaways
- Talk to users, not to a model that stands in for them. A model gives you a statistical average of whoever has posted on the internet. Jess doesn't trust anyone who doesn't want to talk to their customers.
- Speak up in meetings even when you don't have data for every point. You may be the only person in the company who has talked to 50 users of the product, more than the PM, the director or the eng lead.
- Match the rigor to the decision. An academic needs validated measures. A team deciding where to move next needs signal in the right direction, and a quickly built Likert scale can be enough.
- Hand AI the tactical work: summaries, transcription, first drafts of reports, and searching thousands of past reports, such as every study your company has run on long press. Summarizing quotes from a study was never the core of a researcher's job.
Chapters
- 01:34 How Jess Holbrook got into UX research
- 04:36 From Google's People + AI Research to generative AI at Meta
- 09:34 Working in AI since before the ChatGPT moment
- 13:09 What LLMs change in UX research work
- 17:44 Asking a model instead of asking users
- 20:08 A social science of LLMs
- 21:38 Can LLMs bridge the qual and quant divide?
- 28:37 Validity, Likert scales and when signal is enough
- 30:46 Why researchers should trust what they learned from users
- 33:19 UX research and its seat at the table
About Jess Holbrook
Jess Holbrook is principal researcher and acting director of the generative AI UX research team at Meta, where he led the Responsible AI UX research team before that. At Google he joined Blaise Agüera y Arcas's team in Google Research as one of its first UX researchers, and co-founded People + AI Research (PAIR). Before Google he worked at Amazon and Microsoft. He is a social psychologist by training. He got into the field through a summer job at a startup in Eugene, Oregon, while in grad school at the University of Oregon, and started at Microsoft as a contractor after 49 rejections.
Transcript
Henrik Mattsson · 00:09
Hi everyone, and welcome to another season of the UX Research Rundown. I am Henrik Mattsson, the CEO of Lookback and your host for this show. And it’s a brand new season, season number three. For two seasons, we’ve been talking about getting your team along on the journey that is research. And that’s been fun. But now we’re moving on to something else. We’re going to talk this season about AI, LLMs and UX research and all that stuff. And we have a great lineup of very interesting speakers.
The first one that I’m talking to today is Jess Holbrook, who is at Meta, and I’ll let him introduce himself in just a moment. Thanks so much for coming on the show.
Jess Holbrook · 00:55
Yeah, thanks for having me. I appreciate it. It’s fun to be able to kick off the third season. Congrats on that.
Henrik Mattsson · 01:02
Oh, it’s awesome. So, you know, we always do this thing, per ancient tradition, these three seasons that we have now. We ask our guests to introduce yourself and tell us a little bit about how you got into the field, because we always get a lot of good feedback from the listeners who are often getting into the field or just starting out. And it’s great to hear all of these different stories about how people get in and the backgrounds and stuff like that. So we can start with that, and then we’ll get on to the topic at hand.
How Jess Holbrook got into UX research
Jess Holbrook · 01:34
Yeah, that sounds great. So my current role is I’m the principal researcher and the acting director of the generative AI UX research team at Meta. Before that, I led the Responsible AI UX research team here at Meta. Before that, I had a very similar role at Google, and I started a group called People + AI Research, or PAIR, there with two of my co-founders. And before that, I was at Amazon and Microsoft. My story about how I got into the field is fairly happenstance, actually.
I was in grad school at the University of Oregon in Eugene, Oregon, and I kind of needed a summer job to pay my rent that summer. And so I think I joined the only startup in Eugene, Oregon, at that time. And as part of being a startup and being an intern, you just come in and find out what you’re going to do each day. And one day the people I was working for were basically like, “Hey, could you design the page that we sell our ads on for our website?”
And I was like, “Sure, no problem.” They’re like, “Great, just use Dreamweaver.” And I was like, “Cool, what’s Dreamweaver?” And people of a certain vintage listening to your show will either get that reference or they’ll be seriously Googling or TikToking right now. And so I started just playing around with that, and I started to basically comment out code and say, oh, when I get rid of that code, it moves this thing that was in the middle towards the left. Oh, weird. It changes this color.
Oh, okay, that’s where this image is coming from. I went back to my advisor, and I didn’t even have words for design. So I said, “Hey, I like psychology.” I was going to grad school for psychology. “I like psychology, I like technology, and I like art.” I didn’t even have the language. And he connected me with a really brilliant advisor at the University of Oregon, and she totally took me under her wing and got me into all the HCI work and cared about me and invested in me.
And that’s a huge part. So to this day, I try to spend a decent amount of time on mentorship, on working with our pipeline, because I really believe that whatever amount of success I’ve had has been in no small part about people caring about me and deciding I matter and trying to help me along the way. And so I think it’s really important, especially for the people you talk to on a podcast like this, that they’re sending the elevator back down.
Henrik Mattsson · 04:03
100%. That’s great that you’re doing that. Cool. Sorry, I didn’t want to interrupt your story there, but I think that’s what that was.
Jess Holbrook · 04:15
Yeah. And then, I mean, maybe this is another part of the “don’t get discouraged.” Then I applied to 50 different places, and I got rejected by 49 of them. And then I got on as a contractor at Microsoft, with some great people that I still consider mentors to this day, and kind of got full-time positions from there.
From Google’s People + AI Research to generative AI at Meta
Henrik Mattsson · 04:36
Nice. Yeah. I remember a similar background, in terms of all the mentorship and everything. And when I look back at that time, once you’re in… I don’t know if you had this feeling, but I’m like, how did I… this was impossible. How did I get in? And then you’re in, and then you’re trying to help other people. So don’t get discouraged is the message there, I guess. So how did you get into the AI stuff? And broadly, what kind of AI stuff are you working on?
When you say you’re directing the research team on generative AI, what does that mean?
Jess Holbrook · 05:18
Yeah, I always think it’s interesting. So I’ve been in the AI space for, I guess it’s closer to ten years than it is nine at this point. But I work with people who got their PhDs in AI in the 1980s. So I still feel like I’m kind of brand new, and I just showed up and I’m trying to contribute something. I was actually working on the cloud team at Google, and this is when the cloud team there was teeny tiny, and now it’s tens of thousands of people or whatever.
And this director came over from Microsoft, who joined Google and Google Research, and he gave a talk about how he was starting a team and his vision for personal AI and all these things. His name’s Blaise Agüera y Arcas, as people know him. And I was really inspired by it. And so I said, “Hey, let me come over.” And so I joined his team when it was him and ten engineers. I just hopped over, and I was like, wow, this is really exciting. This is kind of just after ImageNet.
So the deep learning models had just proven their superiority to the previous implementations. And so everyone was like, oh my gosh, this is going to work. We’re going to be able to put this into products. And so from there, we started to build up a UX team. To my knowledge, I was one of the first UXers in the research division at Google. I know that there were people before me, but of that era, I think I was one of the very first. And so we started building up the team, and as we were doing that, we were trying to build products, and we started to notice these commonalities, these patterns around the uncertainty of AI, or the anthropomorphizing of AI, or just the false beliefs that people have.
And so we started to put those together into a series of principles and durable insights. Myself and Josh Lovejoy started a group that we called human-centered machine learning. And then later on we started this group called People + AI Research with Fernanda Viégas and Martin Wattenberg. And we built that into a whole lab, I guess you would call it. And from there, I led the UX team. The space has also gone through these metamorphoses over time. We used to talk about it as human-centered AI, and then the umbrella term of responsible AI started to take over.
And so then I led the responsible AI UX team there. And then I came over and did a similar role here. And so that brings me to the day-to-day job. We just launched a few things two weeks ago at Connect, our big event: a series of AI agents that are powered by Meta’s main LLM, Llama 2, an AI assistant called Meta AI, and some image generation tools, one for stickers and another for doing text to imagery. And so the day to day really looks like stuff that would look very familiar to other people, which is just basic product work.
What is our strategy? Who are our users? What do they want? Are we building something that works for them? Is it usable? All that. But then we also get to much deeper levels. One of the things that’s most interesting to me about AI is that we, especially as UX researchers and UXers, have much more of a say in how it’s made than just about any other platform before it. When cloud rose, we weren’t having these discussions about a responsible cloud, or mobile, or anything like that.
And so then we get into things like how we define success for the underlying base models or the tuned models, what kind of prompt engineering techniques we can develop and replicate, tons around safety and red teaming of models, which have really risen in importance recently, and then all the way to working on some of the marketing and messaging work too.
Working in AI since before the ChatGPT moment
Henrik Mattsson · 09:34
Interesting. Yeah, it’s so interesting to hear you talk about this, because it’s a good reminder that this is actually… I wouldn’t say it’s old technology, but it’s been around for a while. And now, with the LLM stuff, it’s had this big rise to global fame. Everyone is using the different models and stuff. So how is that when you’re on the inside? How does it feel to be in your field in the last, whatever, year, six months or something?
Jess Holbrook · 10:04
Yeah. Like being in a dishwasher, or in the washing machine. Maybe it’s more like, anybody who claims that they’re up on everything that’s happening has a bridge in Brooklyn to sell you, kind of thing.
Henrik Mattsson · 10:17
Yeah. It’s been fascinating to see how many people are now gen AI experts. Right? You know, that’s how these things go.
Jess Holbrook · 10:25
Yeah, for sure. I mean, before gen AI, before the ChatGPT moment or whatever you want to call it, and to this day, I still think it’s incredibly fun stuff to work with. I’m not here by accident, not here sleepwalking. I regularly check in on myself: okay, is it time to do something else? And I still say to myself, no, I think this is the most interesting thing I could do with my time. And what was fascinating is, we’re coming up on 11 months now since ChatGPT.
This is giving Engelbart some run for his money as greatest demo of all time. They put out ChatGPT as basically a demo of the model, a website to be like, “Hey, check it out, our model is really good.” And they had the greatest accidental consumer product success in history. They just skyrocketed. And it’s always fascinating to me too, what pierces into the public conscious, the public conversation. When the Few-Shot Learners paper came out, we were all nerding out on it and going, oh my God, do you understand what this means?
So we kind of hype-cycled ahead of that, right? We’re like, oh my God, this is amazing. And then you kind of go, wait, wait, wait. And you pull back and you say, okay, and you’ll have points of view on both sides. And what I always try to be, and have maintained, is this kind of pragmatist in the middle: okay, so here’s what it can do, here’s what it can’t do, here’s what we need to figure out. And at the end of the day, to use a cliche, it still has to solve a problem.
Or else you can push it into as many things as you want, and people will reject it, a big “no thank you,” essentially, when you’re talking about a lot of consumer products and products that have choice around them. And so that’s been really fascinating. Ever since then, it’s just been this kind of rocket ship. And I think that whenever that happens, you have a bunch of incentive misalignment across the board. Some people are trying to push an agenda or a message, some people are trying to go super fast to catch some kind of reward.
And so for UX research, it’s really important that we both ride the tiger but stay focused on what we know matters, which is: are you going to make people’s lives better, in a way that they can articulate back to you and that you can measure somehow? And then, is that good for the business that you’re working for?
What LLMs change in UX research work
Henrik Mattsson · 13:09
Yeah, it’s been super interesting. So just to make sure I understand it, would you say it’s fair to say that there’s the rise of this technology… And if we talk about LLMs now, in the ChatGPT moment… I still don’t get those letters right. I think I got it right now, but I’m always like, did I say it right? So that presents new challenges to researchers: how do we actually research the kind of product that we build on top of these technologies, and what it does?
But it also introduces these technologies as a tool into our work. So how much of each have you experienced? Does the rise of the tooling space feel distinct from the other stuff, or is it all the same? Tell us a little bit about that experience, coming from this long ten-year experience in this field.
Jess Holbrook · 14:15
Yeah, it’s been interesting, because before the LLMs, there wasn’t a huge AI shift that really affected the UX research world. At the margin, but it didn’t upend how we do what we do or anything like that. And I don’t know if this one is going to upend how we do what we do. It’s going to change it. My experience has been far more on how do we apply this to products, what is the responsible way of using this kind of technology, and far less on what does this mean for UX research.
And it is funny, because I need to actually do a lot more here, because I get asked this. People are like, “Oh, you’re the gen AI researcher, you’re doing this, right?” So I will tell you what I’ve observed in the conversation, though. What’s interesting is, one, it’s kind of funny to see UX research had the same reaction as a lot of other people, which is like, oh my God, it’s going to do my job for me. What am I going to do now? I look at that and say, if you think that your job is summarizing quotes from a study, I don’t think you understand what the job of a UX researcher is.
“If you think that your job is summarizing quotes from a study, I don't think you understand what the job of a UX researcher is.”
And I think that’s more of where we need to start. So there’s this really tactical level that it will hit. It will probably help us summarize things faster. Transcriptions are going to get a lot better. Searching through structured databases, or even unstructured transcripts, for insights, we’re going to get a leg up there for sure. We’re going to figure out how well the models do with that. We’re going to figure out where the hallucinations come in, we’re going to address that, we’re going to march on, and we’ll probably be faster at all of that.
I think one of the more exciting things there, too, is being able to look across broad structured and unstructured data, archives, I think sometimes they’re called data lakes. I’m talking about tens of thousands of reports or data points, and being able to really focus in on insights, versus what team did this come from or what product was it about, to look across everything. Even just tiny little interaction patterns. Everyone at every company, at some point, for a mobile app, has been asked, “Should we do long press?”
And you’re like, well, there’s lots of research on long press across a zillion different things, so maybe we can pull that all together. And then you have these other little accelerations. The cost of first drafts just went to zero. So writing your report, the template stuff, it’s all good to go. This is also highly bounded by the current abilities of the models, especially around languages. We’re just talking about English right now. Outside of that, mostly it’s not going to be much help to you. And then the more provocative thing I’ve seen, of the “AI is going to take the UX researcher’s job,” is there’ll be a proposal or a company or something that’s like, oh, you can just ask a model, you don’t have to ask users.
In Lookback Eureka chat answers questions across all of your team's research, not one project at a time. Learn more
Henrik Mattsson · 17:41
Yeah, I love that kind of thing.
Asking a model instead of asking users
Jess Holbrook · 17:44
Yeah. And so I’ll say, for me, I have the emotional reaction to that when I hear it, which I think anybody does. One is, I think it’s actually really good for us to hear that, to build our empathy better with people, because a lot of times we sit in a spot that feels un-AI-able. So I like that little shock. But then I kind of said, okay, take the emotion, but what is it? And I said, well, if it’s a tool… So if what you really think is the best feedback for you is to query a statistical representation of what an average human on the internet, based on who is on the internet and who has put content on the internet, would reply to this question, then you should use that.
That makes sense. Now, I’m going to bet that’s not actually what you need. And then for me personally, I kind of have a thing of, I don’t trust anybody who doesn’t want to talk to their customers.
“I don't trust anybody who doesn't want to talk to their customers.”
Henrik Mattsson · 18:52
That’s the quote right there.
Jess Holbrook · 18:56
Yeah. No, it’s like, would you do business with somebody who… If I get my oil changed and the manager doesn’t want to talk to me, like, what is that? And I look at it as one of two things. Either one, their business is going to fail, so why prolong it? Or the other is they have such a structural advantage, or something else in the market, that nothing will beat that, so the work is irrelevant. I’ll add one more thing on to this, about the role of gen AI in the work of UX research.
I do think it has to come back to how do we use this to better understand people and have more durable insights about human behavior as it relates to technology and business, rather than how does it speed some tasks up or change the way we do some tasks. And I don’t think we’re scratching at that. I think we’re actually highly distracted by “it’s going to speed up some tasks” or “I’m not going to be able to do these tasks,” and we’re not nearly focused enough on how is this helping us understand human nature much more deeply.
A social science of LLMs
Henrik Mattsson · 20:08
Right. And do you mean, totally apart from the conversations we’re having, having the models figure out something about humans that we don’t know? Or are you talking about the conversations, how can we enhance that somehow?
Jess Holbrook · 20:25
I think more of the former. I’m a social psychologist by training. And I’m like, this is one of the greatest statistical representations of human communication we’ve ever been able to construct. And yet a lot of times we’re treating it like a fairly surface-level implementation detail on some things. And I think that there will be a whole social science around LLMs. I don’t think it will dominate, but you have tons of social scientists who look very closely at network theory and social media platforms, and these will be an artifact of that level that they will mine for insights.
“I think that there will be a whole social science around LLMs.”
And not blindly, because I can already hear many of my colleagues, they just hopped up on my shoulder, and they’re like, “Jess, hold on. You’ve got to talk about the bias. You’ve got to talk about all the errors. You’ve got to talk about all the problems.” And there is all that. But everyone I’ve talked to that’s thought seriously about this idea is very eyes wide open about that, and trying to examine it and use it as a tool with those caveats.
Can LLMs bridge the qual and quant divide?
Henrik Mattsson · 21:38
Yeah. I could nerd out on this. My background is economic geography. So I grew up reading all of these geographers in the 60s who tried to build these time geography models, as they call them. They’re basically trying to figure out… We’re corporeal beings, we live in our little worlds that are local environments and their networks, and then we have our desires and stuff, and they just try to model all of that to predict society, basically. And back in the 60s, they were just like, yeah, we’re going to figure this out.
They didn’t have the data to do it. But there’s a lot of stuff like that that could actually be done now, in a sense, with this kind of model. So it’s an interesting… and that’s obviously going to help us as a context for the research. But if I can dig down into where do we go with the tool and stuff like that. Someone sent me this paper, an investor sent me a paper: “You should just build AI participants,” these fake participants, basically. “Here’s the paper that proves that that’s possible.”
And the paper was basically around how you could do this to replace a survey or something like that. This model will perform better than a survey. But that makes me think about something that I want to try out on you, since you know way more about this. The whole quant-qual war, the dichotomy, right? Why do we do quantitative stuff like surveys? Well, we do it because we need to be able to analyze the data. We’re kind of forcing humans to answer in pre-selected buckets so that we can make a nice graph on that, instead of just having them speak freely, which we do in a more deep qualitative way.
But it struck me that these LLMs, and I’d love to hear your opinion on this, what they do, actually, is they kind of solve that problem, because you can now look at extremely rich qualitative datasets. So you have no need to do a survey now. What do you think about the potential of that stuff? I might be completely misunderstanding something here, or…
Jess Holbrook · 23:58
No, I think that… I mean, yeah. So the quant-qual divide, a couple of things you hit on in there. The quant-qual divide to me has always been weird. My whole undergrad and grad school was all about applying cognitive methods to social phenomena. My undergrad lab was the implicit association test lab, Tony Greenwald’s lab at the University of Washington. And so it was basically, how can we quant these super qual things? And so, I don’t know, to me you’re always kind of doing both. With quant and qual, it’s like, do you want the tyranny of math or the tyranny of language?
Because it’s going to force you down a particular path, depending on when you say, I’m going to bundle up these observations, if you will. The quantification is nice because it allows you to fulfill the assumptions of most statistical analyses. Again, these are things that I think get lost in a lot of these conversations. I love the one paper thing, where people are like, “A paper says blank.” I’m like, do you know how science works? Do you think we do anything based on a single paper?
Especially in certain fields. Sometimes in some physics fields or something, even then you would still want multiple papers to validate something or to build evidence toward it. And so in the qual realm, I think that’s the really interesting thing, that it’ll start to give structure. And by structure, we need some kind of reduction, some kind of simplification. Oh, this is fun, we’re going there now. This comes back to the limitations of human attention and our ability to take in these different inputs. That’s why we need abstractions, and that’s why we need to have these simplified representations of knowledge, because the whole wide world would be too much to take.
And so, in theory, it will allow us to abstract that in ways that are more useful. But your question is really prompting: we don’t exactly understand how it’s becoming structured. And that could be problematic, because with a relational database, or if we’re just running an ANOVA or something like that, I know how everything’s being structured. It’s a formula, it’s very clear. And so I can interrogate that, I can examine the assumptions, and I can decide how much I trust it. This is one of the challenges, although the field is moving rapidly.
I haven’t read it, but Anthropic put out a paper this week that seems to be a big step forward in terms of the interpretability of models. So, yeah, with all of this, we’re not even a year into the ChatGPT era.
Henrik Mattsson · 26:58
Oh, yeah. Crazy times. So thank you for that. A follow-up question on the structure. I agree, this is obviously a big problem. What we’re trying to do when we’re building tooling is to always keep the transparency, right? Where does this come from, why did it say this? Because we’re building two things, just quickly. One is just headlines, so you can get an overview of things. It takes the transcripts, and it’s just like, here they’re talking about this, here they’re talking about this. And because it’s timestamped, you can always go and see: “Well, give me a highlight reel of that.”
“Okay, well, actually it’s correct, you can see that the humans said this thing.” And the other thing is basically, whenever a researcher presses a button instead of having to take a note, the LLM makes a proposal for a note, right? And that’s also this kind of transparency, so you can check it. But of course, the whole point, the power of this, is to be able to look at a lot of things that you would never be able to look at otherwise, right? But then we don’t know how they’re doing it.
So in that case, I totally buy what you’re saying about the quantitative advantage there. But one could say, and I’d love to hear your view on this, when we’re forcing people to rank things on a 1 to 5, we don’t really know how that’s being structured either. How did this thing become a five? How did this thing become a four, and how did this thing become a three? So I wonder if there’s some way here that we could actually improve quant models too, by studying that phenomenon with this kind of stuff.
I don’t know, perhaps I’m just going on a big rant now. But this is extremely fascinating to me as an ex-social scientist.
In Lookback Headlines give each session a clickable table of contents, so you can jump to the moment behind each summary. Learn more
Validity, Likert scales and when signal is enough
Jess Holbrook · 28:37
Yeah. I mean, what you’re hitting on in a ton of ways is how do you know that the measures are valid, right? These are the validity questions. And for what we do, I think it’s interesting, because this comes up a lot. To me, this depends a ton on what your goal is. If you are an academic whose goal is to clearly describe and show control and manipulation of variables that exist in the natural world, I think it matters deeply what your three on the Likert scale is.
Versus, if you are trying to make business decisions in a fast-moving environment, where you’re simply trying to get signal of where to move next, you don’t need it. I know I’ll get some angry messages from some people. I’m not saying it’s not good and we shouldn’t try, but most of the time you don’t need it. What you need is signal in the right direction. And to be clear, what I’m saying is about running a full validity evaluation on whatever your measurement instrument is. It’s a little bit of, come on, people, we throw together Likert scales or whatever all the time.
And the best people, I think, use the ones that have been validated in other ways. But a lot of times we’ll just throw it together, because that’s kind of what you need in some parts of our job. In other parts of our job, you need strong validity and very, very strong rigor. And other times you need light signal. To tie it back, I think that’s what the top researchers do: they’re able to flow between those with ease, and they know when to do each one.
Henrik Mattsson · 30:29
Yeah, yeah, yeah. I mean, we still have this problem today. When you do five interviews or whatever, you can learn a lot from just looking at five moderated usability tests, even if it’s not statistically significant. So it’s not that unsound.
Why researchers should trust what they learned from users
Jess Holbrook · 30:46
Yeah. I mean, sometimes that’s what I’ll tell people. As a UX researcher, my superpower is I’ll talk to a stranger for an hour and a half without being the one in power. And very few people are willing to do that. And if you’re the only person who’s done that on the whole team, you have a deep insight that nobody else actually has. This is what we used to do, a little bit of a tangent, but hopefully helpful. We used to talk to junior researchers especially.
“As a UX researcher, my superpower is I'll talk to a stranger for an hour and a half without being the one in power.”
They’ll go to a meeting and somebody will challenge their research, and they’ll kind of back down, or they don’t want to speak beyond the data. And we used to talk to them and say, a lot of times you’re the only human on earth who has talked to 50 users of this product. Literally. Not the PM, not the director, not the eng lead, nobody else. You’re the only human on the whole planet who has talked to this many people who use our product. So your point of view, even if you don’t have data, is deeply valuable.
Your instincts are deeply built. If I took away the title, and I said it’s not a UX researcher, it’s a whatever, and they’ve talked to 100 people who have used our product, do you want to hear what their take is on this thing? Nobody in management would say no. Nobody would be like, “No, no, we’re good, they should stick to data, I don’t want to hear that.” It’s ridiculous.
Henrik Mattsson · 32:05
So yeah. And I mean, that’s the classic in management consulting. I think one of the McKinsey founders or something like that has this famous reputation of having pumped gas for a week or something to understand a gas station. That’s what management will go do when they need to. So I think that’s a very good point actually, in all of this, a good thing to remember. It’s easy to get lost in all this excitement about the LLMs and the AI and all this stuff that is very abstract, right?
It’s going to do something, it’s going to look at all of this huge data, it’s going to come up with something. And it’s actually a great reminder of something that is already true today, which is that just having people talking to real humans about real problems in real context is actually super powerful. So it’s like what you said about the emotional reaction of hearing that this technology is going to replace UX researchers. It’s kind of making us realize that, well, actually, there’s some stuff that we do here that is very valuable and unique, and it’s still the case today, even if we’re not using these technologies.
“Just having people talking to real humans about real problems in real context is actually super powerful.”
I think this is kind of an old problem.
UX research and its seat at the table
Jess Holbrook · 33:19
So yeah, I mean, it’s a gift. It’s honestly a gift, because it makes us examine what it is that we do. What is it that we’re doing here? And I also believe that it’s this. I think UX research, and design to some degree, but I think UX research has it the worst: I believe that we have this collective Jungian trauma of not being at the table. And so we take anything that could be status taken from us, and it’s like, oh my gosh, oh my gosh, we fought so hard to get here, and all this kind of stuff.
“I think UX research has it the worst: I believe that we have this collective Jungian trauma of not being at the table.”
And yeah, it’s a funny thing too. I feel like, in a lot of ways, UX researchers don’t know how to not be the underdogs. And so I would challenge people to say, no, you have a seat at the table. Maybe you do. But if you have a seat at the table, and you know the value that you bring, which is not your ability to transcribe things quickly, and now you have a brand new, hopefully powerful tool that will help you do those important things, that’s a good thing.
That’s power and influence. Those are positive things that the field has been looking for for a long time.
Henrik Mattsson · 34:43
That was the perfect ending to the first episode of this season. Wow. Yeah, I totally agree. I have nothing to add. Thank you so much.
Jess Holbrook · 34:54
Well, thank you for having me. This was really fun.
Henrik Mattsson · 34:56
Yeah, I would have loved to continue this. Listeners of this podcast know that this is how every episode ends. I could have gone on here for four hours, but we’ve got to respect people’s time, and these are endless discussions. So thanks so much for coming on the show. Is there any place where people can reach out or read your thoughts, or anything that you want to plug?
Jess Holbrook · 35:20
Yeah, you can find me on LinkedIn, Jess Holbrook. My Twitter, X, or whatever we call it these days, is @jessscon. It’s the same on Threads. Yeah, I’m on Threads, I’m using Threads. Check it out. I really like it. I’m not just saying that, I actually really like it. Yeah, I don’t know. You can search. Everyone has the ability.
Henrik Mattsson · 35:44
Has the ability?
Jess Holbrook · 35:47
Yeah. So go ask. Actually, that’d be really interesting. We should go ask ChatGPT, Bard, Meta AI, maybe Claude, a few others. See what they say.
Henrik Mattsson · 35:58
Awesome. Cool. Well, thank you so much.