No Priors 🎙️115: Personalizing AI Models with Kelvin Guu, Staff Research Scientist, Google Brain (TRANSCRIPT)

EPISODE DESCRIPTION:
How do you personalize AI models? A popular school of thought in AI is to just dump all the data you need into pre-training or fine tuning. But that may be less efficient and less controllable than alternatives — using AI models as a reasoning engine against external data sources.

Kelvin Guu, Staff Research Scientist at Google, joins Sarah and Elad this week to talk about retrieval, memory, training data attribution and model orchestration.  At Google, he led some of the first efforts to leverage pre-trained LMs and neural retrievers, with >30 launches across multiple products. He has done some of the earliest work on retrieval-augmented language models (REALM) and training LLMs to follow instructions (FLAN).

No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode.

Show Links:

• Kelvin Guu Website
• Google Scholar
• FLAN: Finetuned Language Models Are Zero-Shot Learners
• Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs
• ROME: Locating and Editing Factual Associations in GPT
• Branch-Train-Merge: Scaling Expert Language Models with Unsupervised Domain Discovery
• Large Language Models Struggle to Learn Long-Tail Knowledge 

Sign up for new podcasts every week. Email feedback to show@no-priors.com
Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Kelvin_Guu

Show Notes:

[1:44] - Kelvin’s background in math, statistics and natural language processing at Stanford
[3:24] - The questions driving the REALM Paper
[7:08] - Frameworks around retrieval augmentation & expert models
[10:16] - Why is modularity important
[11:36] - FLAN Paper and instruction following
[13:28] - Updating model weights in real time and other continuous learning methods
[15:08] - Simfluence Paper & explainability with large language models
[18:11] - ROME paper, “Model Surgery” exciting research areas
[19:51] - Personal opinions and thoughts on AI agents & research
[24:59] - How the human brain compares to AGI regarding memory and emotions
[28:08] - How models become more contextually available
[30:45] - Accessibility of models
[33:47] - Advice to future researchers

-----------------------------------------------------------------------

SARAH:

How do you personalize AI models? A popular school of thought in AI is just to dump all the data you need into pre-training or fine tuning, but that's costly and less controllable than using AI models as a reasoning engine against an external data source. And thus the intersection of retrieval with LMS has become an increasingly interesting topic. Today we're gonna talk to Kelvin Goo senior staff research scientist at Google who wants to make machine learning cheaper, accessible and easier. He's joining us to talk about the newest methods his team is working on in AI training and language understanding. Kelvin, welcome to no priors.

KELVIN GUU, GUEST:

Hey, thanks for having me.

SARAH:

Let's do a little bit of background. You started out studying math but pivoted to natural language. What encouraged that switch?

KELVIN GUU, GUEST:

Yeah, I actually had always wanted to get into tools that would help people learn more easily and find information more quickly. My motivation for going into the math was really to build a good foundation for getting into many of those deeper questions and I would say that also motivated my venturing into statistics. So my PhD was actually in the statistics department, but I quickly then migrated over to the NLP group at Stanford where I just learned a lot and had a great time.

SARAH:

You chose to go to Google after your PhD. You've been there since 2018. Like how has your focus or work changed during that time?

KELVIN GUU, GUEST:

Yeah, I would say at Google this was really just the dream place for me to deepen my focus on building tools that help people find information better. And at the time that I joined, it was also just an extremely productive period where lots of new ideas were coming out. The group that I'm part of right now, which is now part of Google Brain at the time, was looking into very, at the time early ideas on pre-training language models, which eventually became Bert. And that work just set, opened up a lot of frontiers for me, both in terms of product work at Google and inspiring new research ideas. I think one of the things that became very apparent early on when playing with Bert was unlike all the prior generations of models, it had a large amount of world knowledge that we didn't deliberately encode into it. It wasn't in, you know, the fine tuning data, it was all within pre-training and that that has just many, I think very obvious use cases now. But that also inspired me to and my colleagues to start thinking about retrieval augmented models and getting even more knowledge into them. And I'm sure we'll be talking about that more here.

SARAH:

Yeah, let's do that. Let's talk about realm, which our listeners was a really landmark paper in the field. In the paper and your talks about your model, you described the limitation of AI in domain knowledge or specialized knowledge that became retrieved and represented more accurately What motivated the paper?

KELVIN GUU, GUEST:

Sure. Yeah. So there are a few different things that can bring people to this retrieval, augmented modeling literature. One of them was kind of our original goal, which was to increase the memorization capacity of these models. A second goal that you might come to this from is for modularity. So you might imagine that you have different data sources and you'd like to be able to swap one in or take one out the same way you could do with the database. There are just so many business applications where that's helpful. A third is anytime you're dealing with a very timely application. So there's new information arriving daily about say a sports team or any other type of event, you really want to be able to incorporate that quickly without retraining. So these are some of the common ways that people arrive in the space and it's a very natural thing I think now to think about, well I don't want to retrain and yet I've got all this information out there and it's human interpretable, it's text, how can I bring that in? That's kind of what brought us to that in the first place and since then we've encountered many interesting challenges On top of that idea that I would say even in, in the sort of systems you see today, these tool using models that issue Google searches or provide citations, they still face some challenges in terms of fulfilling those original promises.

ELAD:

Could you talk a little bit more about the realm approach in architecture and then how that actually leads to some of these solutions? And you know, maybe more generally, I know that people have been focusing on sort of mixture of expert models and things like that, like Moes and so it feels like uh, derivative of some of this work in some ways. And so can you just talk about how realm was initially set up and then how it's evolved? The

KELVIN GUU, GUEST:

Basic idea if we were to describe it is let's say the user provides some sort of input to the model. I'm gonna describe the inference time way of using it and then the pre-training that was involved as well. So the user provides some sort of input to the model. The model is then able to encode, embed that information into a dense vector. So this is a vector that will be situated in a larger vector space where we've also embedded other documents. So any other corra we have on the web are converted into dense vectors as well. And a nearest neighbor search method is used to find the closest documents to the embedding of the input. And once you have those documents, you can then encode those as well in some form, allows the model to then use cross attention over those documents. And based on that it's then able to make a prediction.

So the way in which the model depends on that information is through the cross attention and there are multiple different ways to train the cross attention to kind of use that information. And in the realm paper in particular, we tried to learn that from the language modeling task itself. So I guess for our listeners, many of you are probably familiar with the fact that language models predict the next token. So given a sequence of existing tokens, you predict the next one. At the time of the realm paper we were doing something called mask language modeling, which was popularized by burn. So a very similar variation. Instead of only seeing kind of the tokens leading up to your current token from the left, you would also, we would instead take a whole piece of text and just blank out a few words and train the model to fill it back in. And the way realm was being trained in this method was we would say, okay, you can go and you can retrieve some documents and if those documents help you fill the blank better learn that those are useful. And if they don't help you fill the blank better, then learn that these are not useful and don't retrieve them going forward.

SARAH:

Zooming out in terms of thinking about when to retrieve and when to train, like do you have a framework for this? How do you think about this for different domains or you know, the idea of vertical specific foundation models?

KELVIN GUU, GUEST:

Yeah, so I think there are clearly a lot of different tasks where we're seeing you don't really need retrieval. So for example, anything that's common knowledge on the web. More and more we're seeing that if you build a dense model and that information is represented in enough quantity, these things with scaling improve. So it's actually been a case where I think more and more benefits of a retrieval augmented model are for modularity for personal information that you might not want in the main model for adapting to say like an enterprise customer that has special information. I think that's a much more compelling application at this point. There are some papers that I can point to later in the show notes that kind of show how memorization scales with the number of times something shows up in the corpus and you have to hit a certain frequency before these models can kind of accurately remember things.

ELAD:

The basic description would be if you read a bunch of documents over and over again, you're gonna remember more details out of them. Yeah, or if you see the same sort of sentence or words over and over the same concept. And so as these models get bigger and bigger, you're basically incorporating more and more knowledge just inherently. And what you folks really focused on is the model may be a little bit smaller and so how do you make sure that you can actually retrieve information in a performant way relative to all sorts of domains of knowledge without having to build a giant model?

KELVIN GUU, GUEST:

Yeah, and I would say Eli, you mentioned mixture of expert models earlier and I think there's a, there's an interesting overlap between mixture of expert models and retrieval augmented models. So there's been a recent mixture of expert approach which is very easy to describe. So I'll be describing a a paper called branch train and merge. You take your corpus of pre-training data and you just split it up into different sub corra and you just train a language model on each one of those. And then when you want to use the language model at inference time use sort of use some kind of routing mechanism or weighted voting mechanism to choose among those experts. And so in that case, each of the experts is responsible for part of the document corpus. And if you think about a retrieval augmented model, instead of having that intermediate layer of experts, you're just going to individual documents.

And so it's both of these approaches try to do the same thing but they don't partition the space in the same level of granularity. You have one approach that's very granular and the other that takes broader strokes and the kinds of use cases where one or the other is best actually differ as well. So if you just want to get very precise factoid information like what is my wifi password retrieval augmentation is going to be very good, but if for example you are trying to adapt a language model to a new enterprise and they have some kind of specialized language in their code base that nobody else uses, then just retrieving any single piece of code from that is not gonna teach you how to use the language. Whereas an expert that's been trained on it is going to be able to adapt more quickly. So I think both of these approaches are gonna be useful in different places.

SARAH:

Is there a a general way you think about modularization, like why that's important overall? I think like you just described a situation where controlling the partitioning gets you better quality for a specific task, like answer questions on this, you know, proprietary language you have in this code base, right? Why else is modularity important?

KELVIN GUU, GUEST:

Yeah, I think modularity is going to potentially have to happen due to the different, like the sort of way these models are developed. So it's not always going to be the case that any organization can invest the resources to build a large language model from scratch. So they're going to be wanting to build on top of something that exists and they want to spend minimal effort to adapt. And that's where the benefit of modularity really comes in. I think it's also interesting to discuss the different ways people are adapting language models. Now if we look at the trend over time, as late as in 2017 people were training specialized models for specialized tasks. So you might define your own special neural network architecture just for sentiment classification. And then we switched over to having a pre-trained model, everybody sharing the same architecture but different fine tuning data. And then very recently we have this sort of instruction following set up, which I was very thankful to have been involved in some of that research at Google.

SARAH:

Just for our listeners, can you explain what instruction following is? Sure,

KELVIN GUU, GUEST:

Yeah. So I'll describe some of the work we did in a paper called flan, which was one of the early papers to show that you could train large language models to follow instructions. So the high level idea is you start with a pre-trained language model and you collect a hundred different tasks. These could be sentiment classification, question answering summarization, and for each of those tasks you annotate each example with an instruction saying what it is that needs to get done and perform multitask training overall those tasks. And that had been done many times before in in other research as well. But what we were able to show was that if you train on a hundred tasks, if we then show the model a new 101 task, it will be able to adapt to that without having seen that particular task before. And that was very surprising at the time many people thought we would need thousands of tasks to make that happen, but because that worked out, things like instruct G P T came out next followed by things like chat G P T and this sort of adaptation is just opens up the space of who can use these models much more rapidly.

Suddenly you don't need specialized fine tuning data, et cetera. But there's still a new frontier of tasks that you can't just get out of prompting. To give an example, like many people are familiar with the problem of large language model hallucinations. They will sometimes say things that are not well supported by external sources and you can't just tell the model, please don't make things up and expect that to just solve the problem. There's so many other things you can say with instructions, but they won't be followed and that's where I think existing adaptation techniques like fine tuning and reinforcement learning are still relevant. It's also still relevant for perhaps personalization or other kind of soft traits that are not always easy to communicate.

ELAD:

How do you think about other approaches people have talked about, like updating model weights over time versus a training or other approaches to kind of have a model effectively learn in real time or adapt in terms of its knowledge base?

KELVIN GUU, GUEST:

Yeah, so when you say updating model weights in real time, I think I'm kind of thinking of continuous learning methods. Those I initially thought would've been more popular in kind of production grade settings, but they come with a maintenance cost, which is if you have something updating live, you don't have the opportunity to validate and check that everything is going well as easily. And so I've been kind of surprised by the fact that they're not actually as popular as I thought that a weekly release cycle or a monthly release cycle is often fine. So for that reason I personally haven't been looking into that line of research as much, but the idea of learning over time and continuously building up more knowledge, I think that's quite relevant. There are a variety of techniques out there for doing that. One example might be prompt tuning, so just to give a little bit of background on that, prompt tuning is where you have a densely parameterized large model that is frozen, the parameters are frozen, but the prompt that you give it is not natural language but rather the vectorized embedded form of that natural language and people use gradient descent on that vectorized form to adapt the model's behavior.

So you can produce a prompt that's good for this week, next week you need something else, you can update that prompt. So the large model is not changing, but a small part of it is updating over time and you have the ability to plug in and plug out different prompts as needed.

SARAH:

You also just published this paper called influence which tracks how much smarter your model gets after consuming each example. Can you talk a little bit about the motivation for this work and then broadly I think the area of like, I mean intersecting with a bunch of your other work, but explainability with large language models is an interesting one. Can we solve that? This seems like something that people generally approach, you know post hoc.

KELVIN GUU, GUEST:

Yeah, so this influence paper is part of a emerging area of research called training data attribution. So the problem there is not specific to large language models, it's really general to any machine learning model. The question that it tries to answer is, given that my model has done a certain thing, which training examples taught it to do that thing, this question goes back all the way to statistics and linear models. But for large language models it's especially difficult to answer the question because of the kind of long training process that it goes through and the many forms of generalization that these models have. So in this influence paper we take a somewhat new approach to the problem. So we say okay, how would you really justify that this training example was important for doing a certain thing? Well if you had infinite compute, what you would do is you would take that example and remove it from the training set and retrain your entire model and see if the behavior changed.

That's obviously too expensive for most people to do. So what we come up with is a very lightweight model that simulates the training process. It observes a few training runs you've done, tries to estimate the effect of each of the individual examples and then you as a developer can sit down and say, imagine that I had done this training run, what would the result be? Any sort of simulation that you do is always gonna come with approximation errors. And so this is still very early work on modeling these things, but I think it would have very beneficial effects if we could figure this out. One of the best use cases is if you know what training data is valuable, you can go out and get more of it. It would also help us understand what these large language models are really capable of. So we've seen impressive feats from large language models recently like simulating a Linux terminal or being able to draw like a unicorn in late tech or something like this <laugh>.

And the question that always comes up for us is, was there something in the training set that actually looked like that and we just can't find it without being able to track that down. We'll never know if these models are generalizing or just kind of cleverly patching together what they know. What is the most surprising thing that uh, language model has done that you didn't expect? Linux terminal one did impress me when that first happened. I would have to say also sometimes their ability to do visual reasoning in pure text is also quite surprising to me. I've seen language models use SVG notation to draw pictures and that just seems very emergent.

ELAD:

I guess one of the other areas that a lot of people are focused on right now or talking about at least is different concepts of memory in the context of LLMs or other types of models. What areas you think are most interesting to explore right now and you know, where do you think things will evolve over the next year or two?

KELVIN GUU, GUEST:

Yeah, so there's been a certain line of work that I've been really impressed with, which is what I'll call model surgery research. This is on editing model parameters to change their knowledge of certain facts. So there was a paper called Rome out of I think the, I think it was a lab in M I T. And the basic thing that they showed they can do is let's say the model believes that the Eiffel Tower is in Paris as it is indeed they show that they're able to edit certain model weights to make the model believe that the Eiffel Tower is in Rome. And not only are they able to do this, but any sort of facts related to the Eiffel Tower or related to Rome are also updated. So the model doesn't just remember this one fact but propagates that knowledge throughout the rest of the network.

So for example, if you ask what is the tallest structure in Rome after making this update, the amount will also say Eiffel Tower. And this is based on a bit of theoretical work showing that the weight matrices inside large language models can be thought of as lookup tables to some extent. So we know that large language models perform a large number of matrix vector multiplication and the matrix is the lookup table and the vector that is multiplying the matrix can be thought of as the query to that lookup table. And so by updating that matrix, they're able to change the knowledge. I feel that this is a very exciting area for research because it provides a different kind of modularity from retrieval augmented models or mixture of experts, one that actually allows a kind of generalization. That's very interesting.

ELAD:

Yeah. We've talked a little bit about modularity and memory. What other areas you think are major components that people need to work on in order to achieve true machine intelligence or to push the field forward?

KELVIN GUU, GUEST:

Yeah, I think since you say kind of true machine intelligence, then we're kind of going the full i I presume sort of AGI level goal. And I think one thing that's very obviously in the public discourse right now is increasingly autonomous agents, I should say by the way, everything that I'm discussing here, it's all personal opinions and I'll be deriving everything from things I've seen in the public discourse, nothing related to any of the work that I'm doing at Google, but I do see this desire for autonomous agents and it seems like the going approach right now is kind of inspired by the touring machine. So we have these models that can operate over a relatively small context with a prompt and folks are trying to take long horizon tasks and reduce them into these smaller profitable tasks. And the component that they're using to bridge the short time horizon into the long time horizon is a memory mechanism, much like the retrieval augmentation we've described where the agent has the ability to store its memories either in text or in a vector store.

And I think more research is definitely going to be needed there. It's a very ambitious goal, but this way of decomposing the problem leaves I think potentially many gaps. So for example, let's say there's an agent pursuing a task and it breaks that task down into sub goals that it defines for itself, it will then write one of those sub goals down into its metaphorical sticky note that decides what to do. And if one of those sub goals is wrong, suddenly that is a canonical goal in the model. And how does it realize to reverse that decision? You could argue that, oh we just need more prompting to get the model to think about every aspect of the problem. Us as humans, we certainly do prompt ourselves quite a bit to think about a problem from different angles, but a lot of what we do is also instinctive and those instincts are developed over time.

So just to give a psychology analogy, there's this concept of chunking where people have shown that a human cannot really easily remember more than seven unique items at a time. But if those seven unique items are then broken into chunks that they can attach some sort of metaphorical or otherwise significance to, they can remember much longer sequences. And I think the same thing kind of happens with humans where initially we do something through explicit reasoning but eventually we distill that down into instincts. That's something that seems to be missing from these autonomous agents right now. Everything is explicit reasoning and at some point that might not scale or that might become brittle. So that seems like a very exciting area.

SARAH:

Do you think that's architectural with how people are building agents right now or if it's a scale thing, right, because the agents that people are like talking about working with an open source are tiny.

KELVIN GUU, GUEST:

I think that it is a bit architectural right now. So just to give an analogy, like in 2018 when question answering systems or reading comprehension systems got really good, everybody tried to reduce a lot of problems into question answering and there was just this dash to just throw that into different components. And now I think now that we know that instruction following works, again folks are building these structures in that way, but it's quite possible that fundamental advances could come along and this kind of workflow hacking or workflow engineering could go the same way as feature engineering or or specialized architectures.

ELAD:

Yeah, it definitely feels like it's almost like a patch on something that should work in a smarter way almost. And to sort of create a concrete example for our listeners in terms of you know, an agent action, it would be something like, hey we're going on a, we wanna go on a trip to Hawaii, could you find the best hotels that are kid friendly? And then book that and then book a series of activities for us to do during the days that we're there And you know, on the flight we need you to book things that you know that we like in terms of aisle versus window or whatever and then the agent would go and do it and so it could keep like a list of all the different tasks and then come back to those lists. So that's what people are kind of hacking together now. They're basically creating a place where the model can go back to like kind of check off the fact that it did something and then it moves on to the next task on the list.

KELVIN GUU, GUEST:

Maybe just to run with that example a little bit so you can kind of specify all the things that you want about your vacation, but there's also another aspect of this where if you had an an actual travel planner, they would gradually learn over time your preferences, not through your explicit instructions but just your feedback, your responses. And that kind of learning loop I think doesn't appear yet in existing methods. I think this becomes even more apparent when a autonomous or semi-autonomous agent needs to interact with more complex tools. So simple tools like maybe issuing a search query, they're very short interaction and it's almost stateless. You just put something in, you get something out. But if you're actually operating anything of significant complexity, there's a learning curve that anyone has to go through. If somebody sees a particular spreadsheet that an organization is using for certain purposes, you need to implicitly understand how that spreadsheet is being used, what is okay, what isn't okay. And that I think is also missing and can lead to a lot of brittleness.

ELAD:

A lot of the things that you've described I feel like are ways that at least a subset of neuroscientist believe the brain works, right? It's very clear we have modular parts of the brain that are involved with vision or information processing or certain types of emotion. So we know that the, at least the human brain is very modular. There's obvious different notions of memory, both short term and long term in terms of how you store and retrieve information. And then it seems like there's enormous amount of things that are just hardwired, right? Like the number of language related tokens we actually see as a person is quite small. The number of words that we're exposed to by the Tamar toddler who can speak seems to be small relative to all the training that happens for a large language model. In terms of the data set, are there other aspects of the human brain that you think people are kind of likely to copy or implement into these types of models in the future or ways that you think these things will just fundamentally differ?

KELVIN GUU, GUEST:

Thanks. Yeah, that's a really great question. One that I was thinking about recently, and this is again very speculative, but when we think about autonomous agents, we worry often about their capacity to do something that is irreversible to get themselves into a state that they can't get themselves out of. And in human behavior there are certain things that we have an instinct to avoid and that is usually driven by an emotion like fear or uncertainty or anxiety. And funnily enough, in the reinforcement learning literature, there are people who have proposed research concepts that they call fear where the model learns to identify states that it can't back out of. And I suspect something along those lines may also be helpful. They may also introduce the kind of corresponding issues with fear that humans have as well. And that's open questions,

ELAD:

It's basically adding a limbic system or something. Yeah,

KELVIN GUU, GUEST:

Yeah. I also think though that this kind of memory consolidation aspect of human learning where there's a deliberate pruning of what you've thought about that we don't have any analog for that, at least in the early systems we're seeing right now.

SARAH:

I think it's also interesting to think about whether or not we've already or will continue to introduce like biases that humans have or new ones, right? And so like this is an analogy, but you can think of these models as having like a, a consistency bias if you know, you think about some of the tasks that they're incapable of doing today correctly math. And one of my favorite examples of this is you ask a, a model is 26 or 21 larger might get it wrong and then you ask the model to count down from 21 to 26 and it counts infinitely because it assumes that you know it's gonna get to 26 at some point <laugh>. And so it's like this extreme version of like the cognitive dissonance that humans can have, but you know, there are mechanisms to get yourself out of that too.

KELVIN GUU, GUEST:

Yeah, I think that's a good example of that. A human at some point will introduce some kind of introspection step like wait a minute, I've been counting for 10 hours. And I think that's often missing as well. Like people try to patch that over with chain of thought reasoning. So they'll do some reasoning then they'll say, is this reasoning right and repeat that loop? But does it have to be as explicit as it is right now? I don't think so. I think we have associative mechanisms that ping us after a certain amount of of time on these things.

SARAH:

I just keep counting

KELVIN GUU, GUEST:

<laugh>, could you explain to us how models are becoming more contextually available as well? Sure, yeah. So at the moment I think there's, there's this paradigm where you sit in front of a chat interface and if you have a problem you think really hard about all the information and context around it and you try to boil it down into this wonderful prompt for the model and that actually requires a lot of brain power. So there are a lot of cases where people will just not think to use the model because of that extra barrier. And I think there's clearly an ambition in the work that we see publicly to make these models more available and more context. So you could imagine just being in the browser and trying to get something done and after 20 minutes you just ask your model like why can't I get this done or how do I solve this? And it should ideally look at everything you've been doing up until this point and infer your intent and just help address that. That seems like a pretty open question that isn't fully addressed.

SARAH:

One more research question and this is a very general one, but you you know have broad ranging interest in the field, like what makes a good knowledge representation?

KELVIN GUU, GUEST:

So when I started doing research I was actually very involved in knowledge bases. I thought those were gonna be the future because they have this wonderful canonical representation of things you can do reasoning over them.

SARAH:

They're a very interesting area to explore if you think retrieval will continue to be important, right? Cuz you might have these representations for models to go retrieve against,

KELVIN GUU, GUEST:

Right? Oh yeah. To be clear, they are useful for many applications in many cases a database, you want that to be your source of ground truth and that's perfect. If you want broad coverage, you ideally want a mechanism that's more expansive than that. One thing we've noticed is that it's very hard to get a representation that is a hundred percent canalize in the sense that information is only represented in one place and not anywhere else. So with text for example, a single fact is in multiple places, but if you want to edit a fact, you need to go to all the places where that fact is mentioned and change it. So there always seems to be this trade off between centralization and coverage. And this is kind of why I mentioned the earlier model surgery work because I think that's some of the first work to try to keep all of the coverage while also allowing some degree of centralization. If I were kind of advising a student or something on looking into knowledge representations at the moment I would say there's a lot of momentum on, on dense models continuing to capture more and more of the different applications. And so if there is a way that you could work with that dense representation to make it more manageable or more controllable, that would be preferred.

SARAH:

Zooming out from your research work, I think a question in society and in the tech ecosystem recently has been like how important is it that training is accessible and that, you know, we don't just have a small set of model a p I providers, but models in open source and a richer set of providers. Like how do you think about that issue?

KELVIN GUU, GUEST:

Yeah, so I'm very interested in recipes that allow people to adapt things easily and it's not yet clear what the right interface is. So training has always been challenging for many reasons I think to do training. Well yes, there are scripts and architectures that are ready to go, but data quality is a really big aspect of good training. And so there is a question of will that be the most common format because curating data is not easy at the scale of maybe 10 or 30 examples. It is easy and maybe that is a sweet spot for a lot of users, but again, I think it depends a lot on the application. So we could think from the perspective of an individual, like if someday for example, people had their own in-home language model, which of course doesn't exist right now, you might even say like, oh these are our family values, we want the language model to have those values. Would that be through just writing down your kind of family values on an instruction prompt and just saying, please follow these everywhere or is it going to be through repeated feedback where you talk to the language model and you say, no, I don't think you should say that. It is not clear to me how that will go

SARAH:

Do, as I say, not as I do. Seems easier, <laugh>, <laugh> in this case.

ELAD:

I think that will train the wrong behavior societally unfortunately. Just given how people tend to act against that. What you described, I think in one form of that is what some people talk about is like constitutional ai or at least that's the philanthropic centric view of it. Do you think approaches like that will propagate in the future in a big way? Or do you think there'll be other approaches to sort of get the behavior that you're talking about?

KELVIN GUU, GUEST:

Yeah, so I think there's a strong incentive to make that happen. So the folks who are providing large language models, they want to make their approaches as easy to use as a possible. And so anything that can go into prompting, it seems to me that people will try to do it because that gives the broadest reach. You don't want to just say, oh you have to fine tune this and we're not gonna help you in any way. So I expect people will try to push that as far as possible. And you could even say that to some extent, just following on what we did for instruction tuning, you can fine tune the model to be more and more responsive to different types of instructions. So if you can anticipate that someone is going to say, I want you to follow these family values, well then at training time we can prepare for that. And so it, it's a game of how much can we anticipate that people will want to adapt.

ELAD:

Yeah, I like the term family values for that cuz it feels very sort of grounded and we're all in it together and you know, I think that's a really nice framing of it versus some of the things that feel more like we're enforcing something in, in a way that may not work in the long run. So it's a, it's an interesting turn of phrase.

SARAH:

Kelvin, last substantive question. Do you have any parting words or advice for those who, you know, want to become productive research scientists in the future? You've created a bunch of really incredible work.

KELVIN GUU, GUEST:

Oh sure, yeah. I think at this time in particular, I encounter folks who fall into like two different camps. Sometimes there are folks who will say, oh my gosh, like everything is happening and even if I sit here and do nothing, it will all get done and I just don't know what to do. And then there are others who feel that, oh, there's too many things to do, I cannot get to all of them in time even. And I think there's a couple different answers. One is if you move the goalpost for yourself, so a lot of researchers, they kind of had set what they felt were ambitious goals, but then given all of the recent progress, those goals now need to become more ambitious. So if you think farther out, you'll find that there's so many things that are not solved, like making agents more autonomous or safe or understanding which training examples caused a language model to do something.

There's a lot of headroom there and especially I think now if you pay attention to what's happening in the product world that can provide so much motivating, uh, use cases and even, you know, if you come to Google or something, there's so many things there to look into. That's one thing. The other thing I would say is there's sort of also folks who ask how much technical knowledge is still important at this stage for doing machine learning? Like has it become kind of an era of just sort of prompting language models to do different things or just curating data and scaling things up? And I would say there are still, well obviously it's great if you can build valuable applications using a simple skillset set. And I think there's an amazing amount of creativity that can go into that. But if you're also looking for something that is more of a technical challenge, there's still so many unsolved problems in terms of understanding how these models work. And ultimately we do need to get to the bottom of that to improve things like safety and predictability.

ELAD:

I had one really random question. Go ahead. Um, yeah, so I was basically thinking back to like a year ago, if I were to try and hope for a career for my kids, it would be to go into computer science. And as I think ahead, you know, 15 years, 20 years, however long ahead, at some point it feels to me that machines will be writing better code and be doing better computer science research than we will. And so I was curious your thoughts on that and you know, over what timeframe Is that a real concern or is that even a real concern?

KELVIN GUU, GUEST:

I love that question. I'm sure many people have asked that. I feel that there's maybe been a shift in what kind of skill is valuable. So at a certain earlier point in time, having technical proficiency was a huge differentiating factor. And if you didn't have that, you just couldn't pursue certain ideas. Whereas now more and more the technical proficiency part is we're being helped out by large language models to, you know, right, in programming languages we don't even know. And so it seems like the, the differentiating factor now shifts more to problem formulation or creativity in identifying problems and being able to frame them in a way that you can then reduce them to a technical problem. So I sure definitely don't know what will be happening 20 or 10 years from now, but being creative and solving different kinds of problems will always still be useful. Another thing that has come up in some discussions is that even if the large language model is autopiloting a lot of work for you, someone still needs to validate that and that may still require a great deal of technical skill unless you prompt a large language model to validate things as well. But then who's validating the validator? So at some point, technical skill will still be needed, whether it will be as broadly needed, it's anyone's guess.

ELAD:

Yeah, it's kind of an interesting concept because if you look at the AI gaming literature, it feels like at every step, ML system works better at chess and people are like, well it'll never solve Go. And then a few years later it beats everybody at go and they're like, oh, well I'll never beat people at poker. And then it beats people at poker and they're like, oh well diplomacy, I mean that's, you know, manipulating people and causing them to act in. It's never gonna do better than people in diplomacy. And then, you know, obviously Nom Brown and other people's work on diplomacy suddenly shows that machines can win it, the a. And so I sometimes wonder about the concept of creativity and problem generation and all the rest of it relative to that same sort of slope of capability and learning. And so that was in part the basis for the question is at what point does the advantages of being human run out and then how do we think about that from the perspective of future decisions for our kids or even what are the things that they start to learn now relative to what that future will be?

And it's obviously very uncertain, so it's very hard to tell. But

SARAH:

Alad and I were just talking about how robotics for range reasons is still significantly behind. And so his kids are gonna be woodworkers <laugh>. That's the answer.

ELAD:

Oh gosh,

SARAH:

Yeah. Something in the physical world,

ELAD:

<laugh>, I don't know. We'll see. I hope they can still be knowledge workers, but we'll see. I mean, physical stuff is great too, but it suggests a lot about humanity's place in the, uh, universe if that were to be the case.

SARAH:

On that note, thanks Kelvin <laugh>, this is an incredible conversation. Thanks for joining us on the podcast.