Amelia “Mia” Glaese

Talk about everything, please. Maybe I'll give you a framing, and then would love as much honest detail as you can give. So my general framing for the article is how Codex became king. I want it to be an oral history of how you went from ... 'Cause for me, I was in the room with Alexander, Alexander, the night before you launched GPT-5. And it was the first time where I remember ... 'Cause I was at that point really psyched about Claude Code and not looking at the code anymore. And on the GPT-5 launch, it was very pair programmery, and there was this split between Codex for professional engineers and vibe coding inside of ChatGPT. And I was like, “This doesn't work for me,” you know? And then in like November, December, it feels like you guys have figured something out. And first with 5.3, and then the Codex desktop app. I and most of the rest of the team who are all Claude people started using Codex for pretty much everything. And so I wanna tell that story a little bit, and both tell it from the perspective of, just help me understand how things happened and why things happened the way they did and what you sort of figured out and what changed. And to the extent we can, I would love to be able to sprinkle in a little bit of actual artifacts from the time of, like, how do you make decisions, like Slack or strategy documents or whatever. Whatever you feel comfortable sharing, just send Codex off on it, and obviously get it approved. But I'd really love to tell the story in detail. So does that make sense? Okay, cool.

Sounds great.

Well, let's talk first about GPT-5 and that launch and where you guys were and what you were thinking and what you were feeling and as a way to set up maybe what changed. And maybe my perception of it is totally different from yours. I'm sort of curious about that launch and that model.

This is already a tough question because I was not responsible for the GPT-5 launch.

You're not. Okay. Well, where were you? Where were you then? Glaese: [laughs]

So yeah, maybe that's helpful context.

Yeah. Yeah.

So I've been working at OpenAI for, like, more than two years, but I've been doing a bunch of ... And for that time, I've been responsible for both the data that we use for our model training and alignment, which is more really about can we get models to do the right thing and what humans want them to do. And I started getting into Codex very early, basically at the beginning of Codex when it was a project of, like, four people and—

Like the separate model line. Yeah.

And it was this project of four people where we're like, “We need to have a coding agent kind of thing.” And so it was this somewhat, like, this bet where we were just like, “Okay, we're just gonna do this right now. If we don't do it, that would be a mistake, so let's just do it.”

Do you remember what triggered that, especially the idea of having it be a separate coding agent? Because I assume the idea of coding agents has been around OpenAI for a very long time, right?

I don't think it was ever intended to be a separate coding agent, right? It was intended to be our coding agent. But it was this very focused effort on just, can we get the best coding agent in the world—

Yeah, I see.

There, as quickly as possible.

Yeah, yeah. Okay, got it.

And so yeah, that's kind of that. But I think—

And when was that specifically, when you started?

That was ... I'm not sure how much we talk about this publicly, but it was a lot more than a year ago.

More than a year ago, yeah. So it was pre-GPT-5, 'cause I think GPT-5 was around a year ago. Yeah. And one of the interesting architectural choices there is a separate model line. Tell me about that decision.

I think it was very much focus on the coding agent that we wanted to build. And it was never meant to be a forever thing, right? But oftentimes it can help to just have a focus, grouped sprint on something. And that was really ... Codex was maybe the most intense thing we've ever done. Well, I've ever done, with a very small group of people. I think having everything in one model was the start of this plan to create more than a coding agent, right? It was like we were like, “Oh, okay, now our coding agent is very good. It's amazing, and what would make it better?” It would be better if it could do more things than coding.

You mean right now?

That was, like—

Oh. That, that—

Basically with 5.3, right? It was in the 5.3 blog where we started to make the model really good at other types of things. And we did that very deliberately because we were like, “How do we make Codex better?” Codex would be better if it could do more things, right? Just making it better at coding right now is not going to— The ROI on that is we want to be better, but the biggest blocker or the biggest thing that would make Codex better now is if it could do more things.

Mm-hmm. But I still wanna push on the separate model line thing. You said sometimes it's nice to just be able to focus on one particular thing. But one of the AI truisms is the bitter lesson of just the bigger, the more general the model, always beats a more specialized one. So that seems like it would be a different— Help me understand that.

Yeah. I think actually the truth is somewhere in the middle, right? A specialized model is probably always going to be better at specialized thing. Right? If you only care about one thing. But I agree, having everything together was always the ultimate goal. We were never planning to have this be a separate thing for forever. I've been doing this type of getting models good at working with humans for a very long time, and it is a very complex process. Especially if there's many demands on the models that we have, and so having some focus is helpful.

Mm-hmm. Yeah. One of the things that Alexander said, and I'm curious how it fed into—

And now we have everything together. And it's been great.

We're getting there.

Yeah. 5.4 and 5.5 have been some of my most favorite launches that we've done.

They're great. I mean, I'm using the current alpha right now all the time, and it's fantastic. It's really good.

Oh, I'm so glad. Yeah. We are especially proud of the front-end work because that was a pain for so long.

Mm-hmm. Yeah. Okay, so, one of the things— 'Cause I was talking to Alexander about Codex Cloud. Which was a big launch at the time. There was a big—

Big launch, yeah.

Big bump, and then it didn't really work as a product. And one of the things he was saying is, what the design considerations were for Codex Cloud was talking to the research team, which I assumed was you or someone that you know.

Mm-hmm. Yeah.

And thinking about, okay, he said, “We're a little bit behind on coding right now, but what we know is that test time compute scaling is gonna work.” And so building a sort of asynchronous style experience was the right product shape for what we thought we could be good at, on the research side. Can you tell me about your perspective on that?

I think it's a bit different. But I actually agree with everything that Alexander is saying. But I think in the future, we will have agents that do work for long stretches of time for us, right? We are already seeing that. I mean, I think you're in the channel where people are showing, “Oh, my model has been working for like 48 hours” or something like that, right?

Yeah.

And that's awesome. And that's the future I think that we were seeing at the time. And clearly, doing that on your laptop is not the most ideal thing. And so I think we were kind of approaching it with this more future vision where we are like, yeah, in the future this is clearly how things are going to be. And we didn't do this, let's go to the people where they are. Which is on their laptop. And so that was something that we then had to adjust quickly.

Mm-hmm. Who's we?

We as a Codex team.

Okay. And then tell me what changed in the direction and maybe from the research side on the research direction, because my memory of, for example, 5.1 and 5.2 was at the time I would use Claude Code all the time, every day, and then if I had a really hard problem, I would ask 5.1 or 5.2, and it would be very slow, very powerful, like a little bit just very literal. So tell me about the shift—

Which 5.2 Codex or 5.2?

5.2 Codex. Yeah.

I think this is where maybe your bitter lesson comes in, right? We were very focused on sort of coding. And that's what you—

That's what we got. Yeah. You successfully replicated a senior engineer.

So yeah. To me actually, I think 5.2 was a turning point where I was like, “Wow, I'm very excited about what this can do.” And what it can do for me. And it was this— It felt like it started to really accelerate a lot of people. And then 5.3 I think even more, but to me it was sort of like the curve started to be—

That's where you started. Okay.

And that was— 5.2 was when you were like, “We actually have to make this better, not just for engineering.” And—

Yeah. And so then after that we did 5.3 Codex, which was the first model where we put everything together and we're like, “Let's do”—

Oh, wait, tell me about that. Because when you say put everything together, what does that mean?

So 5.3 Codex was actually the first model where it wasn't just a coding— It's named Codex, but it was—

But you put other stuff in it.

We put all the other—

Yeah, yeah.

Knowledge work and that kind of— we started building the model for more than just coding agents.

But still called it Codex.

Yeah, we still called it Codex. I think we weren't ready. It was sort of like a step. We weren't ready for it to be like okay, is this a thing that is better at all things?

Yeah, yeah.

Than the models that we had.

Yeah. Well, okay, so one of the other truisms is whenever you train models on code, they get better at everything else. But what it sounds like you're saying is the model just got a lot better at coding or just being useful once you trained it on things other than coding. So is there something there that people miss?

I would say there's this thing where coding itself, code itself is so useful for everything, right? Code controls your computer. If you can write code, you can do all sorts of things with a computer, almost like all the things with a computer that a human can do with a computer. And people don't just do software engineering with their computers, right? People use their computers for all sorts of work or even entertainment, and so you can use code to make music. That's very cool.

Yeah, yeah.

And so coding is really this foundation that we believe is going to be helpful, not just for software engineers but for all sorts of work that happens at a computer. But it is a little bit separate from, say, knowing what a nice slide looks like. Right? So I can use code to actually create slides, but there's this extra step of, do I know what a good financial analysis is, and how can I present that for humans to consume.

What guides the research direction for the model? Is the product telling you, “Okay, here's our product strategy. Make a model for this”? Are you saying, “Here's what models can get good at. Make a product for this”? Are you talking about to users? How do you decide your research program?

My favorite thing about the way that we are working, the way that we also worked from the very beginning when we started working on Codex, is that it was very close together.

Research, product.

Research and product and product engineering. It's a small team.

How many people?

I mean, the very early versions were, like, handful. And then it got a bit bigger when we started to actually do sort of the external, when the cloud Codex launched, for example.

I see. Mm-hmm.

But at that time also, it was product engineers and researchers, they're just sitting next to each other.

As opposed to maybe culturally at AI labs previously, it's like researchers are in their own computer.

Yeah, you could be like here's your function. Research does their thing.

Yeah, yeah.

And then product looks at what there is and they build a product around it or something like that. And it's like, from the options that you gave me, I don't think either of them is true, right? I care a lot about product and maybe I'm not a product professional, right? But I can see what's possible with the models. I can be like, “Here's all the amazing things that we could build.” And also product people are like, “Wouldn't it be amazing if the model could do this thing?” And so we can sit together and we can share our vision and build something together. And we have a lot of strength in that where we can sort of parallelize that work where we can think about what do we want Codex or our models to be able to do. And then we can build the model for that and we can build the product and the harness and everything that we need for that together. Because we are thinking about this as a joint thing that we build together.

That makes total sense. I think that's really interesting and unique, and the reason I care about it is, I think that kind of a team structure is really important but is under-discussed. But what we talked about so far is when things work well together. But tell me about how you resolve conflicts or a specific thing where the incentives or the desires of someone who cares about research might conflict with either product or users or whatever and how you think about that.

I wouldn't say there was a lot of conflict. I think the team had this magical thing where people have a lot of mutual respect for each other's expertise. And so they're like, “Okay, I think we should be doing X, but you're the expert. What do you think?” You make the decision. And we didn't have necessarily one person who told everyone what to do, but as a team, people were really good at sort of being like, “You're the person who knows the best how to do a thing. You decide how we are going to do that.” So I think that's one thing. That doesn't mean that there isn't weird things that are quite different between model training and building a product. Where model training is a lot less deterministic, right? Every time we train a model, we go on this magical adventure. Where we're like, “How are we gonna do that?” And then we think about it. And then it's more like science, right? You have to do experiments. You try out a couple of things and then some of them work and some of them may not work. And then you kind of build on top of that. Whereas on the product side, I think you can think about what do you want to design, what do you want to build, and maybe sometimes there are some surprises. But you can sort of deterministically predict like, okay, this is how we are going to do it. It's probably, with extremely high likelihood, that's possible, like we can do it. Maybe it takes two weeks longer if we are missing something, but, you know, we build this. Whereas on the research side, we are just like, “Sorry.” Like, we tried, you know? Shipper: [laughs]

Not possible.

Or like we got 50%, but it's not—

Yeah, yeah.

Quite there yet. And so the mutual empathy on these differences, and how both engineers and researchers are really building a thing. But the way that we build and the complexities of building these are different. And this can be the other way around, too, right? Like researchers have never built like a giant, production-level piece of infrastructure. So it could be they're just like, "Can't you just do that?" And the engineers are like, "No." And so that's where the trust and the respect kind of comes in, right? Like we have to have that and be like, "That makes sense. You're the expert," you know? Shipper: [laughs]

Yeah, yeah.

Like, if you're like, "This is a hard engineering problem," I trust that. And if you are like, "Hey, from a research perspective, this is not going to work," or like, "We tried and something went wrong and we have to do another revision on this," that is not necessarily like a failure. It's just like, that's part of the process that you have to experiment and some things work and some things don't quite work the way that you hoped.

What are the qualities of a good researcher?

There's many different kinds of researchers.

Maybe your kind of researcher.

It's a very good question. I think a lot of what we do is very evidence-based, and you have to approach a problem with this curiosity and understanding that this is an experiment, like things might go wrong. You kind of have to take risks also, right? Because that's part of the process. If you're working closer to products and they are sort of this in sync with product engineering work, then being really well-calibrated on the risk, like what is the worst outcome here, what is the best outcome here, what should we bet on, making sure that we have a variety of, here's some things that we should bet on, and they are a bit more exploratory, and here's some things that we are certain they are going to make the experience of working with the model a lot better, and we should do those, too. Sort of having that balance is really important. Everyone at OpenAI works really hard, and so that's part of it, too.

We were on your launch schedule, so I know.

Yeah. And then for people on my team, it's also really important that they really care about the end product of what the user experiences once they are using the model. Instead of just being on the side and thinking about numbers — it doesn't matter if the users are like, "I don't like this," right? And so really being caring and being obsessed with the product and the users that we have is super important.

How do you think about the challenge of the things that you can quantify about the model that indicate that it's good versus just the vibe of it?

Yeah. It's very difficult. I think benchmarks matter.

Which ones?

Some of them. I think benchmarks matter. Not all benchmarks matter. Basically benchmarks, often when they are created for the first time and they measure something, they measure something very valuable, most of the time it's like, okay, models get like 5% or 30%, right? And at that point it's like, okay, there's like a very clear hill to climb. And if you improve on that benchmark it's probably really, really, it measures something meaningful. Especially if it's corroborated with vibes, right? Like, I think just trusting vibes is not that useful in research. But, like, if you—

Why not?

It's hard to measure, right? Like, we have to do many repetitions, and sort of just me trying the model and be like, "I think the vibe is good" — I'm missing a lot of what other people might be doing with the model, right? Like, that's kind of like I'm not the only user for the model, right? And so we would be missing a lot of use cases and preferences and what people need if we just did that. I think if benchmarks stop being correlated to vibes, it's probably there's something wrong. You can usually see it, like, okay, your benchmark hits like 80%. It's very hard to actually make benchmarks that have less than like 20% noise, right?

Yeah, I see.

So these kinds of things. So at that moment where it's like you stop really climbing, you're like, "Oh, it's like 81%."

It's not 100, yeah, yeah.

This is 81.5%. It's all within noise, statistically. But you also know like 20% of that benchmark problem is probably broken already. Like, at that point, the benchmark stops becoming... that doesn't mean it wasn't a good one. It actually served its purpose. And we have to continue to build new benchmarks that capture what people, that capture the value that people might get. But then having evaluations and having benchmarks is very important for research because we do need to be able to measure our progress in not just vibes.

Yeah. Well, let's talk about the new model. I mean, I have one that I'm using now. And I assume we're projecting that'll be out for the embargo. And I'm happy to hold this until whenever, so we can talk about it. Is that okay?

Yeah, we can chat about it. I think we'll still just determine what is the model still and share updates.

Yeah, yeah.

But yeah.

But there will be a new model.

We're testing, yes.

We've tested a couple different release candidates.

Yeah, I think we have a fairly good sense of—

Which was your favorite?

I think [redacted]'s the best. Yeah.

What were you trying to achieve with it? And how do you know that you've achieved it or whether you have?

Whenever we set out to build a new model, as a team we kind of decide what are the things that we really want this model to be better at. There's like unlimited things that you could kind of optimize for, and so we pick a couple of directions that we really want to focus on. And some of them might be like things that— I talked about this before, right? Like a bit more like bets. That could work out, could not work out. And some of them are like things that we think as just like, it's like this is super important. It's table stakes. I think like front end for example. Right? We're like very happy about this because we did set out for this model to make it significantly better at front end. And I'm very happy we succeeded. So there's like a number of these types of improvements that we pick, and then sometimes they continue, right? Like you never want to not be good at front end, right? That's our sort of—

Yeah, yeah.

Yeah. But we think about it as a team.

Are there any other dimensions that you care about for this model other than front end?

Yes. But did you find anything different other than front end?

Yeah. I think for me specifically, it has this combination of being really fast. And generally just like doing what I want for knowledge work tasks. So it writes all my emails. It's a very good writer actually. In a way that I think like Claude models— what we talk about a lot is Claude models are a bit denser and more literary. And this is actually like one-shotting like marketing emails and stuff for us. So it's—

I think, yeah, some things that— I mean, you often see that in the blog post, right? So things like token efficiency is something that we care about because that is like how fast do you get your answer? Like a lot of users care about that. We also care about the other side of things, right? Like how long can the model go? I've been having fun seeing people send me screenshot of like, "Ooh, it's been 147 hours."

Yeah, yeah, yeah.

And still going.

Yeah, yeah.

And I'm like, "Oh, that's great." Like it's working. And then we have this big moment of like merging— the bringing like the Codex sort of capabilities—

Oh, yeah.

to ChatGPT. And so generally like knowledge work and things that are important for other, like writing emails but also say like slide decks. Or making good sheets with correct formulas and these kinds of things are things that we paid a lot of attention to.

Is there something in particular that you're particularly proud of that people might underestimate how hard it was other than— I mean, the design one we talked about, but—

I think that one was hard.

Yeah.

That one is hard. It's like a lot of it is like taste, right? Taste is like we talked about benchmarks. Taste, super hard to measure. And so that makes it actually like a complex—

What is taste?

But yes. I know. I don't know.

Well, I assume you have to define it a little bit in order to hill climb it, right?

Yeah. We often work with experts, and that's why it's like amazing, right? Like the Codex team has Ed, he's our designer. He's great. And we can be like, "What do you think?"

Yeah, right.

And so I actually try to not be like this is what good taste is like, because I think— I think maybe I'm a little bit better than maybe the average researcher on taste. Maybe not. I don't know. It's hard to tell. But clearly I'm not Ed, right? And I want the model to be like Ed. And so I have to talk with Ed to figure out what does good look like.

Yeah. Anything on your mind that we haven't covered?

What is something about this model that surprised even you?

Hmm.

That's a great question.

I think this is such a hard question to answer because we're so close to it. Right? Like, it's not like we just produce... I mean, you've tried three models, right? But it's not like we just produce the models, like this black box thing, and then it appears and then I explore it. It does happen sometimes. Something that I thought was magical, not necessarily with this model before, but the first time that we did steering, where you could talk to the model and it responds. And it was able to do a lot of things that we didn't actually train it to do. We thought like, "Oh, yeah, it should give updates and stuff, and you should be able to correct it." But you can actually ask it a question and it will respond. And the answer— and we're like, "Oh, that is, that is like quite cool." But we are in general, we are so close to the model and we obsessively, every day, we look at how our model is doing today. So there's rarely things where I'm like, I'm—

Shocked.

Shocked. Yeah. I have had delightful experiences. Yeah.

So it's no longer a separate part of... So no longer a separate model line. What was it like to merge it? And do you expect to be doing more of these, like, kinda like break it off and improve it on a particular thing type moves?

Merging. It was crazy time. Shipper: [laughs]

That's going in the piece for sure.

No. It was actually— I had a great time merging it, because people came together and we really planned out the thing, and then we went into it together and built the thing together, and I was very proud of how we did that. And how... I mean, like, 5.3 Codex and 5.4 were sort of like the results of that, and I'm still very proud of what we did there.

That's great. And then do you think that the move of splitting off a model line is one that you'll continue to do?

It's not something that we are sort of... I mean, we have some— there's like GPT Rosalind and these somewhat separate models. We're not planning to do a new Codex separate line anytime soon.