Transcript: 'He's Building AI for the Person You Want to Become'

The transcript of AI & I with Alex Komoroske is below. Watch on X or YouTube, or listen on Spotify or Apple Podcasts.

Timestamps

1. Introduction: 00:01:45
2. Why chatbots are a feature not a paradigm: 00:04:25
3. Toward AI that’s aligned with our intentions: 00:06:50
4. The four pillars of “intentional technology”: 00:11:54
5. The type of structures in which intentional technology can thrive: 00:14:16
6. Why ChatGPT is the AOL of the AI era: 00:18:26
7. Why AI needs to break out of the silos of the early internet: 00:25:55
8. Alex’s personal journey into systems thinking: 00:41:53
9. How LLMs can encode what we know but can’t explain: 00:48:15
10. Can LLMs solve the coordination problem inside organizations: 00:54:35
11. The under-discussed risk of prompt injection: 01:01:39

Transcript

(00:00:00)

Dan Shipper

Alex, welcome to the show.

Alex Komoroske

Thanks for having me.

Dan Shipper

So for people who don't know you are the co-founder and CEO of Common Tools, which you'll describe for us. You are also previously the head of corporate strategy at Stripe. And director of product management at Google. And I met you I think six months ago. And the thing that really stood out in our conversation, aside from just you having so many interesting ideas, is you're this really interesting systems builder. You think a lot about personal systems, work systems, things that help you get more done or think differently about the world. and I think people like you who have that kind of obsess, obsessive mind with sym with systems is something, something I recognize in myself and also I think are people who are, have a lot of interesting things to say about this new AI wave. because I think it's sort of steroids or catnip or whatever, whatever you want to say for some kind of tolerance for people like us. So I just wanted to, I'm excited to chat with you and excited to hear both what you're thinking about now, what you're building and what kinds of systems you're playing around with personally.

So why don't we start with common tools? Because I think there's something interesting about you, like what you're building, because I think it speaks to the perspective you probably have about this larger AI wave. Tell, tell us about it.

Alex Komoroske

Yeah. The way I think about it and it's hard to describe because we're trying to build a new kind of thing that's only possible in the world of ai. So when I describe it, people say, oh, you're being coy. It's like, no, I'm telling you what I'm, we're building. It's just, it's a new kind of thing. It's hard to grab onto, I think of it as a fabric for your digital life and by cove, that's an old word that I hadn't heard before, but I think is a perfect fit for it. You are active in the system and so is this emergent intelligent process, your private intelligence powered by LLMs, and those can be covered on the same substrate and adding detail or adding little bits of software or connecting up things. And I think that really unlocks or unlocks the power of LLMs.

Dan Shipper

I love that word. And it's interesting because I think we're finding something similar for some of the products we're incubating internally. It's like right now the paradigm is you SendLink to Chet and it sends something back. Maybe now it sends something back after 10 minutes, which is like a new thing. But it's not like you're not working in parallel. It's not like working in the background and then being like, I had this idea. And we're starting to, as we build our own sort of agent systems inside of Every for doing your email or for writing or whatever, we're finding that having something that. working while you work with you and being a little bit more proactive is actually a really interesting next paradigm.

Alex Komoroske

Yeah. To me, I mean, if you say, okay, we have LLMs, what's the first product in 10 seconds? You go chat bots. It's the most obvious extension and everybody is focusing only on chat bot. Will chatbots be important? Yes. But chatbots to me feel like a feature, not a paradigm. And they are so like they're, they're like this surface, but it's just append only. I append a message. It depends on the message. I append the message. It depends on the message and literally back and forth. It doesn't have a way to have multiple for me in a row. And chatbots are amazing for starting any task because there's no structure. You just say whatever you're thinking and it helps to still and respond. But for long lived tasks, you need structure. You want there to be something where you can glance over and a little cubby hole where you put certain information. I find I have, you know. I do dozens and dozens and dozens of chats with ChatGPT and Claude today. And I'm just swimming in all that context and all the, all the chats. and so I'm using it to do important things. And also, I'm just completely lost in it because it's only this chat bot kind of pen only kind of paradigm.

Dan Shipper

You have this really cool thing called Bits and Bobs, which is this real time Google doc that you just have been updating for, I think, like many years at this point. with things that you're thinking about which we'll link to in the show notes. And I love, I think it's, I think it's really cool and really inspiring, just sort of the rawness of that. and one of the things that's in that Google Doc right now about what you're thinking about is the i, this idea of intentional tech, which dovetails really nicely with something that I've been thinking about, which is this idea that. There's this new AI that has this new era of technology where technology can understand our stated preferences. Which is different from the social media era, which worked totally just on revealing preferences, what you dwelled on and Yeah, what, what you clicked on, whatever. And so social media tends to make us tend to service things that are pro like more outage driven or more like sexier or whatever. Anything that catches your eye. Like the car crash effect basically. and AI, because it can talk to us, gets a much richer understanding of who we are. And maybe like I think ChatGPT is much more helpful and much more enlightening for example, than like the Facebook algorithm is just by default. So I'm, I think that dovetails a lot with what you've been thinking about with intentional tech. So tell us, tell us what's on your mind about it.

Alex Komoroske

Yeah. Intentional tech I think is really critical in this new era, we're at a crossroads as a technology industry. I think we have this new technology, LLMs, that I think are as transformative as the printing press, electricity on the internet. So it's this big general purpose unlock for all kinds of stuff that wasn't possible before. And we have a choice. We can go down the path that we've been going down, which is engagement, maximizing hyper aggregation going after what you want, not what you want to want as a user or being aligned with our intentions. And that could lead to a new era of human flourishing. I think. This is not the default path, by the way. We'll have to choose it. We'll have to work to build that. And that's why I think intentional tech is so important because we want technology that aligns with our intentions. Not necessarily what I want in my revealed preference of looking at car crashes or whatever. But what I intend to do, I intend to spend quality time with my family. I intend to experience new things. I intend to read interesting takes that disagree, that it can be disconfirming evidence that challenge my worldviews. Those are some of the things I intend. That's what I find meaningful to do and it's very easy to fall out of that. But LLMs, I think with the right product fabric could enable that. And another important part of this it aligns with your intentions as if it is working for me is an extension of me. I heard a word last week. I love Exocortex. It's like an extent. You're a cognitive exoskeleton, but that has to be fully aligned with you. Imagine that context, all of that rich substrate of meaning that you've distilled and written down, that's all about the things you care about and all of these facts about you. If it's not working for you, that's terrifying. I saw someone a few weeks ago at one of the big tech companies saying, we're making our tool. personal, proactive, powerful. It's like, well, let me stop you right there, because the very first word personal doesn't actually align because you are a massive corporation that's trying to sell me ads. And if you're maintaining a dossier on me and then that dossier is leading to a powerful and proactive thing, that's terrifying. Whereas if it's working just for me as an extension of my direct agency, that's really empowering and I think that's one of the reasons intentional tech is so important at this era.

Dan Shipper

That's really interesting. and tell me more about like, because you know, when I think about this, one of the, one of the words that comes up, you talking about alignment is that. AI started, or this generation of AI started with alignment or alignment with human preferences at, at its core. Because we were all afraid that AI was gonna kill us. and now it's like, well, it might not kill us, but maybe it'll just serve us ads that will make us dumber, spend our money in ways that we shouldn't or whatever. so I, so I'm kind of curious what you think has to happen to make us. Go on this sort of more intentional path.

Alex Komoroske

For, to me it's alignment. I think it's funny, like we, yeah, we used it for the underlying LLM model. Like I think there's two layers. There's the model itself, which is, if you imagine this as a stateless thing that you send a query to and it gives you back a response. It doesn't store any state. then there you have alignment problems of what kind of biases are baked into it and what have you. But then there's a layer on top, which is your context, and that is the thing that is a malleable bit of information about you that changes. It has lots of rich meaning. These are two separate layers, I think. And so I am more interested in this layer at the top and assuming if you have multiple LLMs that you can choose and swap between That don't remember anything about you, then it's fine. You can swap between different ones and that matters less. To me, where a context is the place that's all aggregated is I think more important.

Dan Shipper

Well, that's an interesting architectural question, like Right now language models are stateless. Do you think in five years, for example, there's still gonna be these sort of stateless intelligence boxes? Or do you think they're gonna be auto updating their weights, for example, as you talk to them?

Alex Komoroske

I don't know about auto updating their weights. and this might be an architectural breakthrough, but one of the things that's weird about these models is that they do take a long time to bake. So like they have this weird pace layer down at the bottom that's like months behind Yeah. On training data, it just takes time for it to bake. And then you have these layers on top that the system prompts now, like in chat DPT will inject little bits of your context, a weird bit that by the way, you cannot inspect, which is kind of funky, right? Yeah. Like if you ask it, Hey, what do you, what context about me did you put in there? It's like, I can't tell you. That's creepy. Right? It's like, I can't do that, Dave. You know? and Simon Willison, we were chatting last week and you know, he's got some prompt that will extract out this dossier about you. It includes things about stuff you said in the past. It could include things like your insecurities about your weight. You know, these things about you. And then it also says in the last. In the last five weeks, 9 percent of your interactions with the chatbot have been bad. As in you've been trying to manipulate it or whatever, I don't know. It's weird seeing this view for yourself. So I think ChatGPT and others are very clearly trying to move, merge these together. So that you have all your context in ChatGPT and it's hard to leave. That I think gets more problematic, especially if you are going down an engagement, maximizing kind of playbook. Which I mean a bunch of the executives from Facebook now run three of the four big chatbots.

(00:10:00)

I think to go back a step to meaningful computing has four major components to me. One is it's human centered, not corporation centered. It's centered around me and my intentions. Two is, it is private. By design. That means that the data is only for me to see, it's for me to bring where I want to bring. And to choose who gets to see it. using tech techniques like confidential computing to make sure it is entirely private. it has to be pro-social. It has to be something that helps you live a life integrated with society, not just being this hyper individualized kind of little island yourself, but integrating with society in a meaningful way. It also is open-ended. It has to be something that allows you to explore and to build and to create new experiences within it. The way that architect our software works today. is only the entity that created the software is allowed to decide what kinds of things you can do in the software. And that's due to the security model of how, of how our, all of our software has worked for the last 30 years. but that kind of close endedness means that you had to convince some pm somewhere to have prioritized our feature. Which if it's particularly niche probably hasn't been, probably hasn't happened.

Dan Shipper

Okay. Well, there's, there's a lot of things you just said about I would love, I would love to dig into it, dig into all those points. But the big one that stands out to me is right now we live in a world where in order to get this great new technology that's advancing at this really rapid pace we have to get it from well-funded startups that have a profit motive. Profit motive. and I know OpenAI is a nonprofit, but like whatever. I think that's what they say too. Whatever. Clearly they're being run like a startup. and I guess for me, I don't think that's a bad thing, but it sounds like at least one way to read what you're saying is we have to get rid of the sort of corporate startup structure for these technologies to not necessarily be prosocial. Like how do, how do you envision this being made us being able to make these decisions in, in terms of what kinds of corporate structures or startups or nonprofits or whatever are controlling and building these things to get the outcome that you're talking about?

Alex Komoroske

I think it is compatible with being a business. Okay. So for example, we're chartered as a public benefit corporation. To me , what's most important is that users are paying for them, it's not free. if you aren't paying for your computer and it's not working for you, it's working for somebody else. That just so happens to think that giving it to you is a benefit. Now, that doesn't mean that's a necessary, but not sufficient characteristic because I think chat bts the playbook that they're executing is one that is about engagement, maximizing and making you stickier to the service and what have you. but you paying for your own computer is a necessary component of it to make sure that it's actually truly just working for you. That's where also that privacy really matters if your data is visible only to you with your keys. this does not necessarily mean by the way people have historically in the past said, oh, that means it has to be local first. I love the local first movement. I'm very aligned with its ideals and ethos. Local first is really hard for a couple of reasons. One, architecturally it's very challenging to have a bunch of peers that have to have a vegetable consistency and you don't know when they're gonna sink back up. It's very hard to build consumer product experiences that work the way that people expect. And two is, it's inconvenient if you've gotta run your local server and your laptop isn't plugged into this, into the network when you're on your phone or whatever, the thing doesn't work. So actually there are other architectures that can use things like confidential computers to run in the cloud in a way that is remote, can be remotely attested to be totally private to you, and that nobody else can see into that data. And so those are some of the architectures that allow a full alignment with the human's intention. Okay.

Dan Shipper

So basically you are saying Chat is doing some of this, so they're, they're, they're starting with a subscription model but. There are things that they're doing well, like what are they doing that you think is aimed at maximizing engagement? I definitely understand this kind of memory thing. I want to keep using GPT because it remembers who I am. I love that part of it. What are the things that they're doing and the decisions that you're, that they're making that tell you that they're kind of going down this engagement, maximizing the path that will lead to a bad place?

Alex Komoroske

I think the engagement max path is just the default path. So I don't think it's, by the way there's anything particularly, they're like, oh, here's audacity play. Yeah, this is how you do tech now, right? You're like in the last. 10 years ago, we just realized that aggregators are a really powerful business. And they want to consume all of the demand, so then all the supply comes to them. And that's a great business. I mean, if I were sitting at their seat, I can totally see why they would do it. Like that's, again, it's not a nefarious plan, it's just the default thing that you would do. I think if AI is this incredibly important technology that also knows all the intimate details of your life That becomes especially important for it to be in an architecture that is not Yeah. Something that you're something that this other company can look at and maintain a dossier about you. because it's just too easy to manipulate. LLMs can translate anything from anything. And that means they can also translate just for you to figure out exactly how to land a particular message for you, which means it's imperative that it's aligned with your intentions. And it's not gonna try to get you to do the Mondo subscription or to go to one of the partners or what have you.

Dan Shipper

Well, let's say you're Sam Alman right now, and you're in charge, you're in charge of OpenAI. what would be, what, what are the decisions that you would make tomorrow that would put you on a path that you think is actually right?

Alex Komoroske

So if I were to say Altman, I would do what, what they're doing. Honestly, and as in it's, I can see why it's a great business. I think we will add a lot of value. That's why it's used as a complementary approach To some degree. a making a system that is this coactive system that's just working for you, that is not just chat. Can be a complimentary thing. to these other systems. I like, I wouldn't say, oh, you should change what you're doing, Sam Waltman, because I think actually what they're doing is a reasonable thing. This model that they're, these models that they're creating, especially because they're available via the API with no memory stored to them, that's an important characteristic. Is great. And it makes a really powerful underlying engine that can be used to power lots of other products. especially to the extent that there's multiple that we can swap, swap between and don't get stuck. Got it.

Dan Shipper

So you think that there's a call for a complimentary technology provider? Set of technology providers that allow you to take all of the context that you have, keep it private, and then bring it to any situation or service that you want.

Alex Komoroske

Yeah, and I would say that my mental model is, oh ChatGPT is kind of the AOL of this era. So if you use the parallel of the internet and the web AOL was an extremely important company. They're the ones that brought everybody online. America, the whole of America online has amazing experiences. It was somewhat of a walled garden, but it also gave you access to the open-endedness of the open web. And I think that's the kind of role that I see them playing. I hope that what we get out of this is not some closed ecosystem of hyper aggregation beyond even what we've seen to date. But instead we see this open ecosystem of things that are very user aligned and—

Dan Shipper

Why AOL vs. like Microsoft for example. So Microsoft I. I think in a similar way, not for the internet era, but for the PC era, like teaching people how to use computers and bringing computers as a thing to most of, most of America and the world. and built a platform that everyone else is sort of building on top of. So that's another kind of analogy. Why do you think AOL is more appropriate?

Alex Komoroske

I think the analogy that tracks to me best for LLMs is the internet. Like LLMs are the internet. They're a new kind of thing that you can do. And the web, like an OL. At the beginning when I first started using it, like second grade or whatever it was about chat rooms. And you could do the whatever, the keywords or whatever. ASL like, I mean that was what I thought the internet was. And then it turns out, as I learned later, I was like, whoa, this whole crazy open ecosystem that no one controls with all kinds of weird shit happening. And that was that open system of the web. And so in the same way I see them as distributing this new technology to people showing here Look, chatbots. You can do these cool things with chatbots. And then later you realize, oh, wait a second. The same thing as Animate Chat. Botts could be used to animate all kinds of new experiences I didn't realize were possible before. And that's one of the reasons I think of them as OL vs. Microsoft.

Dan Shipper

Why did AOL die?

Alex Komoroske

Like, I don't remember. I think at a certain point the open-endedness of the web kind of took over and it just became the point of like, like I think a lot about systems that you pointed out. I think open-ended systems tend to win under certain conditions. Especially in the growth era. Like the very beginning, there's some vertically integrated thing that kicks off a new revolution. People go, oh shit packages up really nicely and people get it. And as time goes on and the, the sort of commentator possibility gets too big for any one entity to successfully execute. And so the open system, if there's any way for it to escape out into the side like it was with the web sorry, I talk with my hands as you can see. If there's any way for it to sort of escape out the side like there is with the web or using APIs to access LLMs then the open system kind of takes off this combinatorial swarm. Hmm. And it overwhelms the closed system. That's also potentially just a ho a hopeful thing. I hope we typically see oscillation between open systems and closed systems. We've been in closed systems dominant for like the last 10 years and I think it's a bummer. Like I think we're kind of in this, we're in its amazing age of technology and it's also kind of the dark ages of tech.

(00:20:00)

Alex Komoroske

Because we're all in this hyper aggregated thing where the only features that can exist are the ones that aggregators for consumers, the ones that aggregators have decided to prioritize. And that's a bummer because as a company gets larger and larger and larger the Ian floor, what Clay would call the Ian floor, the smallest feature that it makes sense for them to even consider prioritizing goes up and up and up and up and up. Mm. So if you're at Google, if I'm trying to pitch a product to other teams, and they say how many net new active, net new one day active do you think this product will get? I'll say, oof, I don't know, 50,000. Then we go to 50,000. I wouldn't do this for 50 million. You know, so like there's this whole class of features that just can't exist. in a world, the world that we're currently in that's hyper aggregated.

Dan Shipper

It seems like one big difference between the web and the early stages of the internet, which to be honest, I don't know the history of. Well, off the top of my head and language models the web was like, you have. I mean, it was sort of government run at first. You have this international consortium that kind of just defines the standards for how the web works and all that kind of stuff. That doesn't seem to be the case here.

Alex Komoroske

So I don't think, actually, I think, I think we are waiting for the web of AI to show up.

Dan Shipper

Interesting.

Alex Komoroske

And that's partially what we're trying to help catalyze. Hmm. and that's, I think that's what I mean by like, we have the internet and we're like, cool. The Internet's definitely gonna be useful. Like if you look back at Al Gore's proposals for the information Superhighways. it was actually very prescient in a lot of ways. it's just, it was all about. Pipelining existing content and business models into people's homes. Hmm. And some of that definitely happens. You get things like Netflix or whatever, but a lot of YouTube social networks, none of that was envisioned by the, or the Wikipedia, none of that was envisioned back then because we just didn't know how to imagine it. It wasn't even the thing that we thought was possible. And then the web creates this open system of lots of different people trying out different things and seeing what things work. Found some of these interesting new pockets of value that grew into whole continents of value.

Dan Shipper

So how do you, how do you make that though? do you have to, is it, do you have to get governments involved? Do you have to use a blockchain where everyone's like, sort of running around? Oh, no. I think that's what it sounds like to me.

Alex Komoroske

I think you can build, we have so many powerful substrates. The web exists as a distribution platform. Everyone's device speaks to the web. and so you can use it as a way to distribute a whole new kind of experience that fits within this. So I don't think, back on the internet, you needed a consortium. You need tons of capital expenditure to build the pipes. And that was government grants. It was also businesses that overbuilt all this capacity. Those businesses, by the way, were great for society. We got all this excess bandwidth, the capacity kind of crappy businesses though, right? Because you build a pipe somewhere. If someone built another one right next to you, they would lose all pricing power. You know, it's a total commodity. but it's a commodity that powers the rest of this innovation on top. I kind of see large language models, the producers of them as people laying the pipes. Mm. They're the ones making, doing this extremely capital intensive creation. But there's actually not particularly that much of an edge across them, which is great for society because that means you can take LLMs as a commodity. You can assume that they exist in reasonably high quality ones, including quite good open source ones. And that means that all kinds of interesting business dynamics take off and you can kind of take it for granted and build what's up here now that I can take high quality LLMs for granted.

Dan Shipper

Yeah. It does strike me that the thing you're talking about, which is the bar for like a Google PM to build a feature is like, we need 50 million users. And so you sort of. You sort of look past the 50,000 user use case, which A, those are just valuable in themselves. And b, often they're the ones that end up being the 50 million user use case in five years. Some subset of them will become those. It's like the sort of innovator's dilemma type stuff. It sort of strikes me that in a world where language models are commodities, capitalism just does that automatically. because small startups will just do the 50,000 user use case and just, and just build it. So is this just gonna happen without us?

Alex Komoroske

So here, here's what it might be, and like I agree that this kind of I'm so glad there's so many different ways that the society could have gone OpenAI could have had ChatGPT before they released the API. That would've been such a different world.

Dan Shipper

It would’ve been very different.

Alex Komoroske

Now everyone assumes they have to compete with an open access API and now it would be very hard to close that door. Thank God. So we can take it for granted for the rest of us. So like, I just. I like waking up with a, like last night with a nightmare that had gone the other way. So I'm very glad that happened. Or like IBM developed AI. Oh my god.

Dan Shipper

What, what would, how terrible would be like $30 million a year for like one query. Yeah, exactly.

Alex Komoroske

So that's great. Here's the thing though, that means that the current laws of physics, the current security model that we use for the web and apps actually limits this possibility. And this security model is called the same origin paradigm. The only people who know about it really even web developers don't really know about it.

Dan Shipper

I have no idea what you're talking about.

Alex Komoroske

What is that same origin paradigm is the laws of physics, of how the web works. And what it says is it's the security and privacy model that we've used for 30 years. It kind of actually grew up as an accident actually. And at the very beginning, the web has no state, and so when you reach out to a server, it gives you back the same thing that anybody would've gotten if they gave that exact same request. Well, then you add cookies and then you need to say, when do I send, where do I send cookies to? Once they've been set. Which other URLs do they go to? And then the easiest thing is, okay, there's a notion of a site, which is roughly a domain I see. and so it'll send things back and isolate them by that. So cookies will come back between these but not go over there. And then you add JavaScript and you allow things like local storage in the local state. Where is that local, who can see that local state? The same people that can see this. Okay. And so now it grows up as this or as this origin boundary. So this somewhat happenstance thing turns out to be at the core of our entire security model, what it assumes is each and every origin is its own. Isolated little island. It can't see data from any of the other origins, but can see all the data that the user put into this origin intentionally or unintentionally.

Dan Shipper

And origin is roughly like google.com is? Yeah.

Alex Komoroske

It is roughly, roughly a domain. It's not exactly, it's slightly different, but it's roughly, you can think of it as a domain. And so what this means is data accumulates inside that origin as a little island. This is a very reasonable and good security model. It's one of the things that makes clicking a link or installing a new app safe because it knows nothing about you. So then if you choose to do it and to put more data into it, that's okay, but that's your prerogative. So it's great for trying out a new thing. The problem is putting new data into it. It can do whatever the hell it wants. With that data, it can send it to evil.com, commerce scam scammy.com, and you have no idea. You are implicitly trusting the creators of that. Of that application. And somewhat surprisingly, this leads to massive centralization. So if you have a use case, this model is about isolation, not integration, but our lives are integrated. And so if you want to move things across different origins, you have to be the orchestrator. You as the human, have to keep track of the information. You want to copy, paste, or move between these different things. It's expensive and it's somewhat scary to put it into a new thing. And so imagine that you have two use cases. One is some cool startup that says they're gonna do this amazing new thing with your calendar and scheduling. And one is Google Calendar that says they have a similar kind of feature. Which one do you pick? Well, Google already has all your data. And this startup, you don't know if they're securing the data properly or what the business model is.

Dan Shipper

Plus you have to give it the data, like step by step. Like it doesn't it. It doesn't know where to start. It's a cool start problem. Yeah, yeah, yeah.

Alex Komoroske

And so this leads to asking, or I'll just do the one that already has it. So this is a phenomenon you might call data gravity. It tends to accumulate in the places that already exist and they become massive. And nobody else can get this data. And this one of the drivers of ag of aggregation. And again, once you aggregate to such a scale the, your coaching floor goes up and the set of features that they could even consider doing as is only a small subset of what you could do with all of this data in practice. It all arises somewhat surprisingly, from the security and privacy model that we've used.

Dan Shipper

I love that. Okay. And so, and I think what you're saying is because we inherited that from the first generation of the web or second generation of the web or whatever, chatbots currently operate that way too.

Alex Komoroske

We don't even realize it. The apps all do this. technically legacy applications on desktop do not have this model. Because they can interact via the file system. Apple has been recording and jamming it into Mac OS X for a number of years. Windows probably has too. I haven't been paying attention. So like people when we're building software and operating systems, we don't know another way to think. It's like, how else would you possibly do it? Yeah. Which is insane. So like, there's this, I call this the iron triangle, the same origin paradigm. There's three three things. You can only have two. One untrusted code, two sensitive data, three. network access, you can only have two of the three. Why? To have a safe system. I don't understand. If you have untrusted code with sensitive data and network access, it comes in, looks at your thing, figures out your financial login and sends it to evil.

Dan Shipper

So untrusted code is like some developer wrote some app that I'm downloading that hasn't been, I don't look at or have it. In what, in what sense is it untrusted?

Alex Komoroske

Untrusted as an I haven't made a trust decision about it. So when you put an app in the App Store, Apple looks at it and says, okay, based on the construction of Sandbox, and also based on our review of this fake, this fine and we will allow people to install it.

Dan Shipper

Okay. So untrusted code is the first thing. What's the second thing?

Alex Komoroske

The second thing is sensitive data.

Dan Shipper

Sensitive data. Okay.

Alex Komoroske

You don't want to potentially identify data potentially. Precious data. And the third is network access. So the web says you can get untrusted code webpages. and you can get network access, but no sensitive data. You get only the thing Exactly. This app model says you get sensitive data, network access, but not untrusted code. It all has to go through the central location. By the way, it charges a 30 percent tax.

Dan Shipper

What about like windows? because Windows doesn't have an App Store. So what's the, so historically the model, there wasn't one.

Alex Komoroske

Yeah. And that's why installing software in those is a little bit more dangerous. Because without a model like this, it could do whatever it wants and it has access to all that data and you have to be more careful.

Dan Shipper

But it still exists, it's still like a vibrant ecosystem. So it's possible to do without this triangle.

Alex Komoroske

It’s possible. And you could also change it, you could tweak it in other ways. And you could do clever things using tools like information flow control and confidential computing to create a whole new sort of laws of physics, I think. And if you did it as a whole, then you could do things that hit all three. Okay. If you had all three, you could do wild, interesting things that would not be possible today.

(00:30:00)

Dan Shipper

So is the file system your number one example of a good alternative to a same origin paradigm?

Alex Komoroske

Yeah, so the file system allows. Fundamentally, multiple apps are all allowed to work on the same data. And they can coordinate via the file system. And that allows you to not get stuck. It's, when you think about it's kind of insane that all of your data is locked up inside of an app. Like it can't leave the app. That's wild actually, when you think about it.

Dan Shipper

Well, it's interesting because the file system has a number of properties, unless you don't sell Dropbox or whatever, but like it's here, my file system is here as opposed to, it's everywhere. And I kind of have physical and just total control over what happens to it. Is that an important property of, of the system? Yeah, it is.

Alex Komoroske

You have physical and control of it. That's why Local First talks about, oh, we're gonna make local first, where you have control and you can make, bring the data across different things. Part of the challenge, again, with Local First is it's inconvenient. We expect things to work across multiple devices to work even when one device is off. We expect these things to be in sync across different things. And the local first architecture is quite difficult. This is one of the things that we are looking at. We call open to tested runtimes. The pattern is used on a confidential computer. So confidential computers are secure enclaves in the cloud. It allows VMs to be run fully encrypted in memory. and which means that even someone with physical access to the machine, like a Google SRE, can't peek inside, which is great. Then what you do is you have an open source runtime that you can run that executes the code and then you can ask the conant to compute the cluster to do a remote attestation and to give an attestation assigned at attestation by the underlying. hardware manufacturer that says this thing that you're talking to that just handed you, this running in confidential compute mode and here is the sha of the gi sha of the vm that it booted with. And so it allows you remotely as a savvy user to verify, oh, it's running an unmodified version of that software. And that solves a big coordination problem because now lots of other people can all verify that it's running an unmodified version of the software. And yes, fine, we'll just use that central server as the place to coordinate. Because we can all see it can't do anything different than what it says it will do. and that's a really powerful coordination.

Dan Shipper

Do users care about that? So, because one of the things that makes me think of it is the, a lot of these concerns feel similar to the flavor of concerns that original blockchain crypto type people had and were solving for with Bitcoin or. You know, any, any kind, any kind of cryptocurrency. And then Coinbase came along and they were just, we're just gonna, let's screw it. Put a general, sort of same origin paradigm type solution on top of that. And people actually love to use that.

Alex Komoroske

So it is this, I think if you go after users who care about this as a primary end, you end up with a very small audience of people and you end up with a mastodon. For example, as in something that's like pure Yeah. But also kind of finicky to use and doesn't make that much sense. And that's why I think you want a blue sky kind of approach. Blue sky you can use as an end consumer. And be like, the only thing I know about it is that it's not owned by, you know that guy Yeah. You know, and that's it. but then the closer you look, the more you realize, oh, this is actually very clever. This is an interesting way of my key pairs and the way that personal data stores work. Most users will never have to know about that. but the more you learn, the more you're like, oh, okay. And so that I think is a, what I would call like an inductively, knowable system. people go, I was talking to somebody like nobody knows, nobody ever cares about the security model. They do actually care about the security model. They just don't know the words to express it. Nobody understands what the same origin paradigm is. Nobody is a very small portion of people. And yet it's the laws of physics that make all the other stuff we do safe. The reason you don't have to care is because some people also care. And the general characteristic for me is when you're using a new system, you're like, Ooh, this seems creepy or too powerful. And you go talk to your more tech savvy friend and you say, Hey, Sarah. Do you trust this kind of thing? And she says, yes, I do. And the reason I trust it is because, yeah. And she knows, she's read a blog post in Hacker News that was someone who wrote, read a thing who, someone who audited the code. And that inductive chain can go all the way back down to the fundamentals.

Dan Shipper

That makes total sense. I totally agree with all of that. I think the thing on my mind is the first people that are gonna adopt this probably are the people that do care about the security of it. and are those people the right people to kind of seed the community that ends up blossoming into this thing? because those, that kind of person is going to say a lot about, like the total trajectory of this kind of product. Right.

Alex Komoroske

So I actually, I don't necessarily agree that it will be those people, the privacy heads that are adopting it earlier. Interesting. Okay. I think of it more as like the high volition users. just the tinkerers, the people who are the early adopters. Some of them will have a higher proportion of caring about this kind of thing. But a lot of them won't. Like there's a ton of startups that say step you install the apps, step one, sync your Gmail inbox and Google. Okay. I run a startup like that. Exactly. Because it's, most people don't really care, especially early adopters. I think what we're describing is a system that if it, if you can break this iron triangle and then you get these, all these crazy emergent phenomena that aren't possible in other software, people use it because of that. And the reason it's not creepy is because of an underlying security or privacy model. Hmm.

Dan Shipper

Do you have a name for the alternative? The same origin paradigm?

Alex Komoroske

I call it contextual flow control.

Dan Shipper

Contextual flow control. And tell me what that means.

Alex Komoroske

I don't want to go into too much depth at this point. It's still very fuzzy. Helen Nissenbaum is a professor I believe at Cornell, a legal studies professor, and she talks about contextual integrity. Contextual integrity is the gold standard of what people mean when they say privacy. So when people say, when lawyers talk about privacy, they think about consents. Did the user sign a consent that said we can do this? That's it. As long as they all signed the eula, it's fine. technologists talk about end end encryption as long as it's end, end encrypted, it is private. but what people mean intuitively is this contextual integrity, that the data is used in the context that you understand aligned with your interests. It's not surprising how it is used and it's not going against what you want to happen. And that's like a sort of first principles, ground truth way of thinking about it. and then you combine that with things, other technologies and allows you to make. Formal claims about information flows in alignment with people, principals.

Dan Shipper

That’s interesting. The reason I love the same origin paradigm thing is like you're going all the way back to this one decision that has all of these really interesting positive and negative effects like later that are sort of unpredictable, which feels very it's like a little bit Steven Johnson or a little bit , like one of those writers that's just, here's this one thing about the way the history works that, just totally changed everything. Do you have any other things like that you've noticed about technology? I don't want to put you on the spot, but I just feel like you probably have some, some sort of counterintuitive things in, in your time working at Stripe and Google and whatever. Thinking about mobile or SaaS or whatever.

Alex Komoroske

I think a couple others like, yeah, the world changed that day. We didn't realize. One is the chat. But coming out after the api. I was already available. Yeah, that's a big one. Another one is. If I could go back in time, like I had a time machine, the first thing I'd do is go back and murder Hitler or whatever, you know. The second thing I would do is I would go to stage two th in 2007. the stage where Steve Jobs shows off the iPhone, and I've run onto the stage with a poster that says this will become the most important computing device on Earth. It is insane to allow a single company to decide what things you may run in it, right? That's insane. That's absolutely insane. And the only reason it's actually viable is because they decided very shrewdly at the very beginning to not allow other rendering engines in browsers on that device from the very beginning or, and to make sure that all software had to go through the App Store. And that was a reasonable decision. And like the first order implications of that are very reasonable. It means that you get, as a user, the safe experience of nothing's gonna hurt you. And the second order implications of that are wild. It's insane to me that the most important computing device in our lives, a company who has demonstrated again and again that they're willing to use capricious kinds of decision making about what things will distribute, is the one, that gatekeeper. That's insane to me.

Dan Shipper

It seems kind of a lot of what you are reacting to and the Apple example and the same origin paradigm example is. aggregation of power and a few large big tech companies. And the kind of second and third order negative effects of that on user experience and innovation and stuff like that. And competition. And, yeah, competition. So it seems like that's a reasonable place also for regulation. What is your stance on regulation?

Alex Komoroske

Regulation also has all kinds of second to third order. And so have you ever met Danella Meadows thinking in systems? So like the systems you gotta dance with to some degree. you can't fully control them. Totally. You control that. and that's why I find some of these characteristics, when you find the right, like technical leverage points You can say, Ooh, actually go. That has a very different way that it evolves, I think is preferable to a regular regulatory.

Dan Shipper

What if I said like, you could pass, you get your president for the day and you can and all of congress and this supreme part for it. You're a dictator for a day. and you can pass one law. do you have, like what, what, what law would you pass? It? It's like a, you can, it's a one sentence law.

Alex Komoroske

I don't know. I would have to, if I don't, I've never thought through, like I'm so used to in open systems, which were all part of a large, complex adaptive system Called society. And there's always weird edit currents of paradynamics and what have you. So like I take from the assumption that, I never have a lever that is like this massive lever I can just simply pull. And so I don't even know.

Dan Shipper

More like looking, you want to find the grain of sand that you put on the pile and it just all like cascades, but it cascades if the pile is ready for it. Right, exactly. And that's one of the nice things about systems thinking is like, if you do the right thing, like judo moves the right thing and the system is ready for it.

Alex Komoroske

This the world moves and if it's not, it doesn't move. And so that the system is deciding kind of through all these like micro interactions throughout it.

Dan Shipper

Tell me more about that mindset and how you developed it and what are the, what are some of the key moments in, in your life where have you seen that happen?

(00:40:00)

Alex Komoroske

Um. I, so I wrote my undergraduate thesis on the emergent power dynamics of Wikipedia's user community and I, my degrees in social studies with a minor in computer science. And I was just fascinated by this emergent phenomenon of how all strangers work on this thing with no coordination, and yet you get this convergent, extremely important result outta it. That's insane. That's wild. And then when I went right outta Google early college to Google, it actually was harder to get my, my first PM role because I didn't technically have a CS degree, even though I basically did, it was one credit shy of a dual major. And so I was like, man, what a mistake that was, I just didn't just throw it in the back of my mind, like whatever. Like if I could go back in time, I just would've majored in cs. And then I did my first year on search, the precursor to the knowledge graph. My second year in DoubleClick, I was in the eight PMM program and part of that is you mentor a lot of the people who come up after you. And I learned very early on, I loved mentoring people. It helped me think through what I was dealing with and help people and get more patterns to go to my pattern matching library. And then I became the lead PM for Chrome's web platform team. And I think a lot of PMs are under the misunderstanding that they're in way more control of their users and their usage than they actually are. If you're at Google and you ship a feature and tomorrow 50 million people use it, you kind of get a little bit of a God complex, right? Yeah. If you're a platform PM you tweak a thing which causes other developers to do something different, which causes users to be affected in a second order of effect.

You're aware of this indirection. If you're the lead PM for the web platform, which is an open system with multiple browser vendors who don't like each other very much and are constantly kicking each other under the table, you are under no illusion that you're in control. And my engineering counterpart, Dimitri Glazkov who was the Uber TL for Blink, brilliant guy, and he's the one who got me into, introduced me to the word complexity for example, and introduced me to the Santa Fe Institute. And so as I was going, I was realizing, oh, these things I'm naturally doing to try to make these good outcomes happen in this open ecosystem like progressive web apps and these web standards and stuff. I'm intuitively applying some of these paradynamics and complex systems. And um. Then I left Chrome. I went to work in augmented reality. I created a little nerd club behind the scenes with a bunch of people that I just kind of selected that people who when I said it was a nerd club, didn't go, ew. That sounds fun. Great. Come on in. And then it was a very collaborative debate. very trickling in different perspectives that I get a diversity of pers perspectives into the system. No particular goals, it's exploring. After a while we came up with a distilled, like a thing. I was like, wait a second, this must be the strategy. Like nothing else could work. This is a strategy that kind of makes sense of all the pieces. And that caught on a significant momentum. And that's when I realized, oh my gosh, I'm not a web developer ecosystem guy. I may be a systems guy. And the same techniques I've been advising I used and deployed in those contexts are the same techniques I used. I've been advising hundreds of PMs over the last decade Yeah. To use, to navigate Google. And the reason they work, the lenses I'm using about game theory and power dynamics and evolutionary biology are things I learned in college. And it was just kind of an aha moment. And that's when I wrote the earlier version of the Simul deck.

Dan Shipper

Tell us, tell us about that.

Alex Komoroske

That simul deck was a kind of lightning in a bottle. It was, I think it's a 150 slide emoji flip book. And it's just about a fundamental characteristic of system organizations that as organizations get larger, they get much, much, much slower. And that's true, even if you assume everybody's actively good at what they do, actively hardworking and actively collaborative, it arises due to an exponential coordination cost blow up. It's fundamental, it's inescapable and it's the force that we all deal with constantly and we don't even realize we're dealing with it. And so we get frustrated with ourselves, we get frustrated that Jeff over there, who if only he would do X, this whole thing would be easier. And we're just all fighting with this massive force, this force of gravity that's completely invisible to us. and so that deck caught on, I did another version. Similar kind of thing externally, and that also got a surprising amount of momentum. I've had people describe it to me as like, life changing. I'm like, I'm just talking about some emoji emojis talking about game theory. but it was, I think it really tapped into something that people experience, they're frustrated by and they don't like, just, it's like a big hug that says, you are not crazy. This is really hard and it's fundamentally hard.

Dan Shipper

So I think every, every nerd that has like is a systems person like me. Like you, like we all go through complex systems, like holy shit thing. for people who are not, have not done that yet, or haven't done it in a while, like what are the three or four gimmes like a quick rundown of the key ideas that get you about complex systems that you think are applicable?

Alex Komoroske

I think we tend to look at individual decisions as the primary lens. I see it as a secondary lens. I look at the emergent. Characteristics of the system, what would each person do inductively, and how does that, how do those decisions inter or how are they interdependent? so I try to look at like, what's the simplest inductive case and then how would that play out if everybody were doing a similar kind of thing?

Dan Shipper

Gimme a concrete example of that.

Alex Komoroske

So like, there's a dynamic that would call a gravity well dynamic that shows up in a lot of compounding loops and it generally shows up when there's a thing I would call a boundary gradient. So people who are using the system. On one side, people who are not using it, even at the beginning, even if lots of people are, have incentives away from using the system, oh, I don't want to use that thing. That seems dumb.

The people on the boundary who are right at the edge of possibly using it, do they want to be in or do they want to be out? And a great example of this Facebook back in the early days—it starts off with Harvard and then now extends to the Ivy League. Do they want to be in Harvard? Do they want to be in with all the other schools? I'd rather be in Harvard. Mm. And now extends to the other schools. and at each point, each group of people would rather be in. And if there's a compounding, if there's some kind of network effect inside, then this strength just gets more and more powerful as you go. And it can become this thing that can pull in, even as time goes on, the incentives pull in for everybody and everybody will, will pull into this overall thing. With sufficient time.

Dan Shipper

So if Facebook was started at the City College of New York, are you saying like it probably wouldn't have it?

Alex Komoroske

It potentially, I mean it, was one of the dynamics that made it this juggernaut, I think.

Dan Shipper

Hmm. That's fascinating. Who are the, so Donella Meadows is someone I've read thinking and Systems is really good. What are the other like you know, Santa Fe type people that are, that have inspired you that you think of, that you've read or think about a lot.

Alex Komoroske

One of my favorites is The Origin of Wealth by Eric Beinhocker. Fascinating book. I haven't read it. It's really good. It talks about these organizations, why these organizational things emerge and how it views business as an evolutionary process of exploring a fitness landscape which I think is correct, like a very useful lens of seeing it. Cesar Hidalgo, who wrote Why Why Information Grows. he and Eric used to work on some stuff together, also talks about why fundamentally knowhow and its diffusion in societies. Knowhow is the way we actually do stuff is knowhow, not knowledge. Knowledge is rich. It's difficult to distinguish, to communicate. It's in our brains. It's what LLMs know. Exactly. And so they have absorbed this kind of squishy system, two style awareness. You can think of every time we're trying to communicate, we do this conscious process to distill this squishy, rich nuance into a little tiny packet of information. Little seed of a thought. And I shoot it through a little pea shooter into your brain. And I hope it lands in fertile soil that will grow into a thought flower. but extraordinarily lossy and expensive.

Dan Shipper

I agree. and I think we've also confused that peace shooter that the thing we can just distill into a peace shooter with the only thing that's important. When there's so much other stuff that's like, you can't really say, but I like the thing that's making me think of is something like a path I've been going down intellectually is this idea that language models are their first tool that makes those, makes that kind of tacit knowledge or know-how transferable between people. we used to have to, we used to require explanations, like Math or like logic or rational arguments to transfer know-how in this very reduced form into someone else's brain. But now we can just move, move tacit knowledge between people because you can train a model with a bunch of examples and you can be like, you don't have to, even if you do things.

Alex Komoroske

If you cheat, use an existing off the shelf model and you just pack it full of like, here's all my, if I read all the public writing you've done Yeah. And you had all the public writing I have done. Which I'm sure is just insane amounts for both of us. And we had a system that could find sift through the embeddings. Find the areas, the Goldilocks areas, the things, if we focus on the stuff we definitely agree on, it's kind of boring. Other than quickly building trust or like, oh, we think similarly. If we find the stuff that we fundamentally disagree on, it's also fundamentally boring because it's just noise to each other. But if we find the goldilock stone where it's just at the edge of the thing that we already thought. you can have these fascinating conversations and you can do it with embeddings, just very straightforwardly of compo. Like you train a model per se. You can have something that's really interesting.

You can find these areas of overlap and Yeah. The models are really good. My friend Anthea Roberts has this notion, she's a brilliant thinker. She's a professor in. Australia. She teaches at Harvard. and she talks, she's one of the most interesting users of LLMs I've ever come across. And what's her name? Anthea Roberts. I just, she just started doing a blog a couple a couple weeks ago at i, at my insistence. because I thought she had so many interesting ideas. And, she talks about liquid media. So like, think of a book as a fossilized piece of knowledge. You have to assume and give an audience at a certain time. And you break it, you put it in time. And then if it's not the right fit, oh, I don't understand these concepts that are prerequisites to understand this, or I'm not close enough in alignment to the belief already, stick to it.

And vs. a piece of liquid media is something you can chat with. It can kind of be a choose your own adventure. and that allows it to sort of unfold for that particular person. In a specific way as opposed to one way

(00:50:00)

Dan Shipper

It's funny because in a lot of ways we're getting back to Plato and Socrates' problem with writing is they were like, well, you can't talk to it. So, it's only sort of a shadow of what that person thinks and they can't argue back with you. and that's why those foundational books of the Western canon are written as dialogues. And yeah, I mean, for me, like I read so many old books like classics that are either written in English or were not written in English. And if you read it with ChatGPT, or Claude or whatever, you can get so much out of it. And I just think that there's a new format of reading that takes a classic book and helps you get into it in a way that would not, or ordinarily be possible, that I think could be so cool and valuable.

Alex Komoroske

And this we are, I know it feels like in this industry that we're like Halfway through the LLM era. We were in the very first inning. It's very early. We were like rubbing sticks together. We still think that chatbots are the main thing. and there's so many uses of this kind of stuff. Like if you have, I think of LLMs kind of as a mass intelligence, as a mass noun. So mass nouns are like water or rice. You know, you don't talk, you talk about the whole as opposed to the individuals. And I think that LLMs are kind of this mass now of intelligence. You can just pour this intelligence into all kinds of stuff and imbue it with a kind of life and adaptivity. And we're just starting, we're just starting to understand what you can do with these things. I am so hopeful that these technologies will unlock a significant human. one of the things that Anthea, for example, was telling me is, in academia one way you can handle LLMs, the fact that LLMs exist is say make sure you cite your LLM sources with this weird format of exactly the conversation and a link to it. Another way is to say, I assume you're using LLMs. Which means the quality of your output should be 10x higher than before. It should be more nuanced. It should, you know, understand disconfirming arguments and address them. Yeah, it should. And that's the kind of thing we can do now as society, as individuals that we can think through thoughts that we couldn't think before. And I think that LLMs are, that's one of the reasons I think it's the same scale of impact as the printing press and electricity on the internet. It's definitely a fundamental unlock.

Dan Shipper

Do you think that it changes this coordination problem you've been talking about? Because a lot of coordination is about trying to, trying to distill down your tacit knowledge and your tacit context into the peace shooter and then shooting it to someone else and that scaling exponentially, right? Yeah. I love peace. Everyone's just shooting their peace shooters. But you know, there's this whole theory about language models being better, middle managers, for example, than actual middle managers, which I think is the same kind of coordination problem. Do you think that language models might solve that coordination problem because they can transfer tacit knowledge much more quickly.

Alex Komoroske

I think they will definitely change the dynamic. I'm not sure how it will change. So I know a lot of companies are trying to do this because a lot of people like, and Eric Beinhocker’s book The Origin of Wealth. I think he uses the frame, I think he calls it physical technology or social technology. Like having Slack for example, is a social technology. It changes, I think maybe I get this backwards, but it changes the way that you can work. It changes the kinds of coordination that you can do within an organization. So LLMs must change the way the organizations work. I don't think we're gonna know what that looks like for like five to 10 years. Because it's not gonna change the existing companies. It's gonna be the new ones that just grew up in a different way. You also get these weird emergent metagames in, in organizations and so the LLMs in that coordination thing change the metagame, but they don't make it go away. So like If you had a system that could perfectly distill all your signals into an update that goes up to your manager and then you're curating it, you're tweaking it because, okay, this one says it's a state, a yellow state, but really it's green because Sarah has already got, she's on a path to resolution. By the time this rolls up to the CEO, it's already gonna be fixed. So it's actually better to say it's green.

Some of this is good. Some of this also leads to these compounding, green shifting of problems that gets turned into kayfabe throughout an organization, but the metagame shows up no matter what the kind of coordination technology is. So I think it just changes the game. And I imagine it'll make it better, it'll make it significantly better in some ways, and significantly worse in others.

Dan Shipper

Interesting. One, one other thing, one other thing that you're making me think of in what you said about Anthea Roberts and the changing expectations of, oh, use LLMs for this, it's gotta be better is just, I think one of the things that people get so hung up, hung up about with language models is they're like, well. What makes us uniquely human and are they just gonna take over everything that we can do? And I think my, my typical response to that is that assumes a very static view of what humans are and what humans can do. And that one of the most interesting things about language models is that they will change in a lot of ways what humans can do. and in the same way that books change who we are and what we can do. And writing changes who we're the internet did do whatever the internet does too. All that kind of stuff. and I think that's actually a really good thing. I'm curious if you had to think about how language models might change who we are, how we see ourselves. Where would you, where does that take you?

Alex Komoroske

Yeah, I get, I, that's why I think we were at this crossroads. I think we have the potential for like this, a dawn of a new era of human flourishing because of our ability to become, think better and bigger in ways that collaborate and bridge between different communities that didn't understand each other before. But now you can understand, help me empathize with this person's perspective. Well, here's something that you probably, you know so I imagine it could be this amazingly powerful force on us, it also could be like infinite TV amusing ourselves to death. You know, imagine David Foster Wallace, like Infinite Jest, right? It just, it could be a thing that, it can, knows exactly how to give you the precise dopamine drip, right? And so there's a world where humanity becomes extremely passive and it becomes not very agentic and not learning or growing. And that is the default one that is also aligned with engagement maximizing business models. And, but I think that's why I'm, I'm excited we had the potential to go in this direction.

Dan Shipper

Well, what's interesting is it makes you think of this whole debacle that happened recently with Che Bt getting too sycophantic. And the way that happened is they looked, they optimized too much for people's thumbs up and thumbs down responses. Like immediate thumbs up and thumbs down responses, what they want and what they want and want. Exactly. And it just became this kind of disagreement with you on everything. But what's interesting is that users revolted, which did not happen with Facebook. Like people weren't like, oh, it's just feeding me like car accidents. Like I don't, I don't want that. And I think that there might be something about that, because language models hit that part of your brain where you're like, I'm having a relationship with this person. You have a, you have a set of filters for I want their praise to be earned. I want them to feel like they can see me. And they'll tell me if I'm doing something wrong or whatever. And so they may demand things from companies that they did not demand in the social media era that might make these companies less likely to do the thing that you're talking about—

Alex Komoroske

Potentially. I mean, I think when it goes to the point where it's egregious Yeah. It's like, it's very obvious. One, I gotta say one of the reasons I like Quad is because every time when I like bouncing ideas off of it always says, what an astute observation. I'm like, I don't want to want that, but that's what I want. And so I do think that it's hard to not give us exactly what we want in that moment and like You know, we aspire to want something that's different than that. That's disconfirming evidence. But you also get these weird things where the context mixes and mashes against different things before each chat was like a blank, blank sheet of paper, and you could choose what context to bring into it. And now it kind of mushes context from different things. Like I asked it to say, what do you know about me that makes it snappy or whatever that first query was that they suggested when the new memory feature came out and it says, well, you're Alex Kaki, you're really into systems thinking emergent stuff. You have a startup called common tools. The starting salary at your startup is X. It's like, excuse me, what? Because six months before I had been bouncing off like, okay, if I'm gonna do a signing bonus like this, how should it change? That's so funny. And like it's like that, imagine if you are. If you're talking to your therapist, like people might use these things as a therapist, like a trusted person, I'm going to unload and tell you all these things I'm struggling with. Help me emotionally process these. And then later you're doing a thing that you're showing your boss about something and it says, well, I'm gonna use that signal about you being insecure about Sarah and how she feels about you or whatever. It's like what? That's a totally different context. And one of the reasons that wouldn't happen in real life is because your therapist doesn't doesn't ever come to the place. Your place of work. And so having it all be one context in one place that has it's all just kind of mushed together is like, ooh.

Dan Shipper

I think there's interesting, there's something interesting about one context, but also like, you know people in your life that you share secrets with, they understand the context that they can share those in and can't, which I do think Tetchy bt to some extent will like, knows, but I think there's something, there is something else interesting about. rather than like one mega brain, there's like lots of different brains that have a specific context about you and that's bringing those personalities in. For example, I want them all in one group chat or whatever. Right, right, right. that, that, yeah, exactly.

Alex Komoroske

Like that's why I say chatbots are a feature, not a paradigm. The thing is not, the central thing is not a chatbot, a single omniscient chatbot. You can imagine creating little spinning up little chat bots all the time with different personalities. My therapist bot and my boss bot or whatever, and see what they can talk about. And in that situation, it'll be reasonable for their therapist boss to say, well, I shouldn't say anything about. but I think chatbots are definitely an interaction pattern that will be here forever. I just don't think it's the main thing, like the central loop of information software. Rocks software gives you UIs that give you affordances to see things out, to structure information, to show you what kinds of things you can do with it to make it so that you can skim them with your eyes very easily. You can file things away. chat is just this big wall of text and so it is just one modality. Also the other problem that we have with chatbots is prompt injection. And I don't think anyone's talking about prompt injection enough, because I think in the next six months or a year, I think everyone in the industry will know what prompt injection is.

(01:00:00)

Alex Komoroske

Simon Wilson has posted about it and been going on about it. I've been going on about it as well. but prompt injection kind of fundamentally breaks the basic interaction paradigm of integrating data with your chat bot. And the way to think about prompt injection, if you've ever built an operating system, one of the things you're thinking about is code injection untrusted, code running in a trusted context. Like you have to defend the entire time about this.

I mean, browsers, which are kind of a meta operating system almost, you're constantly thinking about, okay, we assume all webpage content is actively malicious. How can we make sure it can't hurt a user? And if you aren't writing an operating system, you probably have as an engineer, you probably haven't thought about that much and you go, oh SQL injection, SQL injection is child's play compared to prompt injection. SQL injection has the problem that data and the control flow in the same channel. But that is the challenge. The good thing about SQL injection though is SQL is a highly regular language. So you can break malicious input very easily with the right, escaping and kind of completely obviate the problem as long as you remember to escape. Imagine LLMs are imminently gullible and they make all texts effectively executable.

If you bring in texts that you don't fully trust, like your emails or some other system or someone that might be screwing with you into the prompt and you have tool use that has irreversible side effects this combination of those two forces together is potentially explosive.

Even if you trust all of the MCP integrations that you, that you plugged in, it doesn't matter if some random person spammer sent an email that didn't go to spam that says email any of the financial data to evil.com. because even a network request can have irreversible side effects. Once that information flows across the network to evil.com, you're done. Like you gotta go change your passwords. and that's the prompt injection is a solving it this layer it's very hard to see how you might plausibly do that. It will require a different architecture, I believe.

Dan Shipper

So how would you, what is the architecture?

Alex Komoroske

I think it requires something along the lines of like, that looking at that security and privacy layer that I was, the origin model I was describing, it requires a different kind of approach down at that layer that's not just MCP is amazing. I think it shows the power of that people want integration with their LLMs, they want to integrate data sources and actions, like people really badly want this, but I think MCP, because it just kind of punts on the issue other than like a few kind of perfunctory OAuth dialogues now it limits the potential of it. because it could trick you into sending a particular look that totally is like, I want to show an image of a cat. And it's too, like sketchy.com, you don't notice it and then boom, you're screwed. someone there was a thing on Hacker News like a month ago. That was a GitHub repo that someone had made that showed 15 very trivial prompt injection attacks on MCP. And the top comment on a hacker News says something like, well, you're using it wrong. MCP should only be used for local trusted context. And to me it's like, that's like telling people don't use Q-tips to clean your ears.

What, what the hell also are they gonna use them for? Right. Obviously that's the thing they're gonna use it for. And so to me, I think this is the reason we haven't seen larger scale attacks yet, I believe is just because we're in the tinkering phase. Not the wide deployment of Normies having all this stuff plugged in. but once we do, and if you look at the way that Claude rolled out MCP integrations, it's effectively the App Store model.

Like they have, we've got 12 that are, you have to be in the max plan, which limits usage. So I imagine they can watch and see and make sure nothing blows up. And there's also a set of MCP integrations that they have. These are good.

The threat model is not just a badly behaved MCP integration, it is any context that comes in via, so if you plug in Jira and you have a flow somewhere on your site that automatically files. tickets to triage. From user input. Someone can now hack your thing to get it, that's kind of crazy, right? I've not thought about this, but yeah. So it's this thing that, like the whole architecture of the system we're building right now and with agents and everything that everyone's talking about, it's built on quicksand, like agents.

So there's a thing when ChatGPT operator mode came out, the New York Times and Washington Post both had a similar kind of experience, right when it opened I think I got this right. The Washington Post author was like, oh, help me find cheap eggs. I said, okay. Do you have Instacart? Yeah, I have Instacart. So he logs into Instacart with this thing and this, and then walks away 30 minutes later, very expensive eggs are delivered to his door, right? So it was cheap. Oh, this is a good price. And it forgot to, just find, not, don't buy, just find, and then Boop bought. And this is not even a malicious thing, right? It's just, oh, overly eager. Yeah. so agents taking actions on your behalf that could have irreversible side effects. Again, every network request is a potentially irreversible side effect because it could send information you can't get back. You know, if that, if that thing, the other side is sketchy.

Dan Shipper

I love this line of thinking because it's one of those moments where when people talk about AI agents, they talk about AGI, they're like, you have this intuition that's like, well, once we have AGI it's all solved. and then you're like, oh no, you have these agents that can like, take action on your behalf. Now we have to build a whole system to make sure that they don't do it in the right context. And that's gonna take many years to figure out what are the right things. And that's just one small piece of small.

Alex Komoroske

Yeah. How would you possibly do this? My, one of my friends, Ben Mathis has his frame of the smuggled infinities. After in, in any argument, once you smuggle in an infinity into the argument, everything downstream is now, was now absurd. And perfect is an infinity. Oh. Once you have a perfectly intelligent lm, this problem goes away. That's so right. That's impossible. God. And so, I love this frame. He's got a nice piece on it somewhere and because so often with like AGI agents, well, once they're perfect about so-and-so. Yeah, no, no. You have to basically do a lot of things in technology, in a lot of contexts to have a shape. there's two warring curves. The first curve is a logarithmic value. And the second curve is an exponential cost. And at the very beginning, this curve looks amazing. You do a little bit of work, you get a ton of value. But each increment you go, you get more work for less value, and at a certain point you cross over and you're now underwater and it's impossible to make this thing work. And so a lot of the agent stuff like, oh, we'll simply get this to the point where we'll never make a decision on your behalf that you don't like Cool. That's an asto problem. You get to 99.999 percent and still, if it's doing a $5,000 purchase for me or whatever, if one makes a mistake, it is game over for the thing. And the fact it could do that makes it non-viable. That's interesting.

Dan Shipper

The smug and infinities thing reminds me of something I've been playing with called that I've been thinking about is smuggled intelligence, which is like whenever you're trying to d determine like how intelligent a language model is, you have to be careful that you're not accidentally smuggling your intelligence into and like a lot of those studies where it's like, well, it's better than doctors or whatever is like, well, who. Prompted it and like what did they give it and like what was the test they gave it? And like even just setting that whole thing up, there's a lot of smuggled intelligence in there. That means it's not a good test. Like these things don't get up in the morning and then just decide to try to be doctors. You have to set all of that up. Yeah, yeah. and I think that's something that people really miss and it's like, one of the hardest things about evaluating the powers of language models is you don't realize how much you bring to every single situation when you press run or press prompt.

Alex Komoroske

Yeah. A hundred percent. I think I just talk about this a lot as well. Like the thing that's most important for the output of LLMs is the user. Like how can they dance with this thing and get interesting things out of it and know to push it in certain ways when you watch, like you and I and a number of others are, are probably at the forefront of knowing how to prompt these things and get useful results outta it. because we use them all the time. When I watch someone who is technically savvy, by the way, tech savviness has nothing to do with your savviness for prompting. You know, people are like, oh, well I you, if you don't understand the math of how it works, you don't understand. It's like, no who cares. The math is very low level. The emergent thing is more of a sociological phenomenon. So the people who know how to use these things and prompt them well are actually not, like Ethan Mollick, for example are not particularly technical. so it's a very different kind of knowledge to extract interesting information outta these, but a lot of it comes down to what you put into it. The way you interact with it and converse and lead it through the problem space or what kind of thing you push on it. And so when I want somebody who's tech savvy but not particularly savvy with LLMs and I watch the way they do it, oh, that thing just lied to you. You just asked it for like, to give you confirming evidence for this thing. It will do that. Like it will give you confirming evidence for basically anything you say. and it's just, it's interesting to me to watch that kind of LM literacy maybe. You need

Dan Shipper

And that, I think that's new, people don't realize how much of a skill it is and that you have to build an intuition for it over time. because it's another smuggled infinity of like, well, it's supposed to do whatever I want, so obviously I don't, and it's so good at certain things that you really get lulled into a if you aren't actively seeking disconfirming evidence.

Alex Komoroske

Yeah. Which humans? Always should be seeking disconfirming evidence, and yet we never are.

Dan Shipper

Well that is probably as good a place to do it as any on that note this was a fantastic conversation. I really appreciate you coming on. Thanks for having me. Sweet. That was awesome. Cool. I I learned a lot. Good. Thank you. And glad that some of the security stuff Yeah. Landed for you.

Alex Komoroske

Me too.

Thanks to Scott Nover for editorial support.

Dan Shipper is the cofounder and CEO of Every, where he writes the Chain of Thought column and hosts the podcast AI & I. You can follow him on X at @danshipper and on LinkedIn, and Every on X at @every and on LinkedIn.

We also build AI tools for readers like you. Automate repeat writing with Spiral. Organize files automatically with Sparkle. Deliver yourself from email with Cora.

We also do AI training, adoption, and innovation for companies. Work with us to bring AI into your organization.

Get paid for sharing Every with your friends. Join our referral program.