Ask an environmentalist and a financier whether a company is sustainable. One is asking if it will wreck the planet. The other is asking if it will run out of money. People working toward what could seem like the same goal could be working day to day on tasks that couldn’t be further apart. After automation, we'll find this specificity through data and benchmarks.
I build training environments for AI labs. The hardest parts are picking the right goal and getting how you grade attempts exactly right. What actions should reward the model, and which should punish it? Give a model a score and it will find the fastest way to raise it, including ways you'd never imagine. For example, OpenAI once trained an agent on a boat-racing game. Instead of completing the main course, it learned to circle a small lagoon hitting the same targets as they respawned, optimizing the maximum amount of points possible. It never finished the race, but it successfully outscored the human players. It did exactly what it was told, based on what it was instructed success looked like.
Every goal we hand to AI becomes a score like that. For example, someone has to write down what their goal of being "sustainable" means: emissions per dollar, 10-year returns, or some mix, weighted how? The arguments we used to hide behind a shared word become explicit when they become a benchmark. The benchmarks we use to evaluate “how good AI” is will forever be imperfect and require constant iteration—and it’s the most important work we can do.
The job left for people is the one we've been avoiding: deciding what’s important, writing it down precisely enough for a machine to chase, and checking whether it chased the right thing. The universe doesn't care what we pick as our goal. That's up to us, and it's the fun part.
