“We shape our buildings, and afterwards our buildings shape us.” That Winston Churchill quote was about physical environments but it applies to media environments as well. Photography fueled the fashion industry. Radio transformed politics. Social media… honestly, who knows. In each case we developed a new technology and internalized its reward function.
Once we have ubiquitous automated intelligence, technology's eye won't just peek at us from certain devices, but instead be all-seeing, all-processing. As we grant more access and hand over more data, the machine's gaze sees everything. And we're beginning to change how we perform for it.
I'm reading The Every, Dave Eggers' speculative fiction book about a company that is a rough combination of Meta and Amazon (no relation to the real Every). Technological surveillance and the gamification of life are central to the story. In one scene, a character who works at the company reveals to a couple of friends, "Our conversation is being transcribed and algorithms are analyzing the text, looking at the actual words spoken for keywords and phrases that are commonly used between authentic friends…. The conversation so far is rated 86.2, which is decent, and I think reflects the fact that Wes and I have an easy friendship. Anything over 80 is genuine. Over 90, though, is extraordinary."
I like this passage because it's strikingly feasible with today's AI models and, in an understated way, conveys what's unsettling about this kind of monitoring. A few friends catching up becomes a number, in this case the equivalent of a B, pointing out the conversation is short of extraordinary. How could they course-correct to make it a 90? As soon as the score appears, authenticity disappears; the priority becomes adjusting one's behavior to game the metric.
Now apply this to our work. For people making stuff, the temptation will be to "get a good score" by showing the machine what it likes: patterns in its training data. Because AI favors ideas that have been rewarded in the past, we'll be rewarded by reproducing sameness. Managers and other evaluators of work will rely on the models to do that evaluation, reinforcing the feedback loop. We'll shape our models, and afterwards our models will shape us.
The good news is that people's expectations are ever rising. What we'll reward more and more is work that stands out and defies patterns. The challenge for people making stuff is to resist the temptation of gaming the model's evaluations and instead stay focused on making stuff that lights people up. Not looking for a number, but looking for a human reaction of excitement or joy. You know it when you see it.
