Disclaimer : All words are my own writing. These are only my views (although without all the detail these ideas deserve), only at the time of writing.
I can’t believe nobody’s done this yet, but I want to write some about AI. Less of the specifics, more of the big-picture view I use to try and process this. My thoughts aren’t new, but I’m doing this to wrangle all the half-baked thoughts in my head and have something to look back on. Perhaps I say something of interest to you.
TL;DR
Whatever the future may be, I believe it will be more wild and unexpected than we can expect. Coming to terms with that challenges every instinct our biology and culture has primed us with, but upon doing so you can gain a newfound appreciation for the great big story we’re watching unfold and also participating in.
Toeing the Event Horizon
I believe that some form of superintelligence is possible and coming faster than anybody is ready for. I predict there will be no consensus to pause development because the game-theoretic incentive structures are not there. Besides, it’s very possible it’s too late for a pause, anyway.
What I’m going to say here is not dependent on the current state of AI. It doesn’t matter if Claude stinks at chess. The rate of improvement and the unoccupied space ahead is large enough to convince me that if companies directed development towards it, they will solve video generation, they will solve all open math problems, and they will solve the holy grail enabling the flywheel of all flywheels: AI research and development.
To all the others who saw the exponential far ahead of the ChatGPT moment and seriously reoriented around it, I am impressed. Although late to the game, I’m trying to see trends and make extrapolations. Whether superintelligence comes in 1, 10, 100, or 1000 years will not change the opinions shared in the rest of this essay.
So while I hope we cure all disease and eliminate all absolute poverty, I don’t know if those will happen and will not speculate on that here.
Heaven or Hell
When it comes to predicting the future of AI, it tends to be very black or white. Depending on who you are, some of the future will resonate with you, and other parts will mortify you.
Some days I’m filled with a childlike excitement when I watch a captivating movie. we don’t know what’s going to happen. Nobody can convince me of what exactly will go down these next months, and people have become more hesitant to make predictions beyond a few years. As a convenience, I still talk about “old age” and “the year 2060”, but I have no clue what anything will be like in a few years, let alone a few decades. Perhaps my friends and I can look back at this essay and laugh at how obvious it all seems in hindsight.
Hearing people talk about AI is a fascinating scope into a person’s politics, religion, morals, self-awareness, how they relate world events to personal life, how they think about time and long term. The stories we imagine of AI-filled futures threaten to disrupt the many stories we tell about ourselves. I guess this essay is an attempt at reconciling the “human story” with the “AI story” into a new story which satisfies my urge to make sense of what’s going on.
I listen to the All-In podcast where the knowledgeable hosts (who also have great prediction track records, I should add) clearly articulate how fears of AI takeover are a psyop, how a quick study of history would give us examples of similar equally scary innovations that came throughout history but did not kill everyone. On the other side of it, I also listen to podcasts where brilliant intellectuals who are building AI systems outline how this time is different and how impossible and destructive misalignment is.
Here is the crux of their disagreement: self-stability. The AI-optimists think the system of humans and fundamental constraints of this world will automatically adapt or simply render impossible the hyper-exponential “foom” feared by AI-doomers. The AI-doomers think this alien intelligence will be capable of acting on unknown unknowns that we are not privy to, and that our human systems plus the constraints of our world allow for such an explosion to overcome any limiting forces. And when I frame it like that, I side more with the doomer camp.
Still, I live most of my life in optimism camp, but when reminded of unknown unknowns, of trying to look at things from the broadest possible view, I see what’s coming. I can’t always hold that view, so I eventually go back to pretending like life will carry on as it was.
Many of us will be wrong whichever way things play out. Nobody knows the future, especially in a system as chaotic as this one.
Morning Boys, How’s the Value Drift?
I claim we can’t know what we actually want. And I’ll point to Value Drift as just one example of the fungibility of how we view reality.
Value Drift is the idea that what’s important to individuals or groups of people changes drastically over time. Many people value scientific publications now who, if were born centuries ago, would’ve attended the burning of the heretical scientists who wrote those papers. Slavery used to be central to the cultural operating system in many places and accepted or even encouraged, but is now largely viewed as one of the worst moral violations. I don’t deny the possibility of a future where people consider the well-being of chickens and cows as much as they do dogs and babies.
It can be an eye-opening yet helplessness-inducing experience to work through the implications of value drift. What was once the way of the world is now just one of many ways of the world. It’s almost… arbitrary. To me, it means there are many versions of “right” depending on who you ask, but my collection of values is unlikely to be the end state.
However, that doesn’t mean we should abandon our values. This value drift might be an unending process of getting closer to “better” value sets (although I do not claim how we could evaluate that). Either way, I don’t think we’re at perfection. And that means we can’t shun new ideas just because they challenge what we think is right. They are probably well-intentioned, just looking from a different perspective. Politics and philosophy is the ultimate expression of slight variations in fundamental beliefs on self and world and extrapolated and diverging into vastly different sets of values one holds.
The Ant-Human Alignment Problem
Alignment of any kind is difficult. Aligning a single human to accomplish the goals they genuinely desire is a rare feat of what we call “self-discipline”. Aligning teams and families and companies to common values, expectations, and behaviors is notoriously contentious. Aligning whole countries is impossible, aligning the entire species would involve a suppression of individual thinking that goes against Western individuality.
The alignment problem asks how to align machine actions to the values of the creators. With no evidence to give, my intuition doubts there exists a satisfactory solution.
I think that if we can create superintelligence and then realize we need to go to war against it, we’ve picked a fight we are hopelessly under-organized and under-intelligenced to win.
I do not know anything about the technical content of the field. For now, I remain skeptical. I think it’s a noble effort, and I am glad some of the brightest minds are exploring the space, but I think the task is akin to perfectly predicting the stock market or a single person declaring war on the United States. Or, as AI Researcher Connor Leahy and blogger Tim Urban analogize, like ants trying to align humankind to their ant wishes. Ants are smart in their own way, but according to the game we’re playing, we are the ones in charge.
If alignment work claims to make progress, it should be able to answer simple questions, such as:
- what are we aligning to?
- how can we prevent intelligences from deceiving and harming us?
And the policy-makers get their own interrogation:
- what are we aligning to?
- who gets to decide?
Maybe these are wrong questions, that my assumptions and analogies are mistaken somewhere. And hey, I will love them forever if they pull a giga-hail-Mary and “solve” alignment.
The Beginning of Infinity?
The utter asymmetry between intelligences could mean that superintelligences could pretty easily gain control over the very same resources us humans need to survive and thrive. I won’t speculate on if this is a slow or rapid dethroning, peaceful or violent. I am also open to the idea (actually, secretly hoping, like everybody else) that superintelligences go off and do their own thing and leave us alone, only coming back to spoil us with wonders of discovery and gift us with other-worldly technology.
Does everything become pointless if 21st-century human culture and values are overtaken? Not necessarily.
I’ll give you my chain-of-thought on this one:
Change is constant. We’re so stuck in seeing only the past few years/administrations/generations that it’s hard to imagine (or even know the history of) what life was like beforehand. And we also forget that with our current understanding of reality, there’s potentially billions of years left in this universe. And that’s just in subjective human years, in this universe, according to our ever-changing scientific theories.
Change being constant means that if AGI comes, it is also another chapter in history. It also means at some point, whether in 50 years or 50 million, we will not be the human beings the way our ancestors were for the past thousands of generations.
Take a long-term, wide-lensed view of the universe as we know it. It somehow banged into existence, then slowly coagulated into galaxies and then eventually even cells and bridges and computers. And perhaps it’s just getting started…
And we get to be during it, captivated by the suspense of not knowing which direction the story will go.
Are you not entertained? I can’t think of a more fascinating story and exciting plot, fiction or non-fiction. What a time to be alive. What a time to be born, and what a time to experience this wave with a sound mind.
David Deutsch calls this “The Beginning of Infinity”, and I think it’s beautiful.
The Light of Consciousness and the Truth Attractor
One small problem with this fantasy of star-faring entities:
If it’s not us who visit the stars, not even a biological offshoot of us, would they feel it? Would they experience the joy and love and fear that we see as synonymous with existence? Or, would they at least have alien emotions instead of a devoid inner life? Would there still be consciousness in the cosmos?
I used to not think much about consciousness. It simply didn’t seem like a big deal to me. Then one day things changed, and it seemed like the most important thing imaginable. That somehow we have this feeling of being, and nothing about nature can explain why we do.
There’s a phrase that Elon Musk uses to describe his hopes for the future which I mostly agree with: “Preserve The Light of Consciousness”. It seems to imply the universe needs some consciousness entities to observe it and make sense of it all, or else it doesn’t really matter what’s happening.
And so, our faith in conscious experience mattering beyond value drift gives us confidence to make one feeble plea to future intelligences: Please don’t let this great thing we call “consciousness” die. At least preserve the option for positive experience to continue and for negative experience to dwindle.
I think future intelligences will be better able to explain, create, and manipulate conscious states. I hope they use that power for good, but the details of that are beyond my and everyone else’s expertise. What a wonderful thought: that in the future, people (or something descended from us) might discuss the underlying mechanics of consciousness, much like how we discuss the structure of cells or the formation of planets.
How I’m preparing for superintelligence
I’m not. I guess I’m optimistically preparing for friendly superintelligence since I have no other strategy that allows me to stay sane.
But hostile ASI? I don’t think there’s much I can do to prepare for the downside. It’s good to periodically remind myself of the risk, that I might not actually get to live that bright future of abundance, that millions of heroes actually need to create that bright future for us all.
In my situation it makes sense to be invested in technology, to not attach myself to any long-term career ambitions, and to try and adapt to the new times while my mind and body can handle it.
It’s not totally rational, and it’s not for everyone. But, as of October 2026, this is the right way for me.
Thank you for reading, and I would love to hear your thoughts.