AI is becoming humanity's default thinking partner.

It costs very little now to get expert information on anything, no matter how obscure the topic or how detailed the question. Even on day to day matters, AI provides intelligent answers that might not be obvious to the user. It’s foolish to not utilize a tool that synthesizes humanity’s collective knowledge at will. Not just for understanding the world better, but for personal decisions too.

You’re pretty sure you sound confident yet professional in that email but why not just ask AI one more time, just a little sanity check that you’re not saying anything you’re not supposed to.

You know your workplace is kind of toxic, and you’ve had this itch for a while to start something of your own, to build something meaningful. But why not ask AI one more time if it makes sense to do it, maybe you’re taking an unreasonable risk. AI will tell you what’s reasonable.

You know you want to be a film maker, but what if it’s unrealistic, what would your parents say? Why not ask AI to get a second opinion?

The all-knowing nature of our new tool and propensity of humans to seek approval and validation make it imperative that AI will increasingly be used in personal matters. The genie is out of the bottle, there is no going back at this point.

This is not always bad. In a lot of cases, AI can save people from their own lunacy, and even revert the effects of echo chambers and radicalization brought on by social media. When used well, access to information & decades of research on what works and what doesn’t when it comes to personal development, synthesized in a way that makes is immediately accessible to the user is a giant step forward for humanity.

However, the current AI landscape presents an immediate and existential risk to human ingenuity.

The thoughts and ideas of the entire world are increasingly shaped by a handful of labs in silicon valley. The companies that develop models shape the way we make decisions, the way we reason about our personal life, and ultimately, the way we think.

Even if we assume these companies have our best interests in heart, which is a bold assumption, answers to such deeply personal questions require deep personal knowledge. It is not possible for models alone to handle personal matters in a useful way for everyone, because, by definition, personal problems are, well, personal.

Here, model providers have to make choices, they have to make decisions on what is good for people, and train their model towards those goals. And there are no good choices here.

Most model providers train their model to minimize harm. The model will not take bold stances, nudge people to take risks that might have bad consequences, and in general operate out of fear rather than excitement. Anthropic, especially, is a very vocal proponent of this stance. The issue with this stance is that humanity has been able to achieve tremendous feats of progress precisely because we take big risks, do things that don’t make immediate sense, and push back on what others think is reasonable. By making the model consider possible harms first, they are stubbing the human spirit that caused us to create such a technology in the first place.

Obviously, the opposite approach is not great either. A model that nudge people to take unreasonable risks, validate their harmful thought patterns, and is irresponsible about the risks caused by the information it provides can cause harm on a massive scale not seen before.

Minimizing harm vs encouraging risk-taking isn't the only call model providers have to make. By their nature, personal questions contain ethical frames. The model can either refuse to answer it usefully, or adopt the ethical framework of its creators. In practice, this means models adopt the dominant ethical frameworks of it’s creators. xAI’s Grok aims to be maximally truth seeking even if truth might hurt the user, OpenAI’s ChatGPT puts more value on what the user might feel, while both answer from a Western Individualistic ethical framework.

The real issue here is that in the absence of deep understanding of the user, one way or another, models end up guiding the user towards a singular understanding of the world. The only way out is to not let the model providers make these choices for us.

A personalized model of each one of us that deeply understand our values, principles, judgement and taste. A world where each person uses deeply personal AI that preserves their ingenuity. This is the future we should aim for.