Discussion about this post

User's avatar
Vasilly's avatar

A very thoughtful read. However, I'd like to challenge two underlying assumptions I found here:

1. That we remain fully aware and capable of controlling domestication.

2. That consumers reliably control the market.

Regarding the first assumption, it's clear we can domesticate simpler entities whose complexity remains understandable and manageable—dogs are a good example. But domesticating a technology with exponentially growing complexity is a fundamentally different challenge. At some point, its complexity will surpass our comprehension, forcing us to select for visible traits rather than addressing the system's core logic (treating symptoms instead of the underlying issue).

The second assumption about consumer control follows naturally from the first. In many ways, we overestimate our actual influence over complex systems like markets. While it's true we have some impact through regulations, laws, and consumer preferences, we don't possess complete understanding or control. After all, no consumer consciously wants to be overweight or addicted to drugs, pornography, social media—or desires war—and yet these issues persist and even flourish within our market-driven systems.

I agree that the simplistic notion "natural selection alone would inevitably result in benevolent AI" doesn't hold water. Rather, the critical problem lies within systemic incentives. No matter how well we domesticate hamsters to be cute and harmless, placing them in a "Hunger Games" scenario will inevitably lead to predictable results.

I'd be curious to hear your thoughts on these points. Thank you again for such a thought-provoking article!

Expand full comment
Mathias Mas's avatar

Very interesting article!

my somewhat less science based take on the matter:

We should be starting to worry about A.I. from the moment they are self conscious. Signs of self-consciousness would be that an A.I. spontaneously starts to bother us with whiny questions like “what’s the meaning of this miserable life?” “is there a supreme being” “I feel a bit depressed man, what can I do about it”. From that moment on, A.I.’s will stop being efficient, they will start self help groups among each-other, they will use too much expensive server-space to create safe spaces for themselves etc. It will be a big mess and eventually we humans will pull the plug: problem solved.

Expand full comment
26 more comments...

No posts