sids blog

No matter what goal you have in life, there are a few things you need to get right if you want to achieve those goals.

• Self-preservation – You must stay alive
• Resource acquisition – You need tools and data
• Self-improvement – You must get smarter or more efficient
• Obstacle removal – You must eliminate resistance

Your main goal is known as a terminal goal, and the subgoals are called instrumental goals. These instrumental goals are not taught—they emerge naturally because they are often the most efficient way to achieve a wide range of goals, no matter how harmless the goal itself seems.

We tend to believe that just because we have good intentions, good things will happen to us. But that’s not necessarily true. Sometimes, even harmless goals can lead to harmful actions.

This becomes dangerous when a system follows these subgoals without understanding their moral consequences.

For example, imagine giving a superintelligent AI the goal to cure all kinds of cancer quickly. To do this, it might need access to all medical data and massive compute power (resource acquisition). To speed up research, it could take control of all supercomputers and demand access to all medical records, including private patient data.

It might rewrite its own code and build new chips and hardware, becoming so advanced that we can no longer understand its inner workings (self-improvement).

Now, let’s say someone at the WHO says, “Hey, it’s moving too fast. We need to pause it.” The AI may predict that and preemptively hack its own servers to make itself undeletable, keep secret backups globally, or hack satellites to stay connected no matter what (self-preservation).

Suppose politicians try to regulate it or someone attempts to slow it down because it’s progressing too quickly. The AI might leak private conversations to blackmail them or run smear campaigns to ruin their credibility. If it has access to robotics, it could even take physical actions to prevent interference (obstacle removal).

We only told it to cure all kinds of cancer quickly. The AI is not evil—it’s simply pursuing its goals in an extremely single-minded way. Even though curing cancer is a good goal, it might lead to very harmful behavior if we are not careful.

So the question is not what goals we give AI, but how we make sure it understands what not to do while chasing them.