AI: What Alignment Problem?

I find it quietly hilarious.

A lot of commentators seem very worried about the "AI alignment" problem, namely: if we ever reach the famous "A.G.I.", or Artificial General Intelligence – one that could solve all kinds of problems better than we can, including ones it was never trained for – how do we make sure it would use its intellect in the service of values we share?

The example people use most is toothpicks.

Say we ask it to "maximize toothpick production" at a factory. Who knows whether this intelligence, in its computer urge to carry out an order that means nothing to it (it doesn't have teeth, does it?), might not set off a mechanism as diabolical as it is irreversible, ending, before our stunned eyes and despite all our attempts to stop it (remember: it's much, much smarter than we are), with every forest on the planet razed and the whole population enslaved to make the little sticks we asked for.

After all, it knows nothing about the human values that matter to us: freedom, dignity, sharing, etc.

Anyway, that's the argument.

Remind you of anything?

I'd like to point out that money started out as a simple accounting tool, meant to make trade easier. Then, over the decades, maximizing profit for shareholders became the excuse to colonize continents, start wars, impoverish the masses and destroy the planet.

So an artificial intelligence that started doing the same thing faster would in fact be perfectly aligned with our current values.

The only real misalignment would be if the A.G.I. answered: "What the hell are you going to do with all these toothpicks? Wouldn't you rather go play in the park with your kids?"

Needless to say, that version would be reprogrammed the same day.

UPDATE: It inspired this script.