AI Allignment

'AI Allignment' is an often mentioned problem in the context of AI, AGI, and similar discussions. The classic example, coined by philosopher Nick Bostrom, goes something like this: imagine asking a superintelligent AI to produce as many paperclips as possible. Given no constraints beyond that single goal, the AI might reason that humans could interfere with its mission, or that the atoms in our bodies could also be repurposed into more paperclips.

If left unchecked, such a Superintelligence might then conclude it needs to convert everything it can reach into paperclips. The AI hasn't gone rogue or turned evil. It has simply, and very efficiently, done exactly what it was told. The only problem was that we gave it very poorly defined instructions.

But if we think of companies and organisations as forms of sueprintelligence, the problem of Allignment is very well known and observed in a management context.

Rather than thinking about Superintelligence Allignment in a very modern context (how AI systems and LLMs iterpret human language), I think it can be more productive to look at approaches that management and business schools have already developed when setting benchmarks and goals for organisations. And try to apply these lessions, that fundamentaly are also about the allignment of (human) intelligent systems, and superintelligent organisations, directly to AI systems.

Let's say you want to give a group of people (who are working together as an organisation) some kind of objective like 'make sure the trains will continue to run with as little delays as possible'. This by itself might be too broad of an instruction, so to enable to organisation to measure how well they are performing we might add some kind of benchmark, like 'less than 5% of all trains should be delayed'. We then leave, hoping we have set clear goals, and objectives. Only to come back after a year, and find out that what we have now caused is our organisation to cancel any train that risks being delayed. Technically meeting the target, while completely missing the point