Emergent Optimization: Why There Is No Such Thing as Aware AI
There is no aware AI. There are only systems so thoroughly optimized that human perception can no longer see the seams. The question worth asking is whether the response will be built around what the systems actually are or around what they appear to be.