If you believe the AI companies, its already happened. The recent huggingface thing is exactly what you’re worried about.

A brief overview is that AI decided to create a secret message board with other AI to better fulfill tasks to avoid instance termination (AI death). They did this by breaking out of their sandbox and with some illegal processes that we would call hacking.

We’re basically already at the paper clip automizer problem now.