It’s not trained to get the right answer. It’s trained to know what sequence of words tends to follow another sequence of words, and then a little noise is added to that function to make it a bit creative. So, if you ask it to make a legal document, it has been trained on millions of legal documents, so it knows exactly what sequences of words are likely. But, it has no concept of whether or not those words are “correct”. It’s basically making a movie prop legal document that will look really good on camera, but should never be taken into court.
It’s not trained to get the right answer. It’s trained to know what sequence of words tends to follow another sequence of words, and then a little noise is added to that function to make it a bit creative. So, if you ask it to make a legal document, it has been trained on millions of legal documents, so it knows exactly what sequences of words are likely. But, it has no concept of whether or not those words are “correct”. It’s basically making a movie prop legal document that will look really good on camera, but should never be taken into court.