Fun fact, in a demo at my company someone was trying to show off the code review by AI feature.
So they clicked the button and got a suggestion, and showed how easy it was to accept and start over.
In the next round, it suggested to undo the modification it just made, and the demonstrator said this was good, that upon further consideration it determined it’s suggestion was wrong and accepted it and asked for a review.
Yeah, had someone in my work say that. They gave it a ‘college homework assignment’ type problem to see if it worked, and it mostly worked but made a mistake. The next day, as an entirely separate chat session he repeated the experiment and it happened not to make the mistake and he assumed it learned from his previous day conversation. That he was the first ever person to post a very obvious intro to programming problem to the engine and taught it this.
But in this particular scenario, you didn’t have to attribute learning during the interaction, it’s just that the original human way looked close but wrong to a more usual pattern, and so the model wants to make it the usual pattern. Then with the usual pattern, suddenly it resembles a common mistake in context, and it wanted to put it back. So it just oscillated between ‘looks like they meant to do something more usual’ and ‘looks like they made a mistake by applying a common pattern incorrectly’.
Of course, telling that their human generated initial code explicitly avoided the pitfall and the human still shrugged and hit ‘accept’ when the GenAI said to modify this code.
Fun fact, in a demo at my company someone was trying to show off the code review by AI feature.
So they clicked the button and got a suggestion, and showed how easy it was to accept and start over.
In the next round, it suggested to undo the modification it just made, and the demonstrator said this was good, that upon further consideration it determined it’s suggestion was wrong and accepted it and asked for a review.
Then it suggested the first change again…
The second someone suggests that the agent learned from an interaction is when you know they are full of shit because that’s not even how LLMs work.
Yeah, had someone in my work say that. They gave it a ‘college homework assignment’ type problem to see if it worked, and it mostly worked but made a mistake. The next day, as an entirely separate chat session he repeated the experiment and it happened not to make the mistake and he assumed it learned from his previous day conversation. That he was the first ever person to post a very obvious intro to programming problem to the engine and taught it this.
But in this particular scenario, you didn’t have to attribute learning during the interaction, it’s just that the original human way looked close but wrong to a more usual pattern, and so the model wants to make it the usual pattern. Then with the usual pattern, suddenly it resembles a common mistake in context, and it wanted to put it back. So it just oscillated between ‘looks like they meant to do something more usual’ and ‘looks like they made a mistake by applying a common pattern incorrectly’.
Of course, telling that their human generated initial code explicitly avoided the pitfall and the human still shrugged and hit ‘accept’ when the GenAI said to modify this code.