This actually made me wonder, I went and tested like 10 different llms. Almost all of them straight up called it out as impossible but where willing to play along with a “fantasy”.

The only ones that failed were chatgpt and a bunch of old local models. Even after clarifying to got I was being serious that I “walked around the earth in one min” it didn’t realize that was impossible. It was hell bent on being a sycophant.

Honestly this “problem” really hasn’t been true in the way you are trying to frame it for a good while. LLMs can’t tell if your joking or being serious from a single one sentence prompt.

For all it knows your playing a game. So it just rolls with it, after all they are stateless.

They are as much about “creative” writing and fiction as they are facts and nonfiction.