Chain of thought doesn’t quite align with this. It’s possible to predict something not in training if there are sufficient facts in the history to align the probability toward a conclusion. (the “teach a man to fish” approach) Reasoning chains were introduced to take advantage of this, but it’s also possible to prime facts through external sourcing that can be used to extend the predictions. (web searching, document traversal, etc.)

Your larger point remains accurate though. Introspection capabilities are functional but quite limited, and training bias can be quite pathological in a vacuum.