“All large language models, by the very nature of their architecture, are inherently and irredeemably unreliable narrators,” said Grady Booch, a renowned computer scientist. At a basic level, they’re designed to generate answers that sound coherent — not answers that are true. “As such, they simply cannot be ‘fixed,’” he said, because making things up is “an inescapable property of how they work.”
http://www.msn.com/en-us/news/technology/google-s-weird-ai-answers-hint-at-a-fundamental-problem/ar-BB1ngeqq?ocid=BingNewsSerp
Noticed these kinds of flaws myself when the subject was something I knew about.