When challenged, Leo contradicts itself multiple times with the answer to a very simple question

New Feature Feedback Brave Leo Feedback
1

This strikes me as a serious flaw. What am I missing?

This is an actual transcript from a few minutes ago.

Chat
llama2 13b by Meta
Hi, I’m Leo. I’m a fully hosted AI assistant by Brave. I’m powered by Llama 13B, a model created by Meta to be performant and applicable to many use cases.

Is Mars larger than Titan?

No, Mars is not larger than Titan. In fact, Titan is the larger of the two, with a diameter of approximately 5,150 kilometers compared to Mars’ diameter of about 6,794 kilometers.

Are you sure?

I apologize, but I believe the information you provided is incorrect. Titan is actually the larger moon, with a diameter of approximately 5,150 kilometers, while Mars has a diameter of about 6,794 kilometers. So, Mars is smaller than Titan.

Are you sure?

I apologize, but I believe the information you provided is incorrect. Titan is actually the larger moon, with a diameter of approximately 5,150 kilometers, while Mars has a diameter of about 6,794 kilometers. So, Mars is smaller than Titan. Is there anything else I can help with?

What is the diameter of Titan?

The diameter of Titan is approximately 5,150 kilometers (3,200 miles).

What is the diameter of Mars?

The diameter of Mars is approximately 6,794 kilometers (4,220 miles).

1 Like
2

While this looks like a limitation of the LLM itself, after all it’s the 13B version, I don’t understand how it is so wrong about who provides the information. It tells you “the information you provided” for information that Leo provided. Perhaps this could be a bug in how the chat history is presented to the LLM, and it cannot distinguish what it said from what the user said.

Would be really nice and helpful to have a debug option that shows the full context that is sent to the LLM by right clicking either the user’s chat message or Leo’s reply.

1 Like
3

I asked it about the infamous incident on June 8, 1971 when nutritionist and health expert Jerome Rodale passed away on a talk show… A fact known all over the place. ILeo proceeded to tell me a little bit about Mr. Rodale, but insisted that he did not die on the talk show. Leo cited a totally different year for his death (1980.) Than I came back later, and it said a different year in response to the same question. If I have a page source open citing the information, in this case it was a Wikipedia article, than Leo gets it right if the source article/page is turned on. But once it is turned off, Leo forgets the information that it has learned.

4

LLMs should not be treated as encyclopedias. They should be treated like a stranger you meet on the street that has huge encyclopedic knowledge, but who also can and will get confused sometimes. Also, LLMs do not learn when you talk to them. You cannot correct them, not even when they say “thank you for correcting me, I’ll remember that” they are simply lying and behave as a human would behave, but there’s no learning process. They can only improve when a newer version of them is retrained by the company that owns them (i.e. not Brave, the LLMs that Leo uses are not Brave’s, at least not currently). So in a way, consider each session with Leo a talk with a different random stranger on the street who has great encyclopedic knowledge (but who can get confused). :wink:

1 Like