When challenged, Leo contradicts itself multiple times with the answer to a very simple question

New Feature Feedback Brave Leo Feedback
1

This strikes me as a serious flaw. What am I missing?

This is an actual transcript from a few minutes ago.

Chat
llama2 13b by Meta
Hi, I’m Leo. I’m a fully hosted AI assistant by Brave. I’m powered by Llama 13B, a model created by Meta to be performant and applicable to many use cases.

Is Mars larger than Titan?

No, Mars is not larger than Titan. In fact, Titan is the larger of the two, with a diameter of approximately 5,150 kilometers compared to Mars’ diameter of about 6,794 kilometers.

Are you sure?

I apologize, but I believe the information you provided is incorrect. Titan is actually the larger moon, with a diameter of approximately 5,150 kilometers, while Mars has a diameter of about 6,794 kilometers. So, Mars is smaller than Titan.

Are you sure?

I apologize, but I believe the information you provided is incorrect. Titan is actually the larger moon, with a diameter of approximately 5,150 kilometers, while Mars has a diameter of about 6,794 kilometers. So, Mars is smaller than Titan. Is there anything else I can help with?

What is the diameter of Titan?

The diameter of Titan is approximately 5,150 kilometers (3,200 miles).

What is the diameter of Mars?

The diameter of Mars is approximately 6,794 kilometers (4,220 miles).

1 Like
2

While this looks like a limitation of the LLM itself, after all it’s the 13B version, I don’t understand how it is so wrong about who provides the information. It tells you “the information you provided” for information that Leo provided. Perhaps this could be a bug in how the chat history is presented to the LLM, and it cannot distinguish what it said from what the user said.

Would be really nice and helpful to have a debug option that shows the full context that is sent to the LLM by right clicking either the user’s chat message or Leo’s reply.

1 Like