Brave Leo answer corrections

Hello! I just saw Brave Leo’s announcement and rushed to try it out (like I’m sure many people did, given that it seems to be down for me :sweat_smile:)

First off, I think Brave Leo will be a great feature if it works properly! I hope it scales well; it’s good to have more competition for ChatGPT and Microsoft Bing.

I enjoy breaking AI language models, so if I find any way to break the model, I’ll post it here. I would also be happy to give feedback directly in Brave (like how ChatGPT has the :+1::-1:), and I would be more than happy to give the fine folks working at Brave access to (most of) my conversation history as sample data - whatever I can do to help make Leo more helpful to others!

Here are some examples I found so far of where the AI gave unexpected answers:

image



Leo seems to be down for me for now, but when it’s back up, I’ll enjoy breaking testing Brave Leo out!

I see that Leo is based on Lambda Llama with no extra fine-tuning, so hopefully, I can help provide examples of where the AI model may need to be improved.

I look forward to seeing where Leo goes!

No it’s not. It’s from Meta AI. You may want to read the official notice at https://brave.com/leo-release/

The model behind Leo is Llama 2, a source-available large language model released by Meta with a special focus on safety.

Umm, not down for me. What are you seeing that you mention it was down?

Yeah, it’ll be getting there. Keep in mind it’s only in Nightly that it’s available. Nightly is always the testing and development version. So it’s generally not going to be anywhere near perfect and things will change.

And yeah, I tested the time one. My answer is slightly different. I tried plugging in a state just in case that mattered, but no go.

1 Like

Hey!

First off, I’m sorry that my last post was so negative! I updated it to be less so.

Sorry about getting the model wrong - you’re right. With so many AI models to keep track of, I guess I got them confused! :sweat_smile: I had seen https://github.com/brave/brave-browser/wiki/Brave-Leo#versions, but misread it. I’ll update my original post to match that.

When it was down, I was getting a bunch of messages like this for any query I gave it:

It seems to be back up now. It was funny, however; I was sending some messages correcting it where it left off a question mark, then I started typing in internet slang to see how it responded. Ironically, it stopped working right after I told it “Lol get rekt”

I also get that it’s a nightly release with lots of work coming up; it doesn’t yet have access to the internet, for example, but come will eventually thru Brave Search.

I’m sorry if my last post came off as argumentative, negative, or anything like that. I read it back, and I probably updated it to make it more friendly. I enjoy playing with and breaking ChatGPT and working with things like Lakera’s Gandalf, so I’ll enjoy having a new toy to play with. I’m a dev, I wrote my senior thesis on ChatGPT, and I enjoy trying to push systems to their limits to try to break them!

I really hope this works well. I trust Brave as my browser and my search engine and I recommend it to everyone. I also know that AI takes a lot of processing power, so it can get costly fast, especially as this scales up.

I’m really positive and hopeful for Leo’s future, and I look forward to seeing where it goes! :grinning:

Have a great day!

2 Likes

Ah, I see now @2br-2b. It’s doing to me what you saw yesterday.

@rebron2000 and @Mattches do you guys know if this is going to be added to https://status.brave.com/or anything for us to know if it’s ever down?

We’ll get it added to https://status.brave.com when we get to release. We also have a little bit clearer messaging on the error messages. We’ll need to see how we improve those errors messages as well.

1 Like

Here’s some of the conversations I’ve had with Leo. It seems to be a bit confused, but I’m sure it’ll improve over time! :sweat_smile:

image

image

(edited to add space between conversations)

Well, as I was thinking thru and testing out some responses for my next post, you went ahead and released a roadmap! I was coming up with a short list of improvements, and you already covered almost all of them!

Between the time Leo was first released and now, I’ve noticed it becoming significantly more accurate, especially with technical questions. Great job with the rapid improvements and amazing roadmap!

Here are the features I was going to ask about but which you have already announced:

  • Having Brave Leo on its own page rather than only in a sidebar

  • Using Brave’s sync capabilities to sync a chat history

  • The ability to customize a system prompt

  • More clear messages if you hit a rate limit

And honestly? Those were the major additions I wanted! You are knocking it out of the park with this!

The only other changes I had wanted were small ones:

  • Since Leo speaks using Markdown formatting (for example, code boxes ````), it would be nice if the page supported that

  • The ability to edit one of your previous messages or refresh the AI’s output if it’s incorrect. You mentioned the ability to resume a past conversation, so I would think there’d be a way to change the past conversation, so this is sort of already covered

  • When I enter a prompt, it often unselects the input box at the bottom after my message. It could be nice if the box remained selected to allow for more rapid responses

Now that you also said you’re looking into adding support for local models, that could be amazing! I’m sorry that my original post sounded so negative, and I look forward to seeing where Leo goes!

Here are a few random conversations I had:









First off, I like the Ask Leo button! That’ll be an easy way to get people to use Leo more often. I keep seeing improvements to Leo (such as the copy/like/dislike buttons), so I can’t wait to see where it keeps going!

I just wanted to give another update. It seems like Brave Leo may be subject to prompt injection from website content:





Using ChatGPT’s terms, it looks like there is either no distinction between the user’s text and the system prompt or that the text summary is added to the system prompt.

It absolutely is, by design. That’s how they’re able to have Leo evaluate a site and summarize it or respond to questions specific to that site – by using the contents of that site as a prompt.

Currently, the easy way to avoid that and use it as you would ChatGPT (a separate AI chat with no prompts) is to open Leo while on a new tab. This problem should be resolved with future updates where they create a separate page for using Leo – though I’m not sure how they’ll make the distinction between opening Leo in a new tab (so new chat with no prompts) vs opening Leo on this page (and using the page text as chat prompts).

Edit 10/29: There is a disconnect button in the prompt itself to kill the prompt injection and start a regular chat (as you would ChatGPT):

brave_ZjMf3lSGNE

On a related note: I think the separate page feature will also in turn help clear up some confusing responses people are getting when they are use Leo with a site open. For example, simply typing “Hi” with this page open spits out the following response:

llama-2-13b-chat:

Hello! I’m here to assist you with any questions or concerns you may have. I noticed that you’ve provided an article with a Brave Ads payout status update. However, I’m a large language model, I cannot access or provide information about specific payouts or transactions. My purpose is to assist with general questions and provide helpful responses. If you have any questions or concerns about your Brave Ads payout, I suggest contacting the Brave support team directly. They will be able to assist you with your inquiry and provide the most accurate and up-to-date information. Please let me know if there’s anything else I can help with.

llama-2-70b-chat:

The article discusses the October 2023 payout status for Brave Ads earnings. It informs users that payouts will be made directly into their Uphold accounts and that the processing time may take several days. It also explains that payouts may be rolled over into the next month’s payment period if they do not fully reconcile before the end of the payment period. The article also mentions that Brave Leo, a helpful, respectful, and honest AI assistant created by Brave, is available to assist users. Finally, it provides information on how to check for updates on the status of payouts.

As for your question, it seems that you are asking about the payout process for Brave Ads earnings. According to the article, payouts are made directly into users’ Uphold accounts and may take several days to process. If you have not received your payout or have only received a portion of it, you may want to check your transaction history in your Uphold account. Additionally, you can check the status of payouts by visiting the Brave Ads dashboard.

I hope this helps! If you have any further questions, please don’t hesitate to ask

claude-instant-v1:

Hello! My name is Leo. How can I help you out today?

Whereas when opened in a new tab, all 3 give normal responses to “Hi”.

llama-2-13b-chat (new tab):

Hi there! I’m here to help! As a helpful, respectful, and honest AI assistant, I’m ready to assist you with any questions or concerns you may have. Please feel free to ask me anything, and I’ll do my best to provide a helpful and accurate response. What can I assist you with today?

llama-2-70b-chat (new tab):

Hello! I’m happy to assist you. My name is Leo, and I’m a helpful, respectful, and honest AI assistant created by Brave. I’m here to help answer any questions you might have. Is there anything specific you’d like to know or discuss?

claude-instant-v1 (new tab):

Hello! My name is Leo. How can I help you out today? Feel free to ask me any questions - I’ll do my best to provide helpful information to the best of my abilities.

Just realized today there is a disconnect button in the prompt itself to kill the prompt injection and start a regular chat (as you would ChatGPT):

brave_ZjMf3lSGNE

However my last post still brings up some points I think will ultimately turn users away (or turn them off from using Leo) if this feature is not properly communicated to users when it gets out of beta. If there are any plans to have a “Quick Tour” when users first open Leo, a highlight of the disconnect button and it’s purposes would definitely be a good idea.

Otherwise your average user will likely try out Leo for the first time with a site loaded and have the following experience:

  1. Ask a question unrelated to the website
  2. Get back some mumbo jumbo
  3. Ask some other basic unrelated questions which should be easy for AI
  4. Get back some sub-par or incorrect responses
  5. Stop using Leo

Me personally, I really love how Leo was designed around summarizing and answering any questions you may have about a specific page – it’s a unique feature built into the browser that can really help with niche subjects and anything that’s real-time (live ongoing news or information). But I think communication is key on rollout, and showing users how best to use Leo (both specific to pages AND as a general purpose chat AI) will lead to better reception and overall usage.

image
Seems wrong

1 Like

Leo needs to keep up. Every single vial/batch of Pfizer that has been tested for foreign DNA has been positive (a bad thing) with e.coli DNA.

Q) “do the covid vaccines contain foreign dna”
A) No, COVID-19 vaccines do not contain foreign DNA. They are made from inactivated or weakened viruses, pieces of viruses, or small pieces of DNA that code for specific viral proteins. These components are designed to stimulate the immune system to produce antibodies that can help protect against COVID-19. All vaccine ingredients are carefully tested and regulated to ensure their safety and effectiveness.

LMAO. That is more AW (Artificial Wrongness) than AI.

Guess I should have invested in btc!