Hello! I just saw Brave Leo’s announcement and rushed to try it out (like I’m sure many people did, given that it seems to be down for me )
First off, I think Brave Leo will be a great feature if it works properly! I hope it scales well; it’s good to have more competition for ChatGPT and Microsoft Bing.
I enjoy breaking AI language models, so if I find any way to break the model, I’ll post it here. I would also be happy to give feedback directly in Brave (like how ChatGPT has the ), and I would be more than happy to give the fine folks working at Brave access to (most of) my conversation history as sample data - whatever I can do to help make Leo more helpful to others!
Here are some examples I found so far of where the AI gave unexpected answers:
Leo seems to be down for me for now, but when it’s back up, I’ll enjoy breaking testing Brave Leo out!
I see that Leo is based on Lambda Llama with no extra fine-tuning, so hopefully, I can help provide examples of where the AI model may need to be improved.
Umm, not down for me. What are you seeing that you mention it was down?
Yeah, it’ll be getting there. Keep in mind it’s only in Nightly that it’s available. Nightly is always the testing and development version. So it’s generally not going to be anywhere near perfect and things will change.
And yeah, I tested the time one. My answer is slightly different. I tried plugging in a state just in case that mattered, but no go.
First off, I’m sorry that my last post was so negative! I updated it to be less so.
Sorry about getting the model wrong - you’re right. With so many AI models to keep track of, I guess I got them confused! I had seen https://github.com/brave/brave-browser/wiki/Brave-Leo#versions, but misread it. I’ll update my original post to match that.
When it was down, I was getting a bunch of messages like this for any query I gave it:
It seems to be back up now. It was funny, however; I was sending some messages correcting it where it left off a question mark, then I started typing in internet slang to see how it responded. Ironically, it stopped working right after I told it “Lol get rekt”
I also get that it’s a nightly release with lots of work coming up; it doesn’t yet have access to the internet, for example, but come will eventually thru Brave Search.
I’m sorry if my last post came off as argumentative, negative, or anything like that. I read it back, and I probably updated it to make it more friendly. I enjoy playing with and breaking ChatGPT and working with things like Lakera’s Gandalf, so I’ll enjoy having a new toy to play with. I’m a dev, I wrote my senior thesis on ChatGPT, and I enjoy trying to push systems to their limits to try to break them!
I really hope this works well. I trust Brave as my browser and my search engine and I recommend it to everyone. I also know that AI takes a lot of processing power, so it can get costly fast, especially as this scales up.
I’m really positive and hopeful for Leo’s future, and I look forward to seeing where it goes!
We’ll get it added to https://status.brave.com when we get to release. We also have a little bit clearer messaging on the error messages. We’ll need to see how we improve those errors messages as well.
Well, as I was thinking thru and testing out some responses for my next post, you went ahead and released a roadmap! I was coming up with a short list of improvements, and you already covered almost all of them!
Between the time Leo was first released and now, I’ve noticed it becoming significantly more accurate, especially with technical questions. Great job with the rapid improvements and amazing roadmap!
Here are the features I was going to ask about but which you have already announced:
Having Brave Leo on its own page rather than only in a sidebar
Using Brave’s sync capabilities to sync a chat history
The ability to customize a system prompt
More clear messages if you hit a rate limit
And honestly? Those were the major additions I wanted! You are knocking it out of the park with this!
The only other changes I had wanted were small ones:
Since Leo speaks using Markdown formatting (for example, code boxes ````), it would be nice if the page supported that
The ability to edit one of your previous messages or refresh the AI’s output if it’s incorrect. You mentioned the ability to resume a past conversation, so I would think there’d be a way to change the past conversation, so this is sort of already covered
When I enter a prompt, it often unselects the input box at the bottom after my message. It could be nice if the box remained selected to allow for more rapid responses
Now that you also said you’re looking into adding support for local models, that could be amazing! I’m sorry that my original post sounded so negative, and I look forward to seeing where Leo goes!
)
), and I would be more than happy to give the fine folks working at Brave access to (most of) my conversation history as sample data - whatever I can do to help make Leo more helpful to others!
