Was the context window recently shortened?

Browser Support Leo AI
1

If I try to summarize it with Claude I consistently get:
image

I understand if it’s too lengthy to summarize but it should clearly say so. I’m sure I’ve seen it complaining about too much length in the past.

2

Did Brave change some timeout? I keep getting “network issue” on queries on anything that is somewhat long, then I retry like 3 times and eventually (in some cases) it goes through and I get an answer back.

I used Net Exporter and found that I get this reply:

HTTP/1.1 500
date: Mon, 22 Apr 2024 07:04:12 GMT
content-type: application/json
content-length: 69
server: uvicorn
{"error":{"type" :"50001","message":"An error occurred: ReadTimeout"}}

from a POST to this url: https://ai-chat-premium.bsg.brave.com/v2/complete

3

Probably unrelated to the server-side 500 error, but I had Brave crash to desktop twice on me when pressing the Summarize button on this exact video to retry and see if the timeout is fixed.

Can someone please confirm whether they face at least the network issue when summarizing with claude this video?

4

So many network errors lately… Can anyone else confirm? Is it just me? Every big page I try to feed to Leo (claude model) gets the above network error. Sometimes retrying 3-5 times helps, other times it seems a permanent issue (like with the video I linked).

5

I have another case with another video (I swear I’m not that much into UFO videos I just keep noticing it on them because they are lengthy haha) https://www.youtube.com/watch?v=yzi8xcRicGE but it’s when I press “Suggest questions” this time. It fails consistently to show questions, and using the Net Exporter tool I saw that once more it’s the exact same ReadTimeout error behind the scenes. Yet for this video the summarizer works.

6

Claude 3 Sonnet premium.

What is going on the server side? :slight_smile: I used to be able to summarize pages that were about 20KB long, now when summarizing even 16KB it fails. Sometimes it fails with network error (see my other unanswered thread), other times it fails with the usual “this conversation is too long”.

Can we please unleash the full window size capabilities of Sonnet, and increase the network timeout thresholds even if it costs more of our daily anonymized tokens and even if it takes longer to get an answer back?

7

@Tritonio,
Thank you for reaching out to us.
First — I went ahead and merged your two topics to keep things organized if that’s okay.

It looks like the team is already discussing increasing the context window as well as other solutions for this issue. I just pinged the team internally to see where we stand on this and will reply back here when I know more.

Thank you for your feedback.

1 Like
8

Thanks for looking into it. It seems like a misconfiguration somewhere. For example this article https://en.wikipedia.org//wiki/D-Bus can be summarized with Claude 2 Instant but not with Claude 3 Sonnet. Sonnet has a waaaay bigger window than Claude, there’s no reason for it to refuse to summarize it. Though even Claude 2 Instant failed with the usual network error that I’ve documented above, once before actually going through and giving me a summary.

1 Like
9

Yes, it looks like Leo suffers from a very short context window. This article is a relatively short one, but only 59% of it was sent to Leo for summarization: https://fleetwood.dev/posts/you-could-have-designed-SOTA-positional-encoding