I have the same issue when loading a document that is colored and roughly 700 pages. My brave version is: v1.75.181, and my OS version is Version 10.0.26100 Build 26100 for Windows 11 pro. I do not seem to be having the same issue when viewing my pdf from firefox or edge.
EDIT: It seems to get less laggy if I turned off annotation, but the problem seems to be fixed after switching to adobe.