I don’t think it’s possible or at least any more sorry, I’m not seeing anyway to change that from looking myself 
It seems it used to be possible via changing:
brave://flags/#download-bubble
brave://flags/#download-bubble-v2
but not anymore that option unfortunatly is no longer there 
Remember though, you can always use KEYBOARD SHORTCUTS to get to your downloads quick with:
Windows: Ctrl+J
Mac: Option+Command+L
Or just PIN a TAB for: brave://downloads/
Sorry this doesn’t give you what you want but hopefully it might have helped