If you go to brave://settings/appearance you can disable Show autocomplete in address bar, this won’t remove one item, it will stop everything for showing up.
But you can also test out Show top sites in autocomplete suggestions to see if that can remove the site you don’t want showing up.