Google and Bing allow us to not show our company’s website in their search using a combination of robots.txt and a noindex. How can we stop our website from being shown in Brave search too?
I looked for something like this for Brave search and couldn’t find anything. Please help.
Thanks.
Hello. Can someone from Brave please answer this?
Thanks.
Thank you for your patience @burnermcburnface. noindex meta tag is of course respected by Brave search, if you add it to your pages your company’s will not be indexed or shown.
For robots.txt, we do follow the same restrictions applied to googlebot, otherwise Google benefits from its dominant position.
Additionally, you can request we delist your webpages by contacting privacy@brave.com
Hello @steeven. Is there a way to block just Brave in robots.txt or as a meta tag?
We use the granular control to manage the indexing by the different search engine bots, whether it’s different ones (e.g. different ones from Google like Googlebot for web and Googlebot for news) or from different search engines like Google/Bing/Yandex.
The point about Google benefiting from it position is irrelevant to this question. We’re a large publisher and we want full discretion about which search engine indexes our content.
You cannot. Brave does not use a unique user agent and follows googlebot robots.txt rules
If you want to block brave you must also block googlebot.
Surely that’s Brave’s can’t be official position? Bots identify themselves for many good reasons, and I refuse to believe that Brave would be behave in a shady way like this. It would make Brave no different than your average script kiddie’s scraper.
Since this was on the front page of hackernews i suspect this is why you your question was not ignored.
Brave are free to be open and transparent of course.
I run sites with millions of visits per day and can say there is no brave user agent. they have said in the past they follow googlebot rules. They can claim it is to make their job of crawling easier, but in the end you cannot identify brave scraping and have to block googlebot to block brave.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.