Is the WDP fetcher sensitive to metered connections?

tjn · July 17, 2023, 1:23am

I know the Web Discovery Project component isn’t the most recent addition, but hopefully it’s recent enough.

I’m wondering whether the WDP service, which (if enabled) periodically fetches web pages in the background for the Brave Search crawler, refrains from fetching when the OS reports that the internet connection is metered. If it doesn’t, it would be nice if it did.

Connections can be detected and/or marked as metered in the most recent versions of Windows. macOS supports putting connections in “low data mode” as of Ventura. The Linux situation is fragmented as usual, but the default network manager in Ubuntu (creatively named NetworkManager) is also metering-aware.

If you add a second toggle switch below the existing one for WDP for “Include metered connections”, it could be turned off by default, but allow people to turn it on. Then on systems on which Brave can’t obtain metering info, it could be turned on and unable to be turned off, with a brief message below explaining that metering can’t be determined on that system.

Note: If WDP is not already metering aware, please move this post as appropriate for a feature request.

anon57438784 · July 17, 2023, 3:23am

Metered connections don’t matter, why would they? why should the script be aware of them?

I think you should read what the WDP does, then enable it if you want or not, it is pretty easy to even check what it does inside the console, with the commands, because it is a script after all, running in the background.

WDP what it does is every page you visit will be added in a queue in memory, then the script will check the clicks you made, the time you spent in the website and stuff like that, and then it only sends the information that will be useful for Brave Search to Brave servers.

It is just Text, in a JSON style, which is like few bytes of information on each send, if you are browsing the web, with any type of connection, then, any image or script or CSS will spend 1000x more bandwidth than the information being sent by the WDP script.

What would be the point of be metered aware if all that matters is if a page is public or not? You either want to send the information or not, the only thing WDP can do is use more system resources but not even that can be measured because of how tiny the script is working the background.

There doesn’t need to be more BS added to the already bunch of online BS people are talking about Brave because of the WDP and because people apparently don’t want to research or don’t really care to find the truth when talking about Brave and WDP and indexing websites.

github.com

brave/web-discovery-project/blob/main/modules/web-discovery-project/sources/README.md

# Web Discovery Project Overview

*Web Discovery Project* is a methodology and system developed by Brave but heavily inspired by Cliqz's [Human Web](https://0x65.dev/blog/2019-12-03/human-web-collecting-data-in-a-socially-responsible-manner.html), we recommend to check the blog post as additional material even though there are, and will be, significant departures as *WDP* evolves. 

## Motivation

*Web Discovery Project* is a methodology and system developed by Brave to collect data generated by their users while protecting their privacy and anonymity.

Brave needs data to power its privacy search. This data, provided by Brave users, is collected in a very different way than typical data collection. We want to depart from the current standard model, where users must trust that the company collecting the data will not misuse it, ever, in any circumstance. We do not want users to have no other choice but to trust us. There are many ways a trust model can fail. Hackers can steal data. Governments can issue subpoenas, or get direct access to the data. Unethical employees can dig into the data for personal interests. Companies can go bankrupt and the data auctioned to the highest bidder. Finally, companies can unilaterally decide to change their privacy policies.

In the trust model, which is the industry standard, the user has very little control and protection comes only from privacy policy and enforcing bodies. We believe we must do better, if only for selfish reasons because we use our own products, and consequently, our own data is collected. We are not comfortable with only a promise based on Terms of Service and Privacy Policy agreements. It is not enough for us, and should not be enough for our users either. As someone once said, if you do not like reality, feel free to change it. The Web Discovery Project is our proposal for a more responsible and less invasive data collection.

## Fundamentals

The fundamental idea of the *Web Discovery Project* data collection is simple: **to actively prevent [Record Linkage](https://en.wikipedia.org/wiki/Record_linkage)**.

Record linkage is the ability to know that multiple data elements, e.g. messages or records, come from the same user. This linkage leads to sessions, and these sessions are very dangerous with regards to privacy. For instance, [Google Analytics data can be used to build sessions that can sometimes be de-anonymized by anyone that has access to them](http://josepmpujol.net/public/papers/big_green_tracker.pdf). Was it intentional? Most likely not. Will Google Analytics try to de-anonymize the data? I bet not. But still, the session is there, stored somewhere, and trust that it is not going to be misused is the only protection we have.

The *Web Discovery Project* is a methodology and system designed to collect data which cannot be turned into sessions once they reached Brave. How? By strictly forbidding any user-identifier that could be used to link records as belonging to the same person, considering not only explicit UIDs but also implicit ones. Consequently, aggregation of user's data in the server-side (on Brave premises) is not technically feasible, as we have no means to know who is the original owner of the data.

This file has been truncated. show original

that’s the best place you can read what it really does and how it works and what data is sent.

Then if you want to check everything yourself, go to brave://inspect/#extensions and click inspect in Brave extension.
Then you can check all the info with WDP for example WDP.isRunning and it will say true if you enabled it.

Then every example in the https://github.com/brave/web-discovery-project/tree/main like WDP.app.modules['web-discovery-project'].background.webDiscoveryProject.state['v'] all you need to do is to remove the app. in WDP.app.modules and you can use the examples github give you to understand what the script does, and it is not like it will connect and send data to the Brave server all the time only because you opened a website.

So for example, if you watch two youtube videos they will appear there, and when you go to the third one, the URL will disappear from the .webDiscoveryProject.state['v']

Anyway It doesn’t work in Private windows because of privacy reasons and that’s it.

Again, is there a reason why this should be metered aware? to save few bytes? then disable it or use InPrivate windows to avoid connection to Brave servers, even if it is almost nothing in the big picture.