Discovery tools for Verified Brave Publishers

Well… I will “lose attention” by having results hosted externally but I think the value is still there to do this.

After I finish the geo/language filters, search result caching and pagination code, I will post a script for creators to host results on their sites. :+1:

I am thinking about 2 options: Iframed results (easy implementation/configuration) and API JSON results for those who wish to parse/display in their own way.

1 Like

@Mattches another question… When pulling the api data there are many sites with no ‘channel_id’ set. A user mentioned “theguardian.com” was not listed and that is why it got filtered out.

I updated my scripts and now have 22000 additional sites to check :heart_eyes_cat:

I also am capturing http responses such as 404 not found, 502 not available, etc. and will make available via reporting tools in the near future.

In addition, i will be checking https cert validity and make that available as well.

1 Like

Sounds like your making progress on the search results, awesome!

Yea getting there. Today I am rewriting part of one of my core scripts to greatly reduce indexing time.

1 script to process api file 1x/day
1 script to fetch html of homepages 12x/minute
1 script to parse data into db 12x/minute
1 script to fetch Alexa Ranks 3x/minute (rate limits)

This will index 34,560 sites per 2 days or so.

Things I parse:
Title, meta tags, links, urls, imgs, plaintext
Logging http response codes (404 not found, 200 OK, etc).
Validating HTTPS certs

Any other metrics you want to see added?

1 Like

Oh yeah I looked at the site on my phone the other day and it looked good on their too, just wanted to let you know.

:+1: thanks. I need to add a “overflow:hidden;” style to the search result and may got width:100%; on the .body element. Other than that i am ok with it.

Trying to “finish” braveDB by june 1… Eek

1 Like

@Mattches @cory
I placed some reports at https://bravedb.com

Top Alexa sites
Slow sites
404 not found
Ssl errors

More to come as well as API

I like the list of sites you have going there, that’s an awesome idea. Can see how that’s going to help find good sites!
Hey have you been making progress on youtube searches at all or is that something further down the road?

Due to the fact that DNS can be used for verification not all brave publishers are on sale. @joshy

Site will be back up either Tuesday or Wednesday.

Thanks!

I am focusing on other projects at the moment.

The site is down, I believe due to my shared hosting not liking the bandwith used from spidering cron scripts.

If anyone is interested in joining up and forming a team, please message me. @life2k