How can a website I've visited once keep appearing among my Top sites?

I like to use Top Sites on the desktop, but am having issues with them. I regularly visit some sites multiple times daily that don’t ever get added to the Top sites, but I visited a site ONCE and it keeps appearing among my so-called Top sites. I can remove it, but it re-appears almost straight away. Why? How can I stop it appearing all the time?