List of Bots that Spy Your Website Content

Here is a list of search engine bots and SEO spy tool bots that have been hitting my one of my websites the most though I have blocked a few ubint cloudflare firewall. I trying to figure out how to block Chinese bots and SEO tool box. Will update once I have successfully blocked unwanted bots.

The newest bot is Applebot that made 45 hits today.

BOT NamesHITSBandwidthLast Hit
2345Explorer13,76961.48 MB17 Jan 2021 – 16:19
MSIE 6 – ( Rogue Robot )4,73423.94 MB18 Jan 2021 – 02:25
Mediapartners-Google2,015122.57 MB18 Jan 2021 – 02:02
python1,9959.13 MB17 Jan 2021 – 10:35
Unknown robot identified by \*bot1,82521.85 MB17 Jan 2021 – 19:56
wordpress1,46210.29 MB18 Jan 2021 – 03:07
bingbot1,367+3642.82 MB18 Jan 2021 – 03:26
Firefox version 10 and lower – various robots851991.47 KB16 Jan 2021 – 11:55
Googlebot806+711.17 MB18 Jan 2021 – 03:17
Apache-HttpClient734+53.40 MB18 Jan 2021 – 01:26
oBot619+355.12 MB18 Jan 2021 – 02:51
Unknown robot identified by bot\*454+3312.43 MB18 Jan 2021 – 02:48
SemrushBot411+445.24 MB18 Jan 2021 – 02:55
AhrefsBot303+113.64 MB18 Jan 2021 – 03:16
YandexMetrika3024.68 MB18 Jan 2021 – 03:15
infobot237+113.00 MB17 Jan 2021 – 20:52
SeznamBot161+4617.52 MB18 Jan 2021 – 03:09
empty user agent string188+65.09 MB18 Jan 2021 – 02:24
crawl169+31.90 MB18 Jan 2021 – 03:20
BLEXBot148+52.11 MB15 Jan 2021 – 17:47
favicon1231.41 MB18 Jan 2021 – 00:22
YandexBot81+281.20 MB17 Jan 2021 – 22:25
proximic1051.59 MB18 Jan 2021 – 03:06
facebookexternalhit9711.64 MB18 Jan 2021 – 01:51
Baidu ( catchall )821.29 MB16 Jan 2021 – 08:13
DotBot47+27886.02 KB17 Jan 2021 – 20:37
Googlebot-Image665.95 MB17 Jan 2021 – 21:37
Sogou web spider60+2985.88 KB18 Jan 2021 – 01:59
core611.88 MB16 Jan 2021 – 15:35
MJ12bot43+16743.46 KB18 Jan 2021 – 03:05
Applebot48+54.95 MB18 Jan 2021 – 02:41
fetcher53818.69 KB18 Jan 2021 – 03:07
MSIE 5 – ( Rogue Robot )52700.32 KB17 Jan 2021 – 01:33
GrapeshotCrawler46+3614.38 KB18 Jan 2021 – 00:49
mozilla/5.0 – ( Rogue Robot )46553.78 KB17 Jan 2021 – 17:57
spider39+12.07 MB16 Jan 2021 – 09:03
feed374.61 MB17 Jan 2021 – 06:56
CFNetwork3728.61 MB17 Jan 2021 – 16:28
twitterbot30+73.84 MB18 Jan 2021 – 03:23
BingPreview29612.22 KB17 Jan 2021 – 14:49
Dalvik29862.73 KB18 Jan 2021 – 01:11
DuckDuckGo-Favicons-Bot29314.84 KB17 Jan 2021 – 11:17
coccoc14+111006.40 KB17 Jan 2021 – 16:32
blog24214.22 KB15 Jan 2021 – 14:06
CCBot19+3227.83 KB14 Jan 2021 – 20:41
Unknown robot (identified by hit on robots.txt)0+2159.06 KB17 Jan 2021 – 23:06
dataprovider.com15+11.09 MB13 Jan 2021 – 19:23
DomainStatsBot10+2132.00 KB15 Jan 2021 – 18:28
custo113.64 KB14 Jan 2021 – 17:05
yahoocachesystem91.78 MB16 Jan 2021 – 05:35
qwantify911.87 KB18 Jan 2021 – 00:06
Mail.RU Bot5+498.04 KB17 Jan 2021 – 23:00
YandexImages8807.86 KB16 Jan 2021 – 05:30
bloglovin7103.01 KB17 Jan 2021 – 12:03
SafeDNSBot6100.13 KB14 Jan 2021 – 08:29
MojeekBot3+360.67 KB14 Jan 2021 – 16:43
Baiduspider4+125.28 KB13 Jan 2021 – 23:36
360Spider466.83 KB14 Jan 2021 – 00:02
Yandex ( catchall )4186.51 KB14 Jan 2021 – 19:41
testbot466.26 KB15 Jan 2021 – 03:34
Deepnet Explorer341.76 KB15 Jan 2021 – 12:53
MSIE 4 – ( Rogue Robot )227.97 KB12 Jan 2021 – 20:34
Curl2218.83 KB11 Jan 2021 – 20:47
Java ( catchall )213.38 KB16 Jan 2021 – 00:30
link1+119.18 KB17 Jan 2021 – 18:34
Screaming Frog SEO Spider1+124.06 KB12 Jan 2021 – 17:48
Ruby script243.70 KB18 Jan 2021 – 00:35
TelegramBot2225.98 KB14 Jan 2021 – 18:05
LinkCheck112.52 KB16 Jan 2021 – 20:22
PhantomJS1192.18 KB14 Jan 2021 – 23:21
robot1337 Bytes13 Jan 2021 – 00:48
nbot14.57 KB14 Jan 2021 – 04:54
rome client114.75 KB12 Jan 2021 – 18:13
A PHP script14.56 KB15 Jan 2021 – 19:54
cfetch127.62 KB15 Jan 2021 – 06:55
Windows 95 – ( Rogue Robot )114.13 KB12 Jan 2021 – 16:22
Related:   Schema Markup for eCommerce and Local Business Store