Here is a list of search engine bots and SEO spy tool bots that have been hitting my one of my websites the most though I have blocked a few ubint cloudflare firewall. I trying to figure out how to block Chinese bots and SEO tool box. Will update once I have successfully blocked unwanted bots.
The newest bot is Applebot that made 45 hits today.
| BOT Names | HITS | Bandwidth | Last Hit |
| 2345Explorer | 13,769 | 61.48 MB | 17 Jan 2021 – 16:19 |
| MSIE 6 – ( Rogue Robot ) | 4,734 | 23.94 MB | 18 Jan 2021 – 02:25 |
| Mediapartners-Google | 2,015 | 122.57 MB | 18 Jan 2021 – 02:02 |
| python | 1,995 | 9.13 MB | 17 Jan 2021 – 10:35 |
| Unknown robot identified by \*bot | 1,825 | 21.85 MB | 17 Jan 2021 – 19:56 |
| wordpress | 1,462 | 10.29 MB | 18 Jan 2021 – 03:07 |
| bingbot | 1,367+36 | 42.82 MB | 18 Jan 2021 – 03:26 |
| Firefox version 10 and lower – various robots | 851 | 991.47 KB | 16 Jan 2021 – 11:55 |
| Googlebot | 806+7 | 11.17 MB | 18 Jan 2021 – 03:17 |
| Apache-HttpClient | 734+5 | 3.40 MB | 18 Jan 2021 – 01:26 |
| oBot | 619+3 | 55.12 MB | 18 Jan 2021 – 02:51 |
| Unknown robot identified by bot\* | 454+33 | 12.43 MB | 18 Jan 2021 – 02:48 |
| SemrushBot | 411+44 | 5.24 MB | 18 Jan 2021 – 02:55 |
| AhrefsBot | 303+11 | 3.64 MB | 18 Jan 2021 – 03:16 |
| YandexMetrika | 302 | 4.68 MB | 18 Jan 2021 – 03:15 |
| infobot | 237+11 | 3.00 MB | 17 Jan 2021 – 20:52 |
| SeznamBot | 161+46 | 17.52 MB | 18 Jan 2021 – 03:09 |
| empty user agent string | 188+6 | 5.09 MB | 18 Jan 2021 – 02:24 |
| crawl | 169+3 | 1.90 MB | 18 Jan 2021 – 03:20 |
| BLEXBot | 148+5 | 2.11 MB | 15 Jan 2021 – 17:47 |
| favicon | 123 | 1.41 MB | 18 Jan 2021 – 00:22 |
| YandexBot | 81+28 | 1.20 MB | 17 Jan 2021 – 22:25 |
| proximic | 105 | 1.59 MB | 18 Jan 2021 – 03:06 |
| facebookexternalhit | 97 | 11.64 MB | 18 Jan 2021 – 01:51 |
| Baidu ( catchall ) | 82 | 1.29 MB | 16 Jan 2021 – 08:13 |
| DotBot | 47+27 | 886.02 KB | 17 Jan 2021 – 20:37 |
| Googlebot-Image | 66 | 5.95 MB | 17 Jan 2021 – 21:37 |
| Sogou web spider | 60+2 | 985.88 KB | 18 Jan 2021 – 01:59 |
| core | 61 | 1.88 MB | 16 Jan 2021 – 15:35 |
| MJ12bot | 43+16 | 743.46 KB | 18 Jan 2021 – 03:05 |
| Applebot | 48+5 | 4.95 MB | 18 Jan 2021 – 02:41 |
| fetcher | 53 | 818.69 KB | 18 Jan 2021 – 03:07 |
| MSIE 5 – ( Rogue Robot ) | 52 | 700.32 KB | 17 Jan 2021 – 01:33 |
| GrapeshotCrawler | 46+3 | 614.38 KB | 18 Jan 2021 – 00:49 |
| mozilla/5.0 – ( Rogue Robot ) | 46 | 553.78 KB | 17 Jan 2021 – 17:57 |
| spider | 39+1 | 2.07 MB | 16 Jan 2021 – 09:03 |
| feed | 37 | 4.61 MB | 17 Jan 2021 – 06:56 |
| CFNetwork | 37 | 28.61 MB | 17 Jan 2021 – 16:28 |
| twitterbot | 30+7 | 3.84 MB | 18 Jan 2021 – 03:23 |
| BingPreview | 29 | 612.22 KB | 17 Jan 2021 – 14:49 |
| Dalvik | 29 | 862.73 KB | 18 Jan 2021 – 01:11 |
| DuckDuckGo-Favicons-Bot | 29 | 314.84 KB | 17 Jan 2021 – 11:17 |
| coccoc | 14+11 | 1006.40 KB | 17 Jan 2021 – 16:32 |
| blog | 24 | 214.22 KB | 15 Jan 2021 – 14:06 |
| CCBot | 19+3 | 227.83 KB | 14 Jan 2021 – 20:41 |
| Unknown robot (identified by hit on robots.txt) | 0+21 | 59.06 KB | 17 Jan 2021 – 23:06 |
| dataprovider.com | 15+1 | 1.09 MB | 13 Jan 2021 – 19:23 |
| DomainStatsBot | 10+2 | 132.00 KB | 15 Jan 2021 – 18:28 |
| custo | 11 | 3.64 KB | 14 Jan 2021 – 17:05 |
| yahoocachesystem | 9 | 1.78 MB | 16 Jan 2021 – 05:35 |
| qwantify | 9 | 11.87 KB | 18 Jan 2021 – 00:06 |
| Mail.RU Bot | 5+4 | 98.04 KB | 17 Jan 2021 – 23:00 |
| YandexImages | 8 | 807.86 KB | 16 Jan 2021 – 05:30 |
| bloglovin | 7 | 103.01 KB | 17 Jan 2021 – 12:03 |
| SafeDNSBot | 6 | 100.13 KB | 14 Jan 2021 – 08:29 |
| MojeekBot | 3+3 | 60.67 KB | 14 Jan 2021 – 16:43 |
| Baiduspider | 4+1 | 25.28 KB | 13 Jan 2021 – 23:36 |
| 360Spider | 4 | 66.83 KB | 14 Jan 2021 – 00:02 |
| Yandex ( catchall ) | 4 | 186.51 KB | 14 Jan 2021 – 19:41 |
| testbot | 4 | 66.26 KB | 15 Jan 2021 – 03:34 |
| Deepnet Explorer | 3 | 41.76 KB | 15 Jan 2021 – 12:53 |
| MSIE 4 – ( Rogue Robot ) | 2 | 27.97 KB | 12 Jan 2021 – 20:34 |
| Curl | 2 | 218.83 KB | 11 Jan 2021 – 20:47 |
| Java ( catchall ) | 2 | 13.38 KB | 16 Jan 2021 – 00:30 |
| link | 1+1 | 19.18 KB | 17 Jan 2021 – 18:34 |
| Screaming Frog SEO Spider | 1+1 | 24.06 KB | 12 Jan 2021 – 17:48 |
| Ruby script | 2 | 43.70 KB | 18 Jan 2021 – 00:35 |
| TelegramBot | 2 | 225.98 KB | 14 Jan 2021 – 18:05 |
| LinkCheck | 1 | 12.52 KB | 16 Jan 2021 – 20:22 |
| PhantomJS | 1 | 192.18 KB | 14 Jan 2021 – 23:21 |
| robot | 1 | 337 Bytes | 13 Jan 2021 – 00:48 |
| nbot | 1 | 4.57 KB | 14 Jan 2021 – 04:54 |
| rome client | 1 | 14.75 KB | 12 Jan 2021 – 18:13 |
| A PHP script | 1 | 4.56 KB | 15 Jan 2021 – 19:54 |
| cfetch | 1 | 27.62 KB | 15 Jan 2021 – 06:55 |
| Windows 95 – ( Rogue Robot ) | 1 | 14.13 KB | 12 Jan 2021 – 16:22 |
