无效爬虫

  • imagesift bot,

    imagesift bot

    这家爬虫完全不会遵守robots协议，在他们的网站上您可以看到，禁止此爬虫的协议文本为？User-Agent: *Disallow: /User-Agent: Googlebot Allow: / Disallow: /private/Does ImageSiftBot follow Robots.txt rules?Standard directiv...

    imagesift bot

    2024-09-09 14:00
    42
  • Censys,

    Censys

    Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)一个经常被黑客利用的爬虫工具，扫描网站端口等，建议禁止访问

    Censys

    2022-11-05 22:36
    92

  • coccocbot-image/1.0

    User-AgentMozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)传入值 Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com...

    2022-05-02 20:57
    38
  • Python-urllib,

    Python-urllib/3.6

    User-Agent Python-urllib/3.6用于数据采集的一款Python的爬虫

    Python-urllib

    2022-04-23 12:17
    38
  • anasbousselham,

    anasbousselham

    URI address /remote/fgt_lang?lang＝/../../../..//////////dev/cmdb/sslvpn_websessionUser-Agent Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 F...

    anasbousselham

    2022-04-23 12:17
    45
  • Go-http-client,

    Go-http-client/1.1

    User-AgentGo-http-client/1.1此爬虫多用于采集，高频率采集让你网站失去向正常用户提供服务的系统资源，比较厌恶

    Go-http-client

    2022-04-23 12:12
    287
  • CensysInspect,

    CensysInspect/1.1

    User-AgentMozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)Filtering rules \.(bak|inc|old|mdb|sql|php~|swp|java|class)$ >> 1:/backup...

    CensysInspect

    2022-04-23 12:10
    84
  • 垃圾爬虫,

    DataForSeoBot/1.0

    User-AgentMozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)传入值 Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dat...

    垃圾爬虫

    2022-02-22 12:43
    249