# ============================================================ # ROBOTS SEO (analyse concurrentielle, scraping) # ============================================================ User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: linkdexbot Disallow: / User-agent: MegaIndex Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: serpstatbot Disallow: / User-agent: SEOdiver Disallow: / User-agent: XoviBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Rogerbot Disallow: / User-agent: SEOkicks Disallow: / User-agent: sistrix Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Scrapy Disallow: / User-agent: HTTrack Disallow: / User-agent: WinHTTrack Disallow: / User-agent: WebCopy Disallow: / User-agent: SiteSucker Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailCollector Disallow: / User-agent: WebZIP Disallow: / User-agent: Teleport Disallow: / User-agent: Offline Explorer Disallow: / User-agent: wget Disallow: / User-agent: Wget Disallow: / User-agent: curl Disallow: / User-agent: Curl Disallow: / User-agent: python-requests Disallow: / User-agent: Python-urllib Disallow: / User-agent: urllib Disallow: / User-agent: aiohttp Disallow: / User-agent: httpx Disallow: / User-agent: Go-http-client Disallow: / User-agent: libwww-perl Disallow: / User-agent: LWP::Simple Disallow: / User-agent: Java Disallow: / User-agent: Apache-HttpClient Disallow: / User-agent: okhttp Disallow: / User-agent: HeadlessChrome Disallow: / User-agent: PhantomJS Disallow: / User-agent: Puppeteer Disallow: / User-agent: Playwright Disallow: / User-agent: Selenium Disallow: / User-agent: sqlmap Disallow: / User-agent: nikto Disallow: / User-agent: nmap Disallow: / User-agent: masscan Disallow: / User-agent: zgrab Disallow: / User-agent: zmap Disallow: / User-agent: Nessus Disallow: / User-agent: OpenVAS Disallow: / User-agent: Acunetix Disallow: / User-agent: Netsparker Disallow: / User-agent: WPScan Disallow: / User-agent: wpscan Disallow: / User-agent: DirBuster Disallow: / User-agent: gobuster Disallow: / User-agent: ffuf Disallow: / User-agent: Hydra Disallow: / User-agent: Metasploit Disallow: / User-agent: Nuclei Disallow: / User-agent: Jaeles Disallow: / User-agent: Zao Disallow: / User-agent: BlackWidow Disallow: / User-agent: YandexBot Disallow: / User-agent: YandexImages Disallow: / User-agent: YandexVideo Disallow: / User-agent: YandexMedia Disallow: / User-agent: YandexMetrika Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: SputnikBot Disallow: / User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Trendictionbot Disallow: / User-agent: SeobilityBot Disallow: / User-agent: Rogerbot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Pingdom Disallow: / User-agent: GTmetrix Disallow: / User-agent: Barkrowler Disallow: / # ============================================================ # ROBOTS IA (entraînement de modèles, scraping de contenu) # ============================================================ User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: YouBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou inst spider Disallow: / User-agent: 360Spider Disallow: / User-agent: HaosouSpider Disallow: / User-agent: YoudaoBot Disallow: / User-agent: AlibabaCloudCrawler Disallow: / # ============================================================ # RÈGLES GÉNÉRALES # ============================================================ User-agent: * Disallow: /public/images/ Disallow: /attached-files/ Disallow: /avalanchep/ Disallow: /explore/ Sitemap: https://www.data-avalanche.org/public/sitemap.xml