#User-agent: * #Disallow: / User-agent: Googlebot Crawl-delay: 5 User-agent: Twitterbot Disallow: #User-Agent: SemrushBot #Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: PetalBot Disallow: / User-agent: MSNbot Crawl-delay: 5 User-agent: ArchitextSpider Crawl-delay: 5 User-agent: Slurp.so/1.0 Crawl-delay: 5 User-agent: Slurp/2.0j Crawl-delay: 5 User-agent: Slurp/2.0-Kite-Hourly Crawl-delay: 5 User-agent: Slurp/2.0-OwlWeekly Crawl-delay: 5 User-agent: Slurp/3.0-AU Crawl-delay: 5 User-agent: Teoma #ASK.com Crawl-delay: 5 User-agent: Scooter Crawl-delay: 5 User-agent: Lycos_Spider_(T-Rex) Crawl-delay: 5 User-agent: InfoSeek Sidewinder/9.0 Crawl-delay: 5 User-agent: * #Disallow: / Disallow: /newsletter/ # disallow this directory Crawl-delay: 5 ## AI Bots #OpenAI bots User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / #Google AI bots - Bard, Gemini and VertexAI User-agent: Google-Extended Disallow: / #commoncrawl AI User-agent: CCBot Disallow: / #Perplexity AI User-agent: PerplexityBot Disallow: / ##End AI Bots