# ================================ # LAW BAMBA 2025 - Robots.txt # ================================ User-agent: * # 2026-11-04: review form POST target - Bingbot was GETting it ~800x/day (302) Disallow: /review/submit/ Disallow: /profile-update/submit/ Disallow: /case/submit Disallow: /lawyer-widget/ Disallow: /lawyer-badges/ Disallow: /api/review-widget/ Disallow: /login Disallow: /register Disallow: /auth/ Disallow: /mylawbamba/favorites/add/ Disallow: /q/ask Disallow: /*?sort= Disallow: /*?page= Disallow: /*?filter= Disallow: /search Disallow: /admin Disallow: /cgi-bin Disallow: /index.php Disallow: /*.php$ Disallow: /*.php? Disallow: /zip/*/issue/*? Disallow: /lawyer/*? Disallow: /lawyer/*/*/*/* Allow: /zip/ Allow: /lawyer/ Allow: /state/ Allow: / Crawl-delay: 1 # 2026-10-24: every named group below now repeats /login + /register too. # A named group replaces `*` for that crawler, so Bingbot (~1,350/day), # PerplexityBot, Applebot etc. were allowed into them. Both pages are also # . # --- Google Bots --- # A named group REPLACES the `*` group for that crawler (it doesn't add to # it), so every `*` rule Googlebot should also obey has to be repeated here. # 2026-10-04: /login, /register (+ ?next= variants and /redirect/... 302s) # were missing and ate ~1,160 Googlebot requests/day (~15% of its crawl); # /badge/*.svg embed images ate ~800/day more. User-agent: Googlebot Allow: / Disallow: /review/submit/ Disallow: /profile-update/submit/ Disallow: /case/submit Disallow: /lawyer-widget/ Disallow: /lawyer-badges/ Disallow: /api/review-widget/ Disallow: /login Disallow: /register Disallow: /search Disallow: /admin Disallow: /badge/ Disallow: /auth/ Disallow: /*?sort= Disallow: /*?page= Disallow: /*?filter= Disallow: /*.php$ User-agent: Googlebot-Image Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Googlebot-Video Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Googlebot-News Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: AdsBot-Google Allow: / Disallow: /auth/ Disallow: /practice/ Disallow: /near-me/ Disallow: /login Disallow: /register User-agent: AdsBot-Google-Mobile Allow: / Disallow: /auth/ Disallow: /practice/ Disallow: /near-me/ Disallow: /login Disallow: /register User-agent: Google-Adwords Allow: / Disallow: /auth/ Disallow: /practice/ Disallow: /near-me/ Disallow: /login Disallow: /register User-agent: Mediapartners-Google Allow: / Disallow: /auth/ Disallow: /practice/ Disallow: /near-me/ Disallow: /login Disallow: /register # Google AI (Gemini, Bard, Vertex) User-agent: Google-Extended Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # --- Bing Bots --- # 2026-11-12: a named group replaces `*`, so the Nov 4 /review/submit/ rule never # reached Bingbot (still ~150 302s/day, plus ~40/day on the profile-update form # target and ~600/day on badge/widget embeds). Repeated here. User-agent: Bingbot Disallow: /index.php Disallow: /*.php$ Disallow: /auth/ Disallow: /login Disallow: /register Disallow: /review/submit/ Disallow: /profile-update/submit/ Disallow: /case/submit Disallow: /lawyer-widget/ Disallow: /lawyer-badges/ Disallow: /api/review-widget/ Disallow: /badge/ Disallow: /mylawbamba/favorites/add/ Disallow: /q/ask Disallow: /search Disallow: /admin Disallow: /*?sort= Disallow: /*?page= Disallow: /*?filter= User-agent: AdIdxBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: BingPreview Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # --- AI & Research Bots --- User-agent: GPTBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register Disallow: /mylawbamba/ Disallow: /search Disallow: /admin Crawl-delay: 2 User-agent: ChatGPT-User Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: ClaudeBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register Disallow: /mylawbamba/ Disallow: /search Disallow: /admin Crawl-delay: 2 User-agent: Claude-Web Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Anthropic-Web Disallow: / User-agent: PerplexityBot Allow: / Crawl-delay: 2 Disallow: /auth/ Disallow: /login Disallow: /register User-agent: PerplexityBot-2 Allow: / Crawl-delay: 2 Disallow: /auth/ Disallow: /login Disallow: /register User-agent: OAI-SearchBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Grok Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: xAI Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # --- Other Major Crawlers --- User-agent: Applebot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # Apple Intelligence User-agent: Applebot-Extended Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # 2026-11-04: Meta AI search indexer. Surged to ~120 req/min (13:00-15:35 UTC, # 125 IPs in 57.141.x), mostly deep /state listings. Throttled, not blocked, so # profiles can still be cited in Meta AI answers. It also fetched /login and # /auth/ despite the * group, so these rules are repeated (and enforced in Apache). User-agent: meta-webindexer Crawl-delay: 10 Disallow: /state/*/zip/ Disallow: /login Disallow: /register Disallow: /auth/ Disallow: /lawyer-widget/ Disallow: /lawyer-badges/ Disallow: /api/ Disallow: /mylawbamba/ Disallow: /review/submit/ Disallow: /q/ask Disallow: /search Disallow: /*? Allow: /lawyer/ Allow: /state/ Allow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: FacebookBot Disallow: / # ByteDance (TikTok, CapCut) User-agent: Bytespider Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Diffbot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # Common Crawl User-agent: CCBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: CCBot-Extended Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: magpie-crawler Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Seekr Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: TurnitinBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Amazonbot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register Disallow: /mylawbamba/ Disallow: /search Disallow: /admin Crawl-delay: 2 User-agent: DuckDuckBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: YandexBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Baiduspider Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Exabot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: Dotbot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: PetalBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register User-agent: ZoominfoBot Allow: / Disallow: /auth/ Disallow: /login Disallow: /register # --- SEO Crawlers (Disallowed) --- User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SERankingBacklinksBot Disallow: / # === SITEMAPS === Sitemap: https://www.lawbamba.com/sitemaps/sml-index.xml