# Standard crawlers User-agent: * Allow: / # AI Search & Retrieval Bots (ALLOW — these surface content in AI search results) User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: YouBot Allow: / # AI Training Bots (ALLOW — reversed 2026-08-07) # # These were blocked in April under a publisher model: let them cite us, don't # let them absorb us. That trade only pays if the words themselves are the asset # worth licensing. Theanna's product is the platform, so the marketing copy has # no licensing value and every month blocked is a month of training runs we are # absent from. Being in the weights is what lets a model name us when someone # asks "the best accelerator for female founders" without running a search. # Retrieval (OAI-SearchBot, Claude-SearchBot) was never the blocked path. User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / # Still blocked: scrapers that feed neither a search index we appear in nor a # model anyone asks about startup programs. User-agent: Meta-ExternalAgent Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-ai Disallow: / Sitemap: https://theanna.io/sitemap.xml