# ============================================================ # robots.txt – pusc.or.id # President University Student Council # ============================================================ # --- Content usage signals (authoritative source: HTTP headers in .htaccess) --- # Content-Signal: search=yes, ai-train=no, ai-index=yes # search: allow search indexing; ai-train: no bulk AI training; # ai-index: allow AI search indexing. Keep in sync with llms.txt. # NOTE: Content-Signal is intentionally NOT a bare directive here because # Lighthouse reports it as "Unknown directive" (SEO 1.0 -> 0.92). It is # declared as the X-Content-Signal HTTP header instead (Cloudflare spec). # --- Good search crawlers: allow everything --- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / # --- AI search / on-demand browsing: allow --- # ChatGPT-User and PerplexityBot/User are user-triggered browsing # and search indexing, not bulk training. Allowing them improves # AI visibility without violating the Training: no policy. User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-SearchBot Allow: / # --- AI bulk training: block entirely (Training: no policy) --- User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: YouBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Amazonbot Disallow: / User-agent: Diffbot Disallow: / User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: img2dataset Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Scrapy Disallow: / # --- All other crawlers: block sensitive paths --- User-agent: * Disallow: /pusc-admin-dashboard Disallow: /livewire Disallow: /_ignition Disallow: /error/ # --- Content-Signal is declared above (robots.txt) and mirrored as HTTP # --- headers (see public_html/.htaccess) for crawlers that prefer headers. --- # --- Sitemap --- Sitemap: https://pusc.or.id/sitemap.xml