# robots.txt — cloz.uz
#
# All crawlers, including AI search + answer engines (ChatGPT / OAI-SearchBot,
# Claude / ClaudeBot, Perplexity, Gemini / Google-Extended, Grok, Copilot),
# are intentionally allowed to read public content so Cloz appears in both
# classic search and AI answers. Private/authenticated and API routes are
# disallowed.
#
# The Content-Signal line below states that same position in the machine-
# readable form (contentsignals.org):
#
#   search=yes     index us and link to us — this is our distribution.
#   ai-input=yes   quote us in AI-generated answers — also our distribution,
#                  and the reason we serve `Accept: text/markdown`. Setting
#                  this to `no` would opt us out of the channel we are
#                  actively building for.
#   ai-train=no    do not train models on this corpus. The postings are
#                  aggregated public Telegram content, but the normalization,
#                  classification, translation and salary parsing on top of
#                  them are ours, and vacancies go stale within weeks — a
#                  model trained on them learns jobs that no longer exist.
#
# Content-Signal is a declaration of preference, NOT enforcement: it changes
# nothing about what a crawler can fetch. Actually blocking training crawlers
# means giving them their own group, which we have deliberately NOT done:
#     User-agent: GPTBot
#     User-agent: ClaudeBot
#     User-agent: CCBot
#     Disallow: /
# while keeping OAI-SearchBot / Claude-SearchBot / PerplexityBot allowed below.

User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /*/account/
Disallow: /employer/app/
Disallow: /*/employer/app/

Sitemap: https://cloz.uz/sitemap.xml