Want AI to find your site, but not train on it? Keep those choices separate.

01A simpler signal is coming, not a new ranking shortcut

You want customers to find the work you publish. That does not automatically settle whether you want the same work used for model training.

On 30 October, OpenAI committed to honoring a common robots.txt training-preference signal. It aims to implement it in the coming months and says existing crawler controls will continue. This is an announcement of intended support, not evidence that the new behavior is already operating. [1]

The practical change is a possible reduction in policy-maintenance work across participating providers. It is not a promise of search inclusion, citations or higher rankings.

02Three things that should not share one checkbox

Swipe or scroll to compare

OpenAI's documentation explicitly permits allowing its search crawler while disallowing its training crawler. That describes OpenAI's controls, not a universal contract for every AI application. Private information still belongs behind actual access controls, not in a public page with a crawler instruction. [2]

03What the draft adds

The linked 30 October individual Internet-Draft proposes the policy token __usage_ai-training in a User-Agent group, using ordinary robots.txt path rules. Its status page explicitly says it is not IETF-endorsed. Treat it as work in progress, not an adopted standard or a legal determination about content rights. [3]

Read the draft's Introduction and Independence from Crawler Rules section. They explain why a usage preference is distinct from permission to fetch. You do not need to rewrite the site this afternoon to follow the development.

04Make the next review small

• Write down the desired policy separately for public search and training.
• Record the current robots.txt URL, relevant groups, CDN rules, approval owner and inspection date.
• Identify any mismatch between the intended policy and current configuration. Do not infer access from the file alone.
• Keep a copy of the previous configuration and a rollback owner for any approved change.
• Recheck the provider's implementation announcement before treating the new signal as effective.

If discovery is the problem, first ask whether the public page can be fetched and whether it answers a real question. A policy change will not repair a missing answer, an inaccessible page or an unclear offer.

05What would count as evidence?

After an authorized change, inspect the actual served policy and available access logs. Measure search appearances and visits separately. A successful fetch is not proof of citation; a citation is not proof of a customer enquiry. No crawler-policy experiment or visibility gain was measured for this note.