Allowing AI Crawlers Without Losing Control: A Playbook for Switzerland

Swiss publishers keep getting the same request: may our system collect your pages? Answering in the crawler's language, allow or deny, is the wrong frame, because the request that arrives today bundles several very different behaviours under one label. The real decision is which door to open, for which visitor, and on what terms.

Why crawler access became an editorial decision

For most of the web's history, the publisher-automated visitor relationship was simple and mostly good. Search engines crawled pages, built an index, and sent readers back with a link. The publisher gained referrals; the crawler gained content. Almost none needed a strategy.

That arrangement no longer describes arriving traffic. A growing share of automated requests are not building an index at all. They want passages to place inside a generated answer, and the answer may or may not carry an attribution back to you. Access policy therefore stops being a server chore and becomes an editorial and commercial decision. Publishing the policy is the easy half; making the pages you permit quotable is the valuable half.

Four kinds of automated visitor

Most teams split automated traffic into human-like and machine-like, which hides what matters. Four categories are more useful, because each wants something different.

Training-oriented crawlers

These fetch at scale to collect text for building models. They are heavy, patient, and uninterested in interactive features. What they want is volume of published prose, and what they return is rarely a referral. The commercial question is whether your archive is material you hand over or an asset you license on purpose.

User-triggered fetchers

These arrive because a person asked a question elsewhere and the system decided your page would help answer it. Volume is low, intent is high. This is the traffic that can produce a visible recommendation inside an answer, which is what makes access decisions interesting for publishers who depend on expertise.

Search index crawlers

The classic visitor, still worth treating well. Degrading them costs ordinary search visibility, which still feeds the discovery habit many readers have before they try an assistant.

Commercial aggregators and monitoring agents

These fetch to republish, compare, resell, or watch how your content performs elsewhere. Their requests are legitimate in a way bulk training crawls are not, and a licensing conversation often beats a block.

A three-tier access framework

A workable policy needs tiers, because one switch forces you to be wrong about everything at once. Apply tiers to sections of the site, not the whole domain, since a publication usually has material deserving different treatment depending on whether it is news, evergreen reference, or archive.

The first tier is open access. It covers pages you want quoted without friction: evergreen explainers, service descriptions, published methodology, anything answering a question a stranger might type. Give these away freely and do not put them behind a request gate.

The second tier is conditional access, where permission depends on purpose. You may welcome retrieval that produces a visible link or attribution and decline wholesale harvesting that produces none, while serving both from the same server. This works best when the conditions sit on a page a human can read, and when your server can tell the two behaviours apart without guessing.

The third tier is closed access. Embargoed material, subscriber-only analysis, commissioned research, and anything tied to a promise of exclusivity belong here. The point is not blocking but that the default is no, and an exception requires a conversation rather than a guess. Keep the written policy short: two clear pages describing the tiers beat forty pages of edge cases, and short documents get read by staff, partners, and agents alike.

Writing directives that outlive a policy change

The mechanical side lives in the crawler directives file at the root of your domain. Its value is not that it binds anyone. It states your preference in the one place automated systems check first, and records what you decided and when.

Ask each provider for the user-agent labels their systems actually send, then group those labels by category instead of listing dozens of near-identical names. Grouping lets you change your mind about a whole family of services with one edit, which matters because the list of senders changes faster than most policies do. Annotate the file with comments explaining what each group is and why, because humans read them, including the engineer who inherits the configuration after you leave.

Add pacing rules for heavy categories so a collection run does not degrade the experience of readers you are trying to keep. Point the file at a human-readable access statement covering the tiers, the reasoning, and a contact route for vendors who want to discuss terms. Treat the file as a published decision with a review date, since a directives file nobody has read in two years tells the outside world something you did not intend.

Reading the logs before writing the rules

The temptation is to write policy first and read the data afterwards. Reverse that. Server logs are the only unbiased evidence of what is happening, and most publishers are surprised by at least one line.

Look at hourly request volume and which paths are hit hardest. A run that repeatedly requests the full archive behaves differently from a fetcher requesting a handful of pages with a fresh timestamp, and the pattern tells you which is which. Note whether the agent follows links or requests URLs it already knows, and whether it asks for rendered pages or raw documents. Also note which response codes you return.

Track the ratio of content requests to asset requests. A visitor fetching scripts, styles, and images is rendering pages, which usually means a live system is trying to see what a person sees. One fetching only documents works on the source. Watch the referrer field, and note agent labels that are missing or generic.

Publishing what you want quoted

Access gets a machine to your page. Structure decides whether the page survives the trip. Lead with a direct answer in the first paragraph, so a passage lifted out of context still makes sense. Keep entity names, product names, and place names spelled consistently, because a summariser working from several pages will otherwise produce a blurred version of you.

Give stable, readable addresses to pages you want referenced. Include the specifics people ask about: prices, dates, conditions, opening hours, named staff, model numbers, coverage limits. A page stating concrete facts can be quoted accurately, while a page built from adjectives can only be paraphrased, which is where positioning gets lost.

State who wrote a piece and when it was reviewed. Attribute sources. Use tables and lists for structured material, and keep the sentence carrying the key fact outside the table so it survives extraction as prose. Add a question-and-answer block for the queries your audience asks most, since those phrasings sit close to real prompts.

A ninety-day rollout

Spend the first fortnight reading logs and labelling traffic by category, changing nothing yet.
Classify your top pages into the three tiers, starting with the twenty addresses that matter most.
Write the tier policy in plain language and have someone outside the team read it.
Translate the policy into grouped directives with comments and pacing rules.
Publish the access statement and tier list, and point the directives at it.
Review at thirty and ninety days, comparing citation behaviour against the baseline recorded in step one.

Where this is heading

Crawler access is not a switch you set once. It is a standing arrangement between what you publish, what automated systems may read, and what you want to be known for. The publishers who handle it well are neither blanket blockers nor blanket openers. They decided, tier by tier, what access serves the reader and the business at once, then wrote it down so it could be revised. Review it on a schedule, and treat every automated request as a reader.

Frequently Asked Questions

Should a Swiss publisher block AI crawlers by default?

The default depends on what your pages are for and where revenue comes from. If your business depends on being quoted as an authority, open access to your best evergreen material serves you better than a blanket block, since a system that can read you is a system that can recommend you. Blocking makes sense for material you intend to license. The mistake is deciding by reflex.

How do I tell apart a training crawler from a user-triggered fetcher?

Watch the request pattern rather than the label in the user-agent string, because that string is self-declared and easy to change. Training crawls are voluminous, repetitive, patient, and spread across the whole archive. User-triggered fetchers arrive in small bursts tied to real questions and request a handful of specific pages. A third signature is rendering behaviour: an agent fetching scripts wants to see what a person sees.

What is the point of publishing crawler directives at all?

Directives are the one channel every automated visitor is expected to check, so they turn a private preference into a stated one that partners, agents, and journalists can see. That visibility matters because it makes your position legible without private negotiation. They are neither a contract nor self-enforcing, so treat them as the documented layer of a policy, and pair them with server-side rules.