# As a condition of accessing this website, you agree to abide by the following
# content signals:

# (a)  If a Content-Signal = yes, you may collect content for the corresponding
#      use.
# (b)  If a Content-Signal = no, you may not collect content for the
#      corresponding use.
# (c)  If the website operator does not include a Content-Signal for a
#      corresponding use, the website operator neither grants nor restricts
#      permission via Content-Signal with respect to the corresponding use.

# The content signals and their meanings are:

# search:   building a search index and providing search results (e.g., returning
#           hyperlinks and short excerpts from your website's contents). Search does not
#           include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
#           augmented generation, grounding, or other real-time taking of content for
#           generative AI search answers).
# ai-train: training or fine-tuning AI models.
# use:      how AI systems may consume the content (immediate, reference, or full).

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

# https://www.robotstxt.org/robotstxt.html
#
# THIS IS THE ONLY robots.txt THIS ORIGIN HAS, which is why the shop's routes are
# named here rather than in apps/shop. A crawler reads robots.txt from the origin
# ROOT and nowhere else, and `simonphumin.com/*` routes to the portfolio worker
# (the shop's route is `simonphumin.com/shop*`, more specific but not the root).
# So this file answers for both workers, and a rule added under apps/shop/static
# would be served at /shop/robots.txt, which nothing fetches.
#
# NOT A SECURITY CONTROL, and nothing here is load-bearing. robots.txt is a
# request; every route named below is enforced server-side — `/shop/admin` by
# `isAdmin`, `/shop/downloads` by the download session, the Stripe webhook by a
# signature, and the whole shop by the host gate in hooks.server.ts. What this
# buys is keeping a well-behaved crawler off endpoints it can only fail at, and
# keeping a login form out of a search index.

User-agent: *
# Machine endpoints, both workers. Nothing under either path renders: the
# portfolio's are the WebGL texture proxy and the CSP report sink, the shop's are
# the same sink plus the Stripe webhook.
Disallow: /api/
Disallow: /shop/api/
# Gated pages. Their only crawlable form is a login screen, and a soft-404 or a
# sign-in form in an index is worse than no result at all.
Disallow: /shop/admin
Disallow: /shop/downloads
Disallow: /shop/download/
# Thin, stateful pages that are crawlable and indexable today and should be
# neither: an empty cart and a checkout form say nothing about this site, and
# both are useless as a search result.
Disallow: /shop/cart
Disallow: /shop/checkout

# THE STATUTORY PAGES ARE NO LONGER DISALLOWED, and removing them is a fix
# rather than a relaxation.
#
# A page blocked in robots.txt is never FETCHED, so the `noindex` inside it is
# never READ — and Google will still list a blocked URL it has found a link to,
# as a bare result with no title or snippet. Blocking was therefore the weaker
# control and was working against the stronger one. `/imprint`,
# `/privacy-policy`, `/return-policy`, `/terms-of-service` and
# `/privacy-settings` each carry
# `noindex, nofollow, noarchive, nosnippet, noimageindex` in their own
# `pageMetaTags`, which does the whole job — but only if a crawler is allowed to
# fetch the page and see it. They are also absent from sitemap.xml and llms.txt,
# so nothing here invites a crawl either.
#
# `/licensing` is NOT one of them. It states the terms every image on this site
# points its `license` property at, so it is indexable, sitemapped, and listed in
# llms.txt — the one small-print page that is meant to be found.

# LLM and AI Bot Optimization
#
# TWO GROUPS, NOT ONE, and the split is what makes /licensing honest. That page
# reserves these photographs against machine learning and generative AI training
# under Art. 4(3) of Directive (EU) 2019/790 and § 44b(3) UrhG. A reservation
# stated on a page while this file hands the training crawlers `Allow: /` is two
# public documents contradicting each other, and the weaker one is the one a
# crawler actually reads.
#
# The line is drawn between TRAINING and ANSWERING, not between "AI" and "not
# AI". Blocking the training crawlers costs nothing anyone wants. Blocking the
# answer and search crawlers would undo the whole point of llms.txt and the
# JSON-LD graph, which exist to be read by exactly those agents.
#
# A crawler-specific group REPLACES the `*` group rather than adding to it, so
# each group below repeats what it needs. Grouped under shared User-agent lines
# because the rules are identical and a dozen copies is how a file like this
# drifts.
#
# THE NAMES ARE CURRENT AS OF 2026-08. Three that were here before are gone:
# `anthropic-ai` and `Claude-Web` are both retired — Anthropic's crawlers are now
# `ClaudeBot` (training), `Claude-User` (fetches a page because someone asked
# about it) and `Claude-SearchBot` (search indexing) — and `Omgilibot` belonged
# to a service that no longer exists. A retired user-agent string in this file is
# not harmless: it reads as coverage, and the agent that replaced it silently
# falls through to the `*` group instead.

# TRAINING CORPUS CRAWLERS — disallowed, matching the reservation on /licensing.
#
# `Google-Extended` governs Gemini and Vertex training only. It is NOT Googlebot:
# blocking it does not affect Google Search crawling, ranking, or the Licensable
# image treatment this site's structured data is built for. It also does not opt
# the site out of AI Overviews in Search, which are served from Googlebot's index
# and have no separate control — that is a known limit, not an oversight here.
#
# `Applebot-Extended` is likewise the training opt-out only; `Applebot` itself,
# which feeds Siri and Spotlight search, is deliberately not named and falls
# through to the `*` group.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: cohere-ai
User-agent: Bytespider
User-agent: CCBot
Disallow: /

# ANSWER AND SEARCH CRAWLERS — allowed, with the same machine-endpoint and
# gated-page exclusions as everyone else.
#
# These index the site or fetch a page because a person asked a question about
# it. They are how this site appears in AI answers at all, which is what the
# JSON-LD graph and /llms.txt were built for, and none of them is a training
# corpus crawler.
#
# `Amazonbot` is the ambiguous one — Amazon documents it as serving Alexa
# answers rather than assembling a training set — so it sits here. Re-check this
# classification with the rest of the names above; vendors move agents between
# these two roles without renaming them.
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Amazonbot
Allow: /
Disallow: /api/
Disallow: /shop/api/
Disallow: /shop/admin
Disallow: /shop/downloads
Disallow: /shop/download/
Disallow: /shop/cart
Disallow: /shop/checkout

# There is deliberately no directive pointing at /llms.txt. `Llms:` is not a real
# robots.txt directive and no standard defines one, so inventing it would put a
# line every real crawler parses into a file, to be ignored by all of them. The
# file is served at the origin root, which is where the convention says to look.

Sitemap: https://simonphumin.com/sitemap.xml