Key findings
- Of the 48 national news sites measured, 10 (20.8%) block OpenAI's crawlers in robots.txt. Using the same definition, the Reuters Institute found 48% across ten countries at the end of 2023.
- 7 of the 10 blocking sites use an identical AI section in robots.txt: the same 7 bots in the same order. Common Crawl (CCBot) and Google-Extended are not on that list.
- Google-Extended is blocked on only 2 sites and CCBot, a major source of LLM training data, on 3.
- 8 of the blocking sites also shut out the search and user agents of ChatGPT and Perplexity, not only training crawlers.
- 15 sites (31.3%) publish an llms.txt file. Among the world's top 1,000 domains the figure is 9.3%.
- None of the 11 digital-only outlets block any AI bot, and 5 of them publish llms.txt.
- There is no overlap between sites that block AI bots and sites that publish llms.txt. The sector has split into two camps, while 23 sites have taken no position yet.
Why we ran this study
AI companies crawl the web to train models and to cite sources in AI search answers. For news publishers the question is simple: should my content go into these systems? The most visible trace of that decision is robots.txt, the file at the root of every site that tells crawlers where they may go.
In February 2024 the Reuters Institute examined these files on the most-read news sites in ten countries and found that 48% blocked OpenAI's crawlers at the end of 2023. Turkey was not part of that study, and we could not find a current, reproducible measurement for Turkish news sites. This report fills that gap.
Finding 1: Turkish news sites are largely open to AI crawlers
10 of the 48 measured sites block OpenAI's GPTBot or ChatGPT-User (20.8%). We used the Reuters Institute definition exactly: a site "blocks OpenAI" if it blocks GPTBot, ChatGPT-User or all crawlers. Our samples and dates differ, so this is indicative rather than a like-for-like comparison, but the gap is clear.
Blocking also varies by publisher type: 5 of 20 newspapers and 4 of 13 TV channels block OpenAI, while none of the 11 digital-only outlets do.
Finding 2: Copy-paste robots.txt
7 of the 10 blocking sites (Hürriyet, Milliyet, Posta, Fanatik, CNN Türk, NTV, NTV Spor) share an identical AI section: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot and cohere-ai. These sites belong to two different media groups, which suggests a shared template or infrastructure provider.
What is missing from that list matters more:
- CCBot (Common Crawl): Common Crawl is an open archive of a large part of the web and a common source of LLM training data. A site that blocks OpenAI but leaves CCBot open can still end up in training data through that archive.
- Google-Extended: a separate token that controls whether content is used for Google's AI products such as Gemini. It does not affect Google Search ranking. Leaving it out means the content stays available to Google's AI products.
- Claude-SearchBot: Anthropic's newer search crawler is not blocked by name on any measured site. Robots.txt lists freeze on the day they are written.
Finding 3: Training and search crawlers are treated the same
AI crawlers do two different jobs. Some (GPTBot, ClaudeBot, CCBot) collect training data. Others (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch a page when a user asks a question and link to it as a source. Blocking the second group also reduces the chance of being cited, and of receiving traffic, from those answers.
8 of the 10 blocking sites also block at least one search or user agent. Only 2 sites (Cumhuriyet and İHA) have a selective policy aimed at training crawlers alone.
Finding 4: llms.txt is more common than expected
llms.txt is a text file, proposed in September 2024, that summarises a site for language models. It has not been adopted by a standards body, and major AI companies do not clearly state how they use it. Even so, 15 of the measured sites (31.3%) publish one: Sabah, Sözcü, Takvim, Akşam, Yeni Akit, Karar, Fotomaç, A Haber, TGRT Haber, A Spor, En Son Haber, Haberler.com, Son Dakika, Mynet, Diken.
We opened every file. All contain real content: most are short hand-written site summaries, and at least one is a large list generated automatically by a WordPress SEO plugin.
Finding 5: Two camps, no overlap
Not a single site appears both among the 10 that block AI bots and the 15 that publish llms.txt. One group of publishers is closing the door, the other is leaving a guide on how to read their content. The remaining 23 sites neither mention AI bots in robots.txt nor publish llms.txt.
What it means for publishers
- Treat training and search crawlers separately. If the goal is to limit training use, blocking search agents may also cost you source links in AI answers.
- Do not forget CCBot and Google-Extended. A blocking policy without them is incomplete.
- Review the list at least once a year. New crawlers appear and names change.
- robots.txt is a request, not a lock. Crawlers that ignore it need to be handled at the server or CDN level.
Methodology
Sample: 50 national Turkish news sites: 20 newspapers, 13 TV channels, 4 news agencies and 13 digital-only outlets, all listed in the table below. This is a declared list of well-known national publishers, not a traffic ranking.
Date: data collected on 10 November 2026.
Measurement: for each site we requested the homepage, /robots.txt and /llms.txt once, sequentially and with pauses, using a clearly identified user agent (HititMedyaResearch/1.0). robots.txt files were split into user-agent groups. If a group named a bot, its rules applied; otherwise the "*" group applied. A Disallow rule covering the root path (/) counted as a block; partial blocks of specific folders did not. llms.txt counted as present when the response was non-HTML and contained real content.
Bots checked (14): GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, anthropic-ai, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot, cohere-ai.
Excluded: T24 could not be reached, and Medyascope returned a bot challenge page for every request, including robots.txt. Both are excluded from the percentages, leaving 48 measured sites. DHA has no robots.txt (404), so all crawlers are treated as allowed.
Limitations: this study measures only the preference stated in robots.txt. It does not measure blocking at the server or CDN level; two sites (İHA and En Son Haber) returned 403 to our homepage request. The Reuters Institute comparison covers different countries, sites and dates.
Use the data
The full dataset is available as CSV and free to use with attribution (CC BY 4.0). To embed the charts on your site, use the code below.
Bots blocked chart:
<a href="https://www.hititmedya.com/insights/turkish-news-sites-ai-crawlers-study-2026"><img src="https://www.hititmedya.com/data/arastirma/haber-ai-botlari-bot-bazinda-en.svg" alt="How many Turkish news sites block each AI bot" width="1200" style="max-width:100%;height:auto"></a>
<p>Source: <a href="https://www.hititmedya.com/insights/turkish-news-sites-ai-crawlers-study-2026">Hitit Medya Research</a></p>Turkey vs global benchmarks:
<a href="https://www.hititmedya.com/insights/turkish-news-sites-ai-crawlers-study-2026"><img src="https://www.hititmedya.com/data/arastirma/haber-ai-botlari-karsilastirma-en.svg" alt="Turkish news sites and AI crawlers compared with global benchmarks" width="1200" style="max-width:100%;height:auto"></a>
<p>Source: <a href="https://www.hititmedya.com/insights/turkish-news-sites-ai-crawlers-study-2026">Hitit Medya Research</a></p>We plan to repeat this study every year with the same method at the same address. If you spot an error or a missing site, let us know.
Sources
- Fletcher, R. (2024). How many news websites block AI crawlers? Reuters Institute for the Study of Journalism.
- Rankability. llms.txt adoption (scan of 18 September 2026, Tranco top 1,000 domains).
- llmstxt.org: the llms.txt proposal.
- RFC 9309: Robots Exclusion Protocol.