jacy
0ff61c2cd1
nginx 直出 robots.txt 并加 X-Robots-Tag，避免内部三端被收录。 Co-authored-by: Cursor <cursoragent@cursor.com>
12 lines
408 B
Plaintext
12 lines
408 B
Plaintext
# 禁止搜索引擎抓取（shop / partner / admin，含测试域）
|
||
# 合规爬虫读 robots.txt；X-Robots-Tag 覆盖不先读 robots.txt 的抓取器
|
||
|
||
add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
|
||
|
||
location = /robots.txt {
|
||
default_type text/plain;
|
||
charset utf-8;
|
||
add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
|
||
return 200 "User-agent: *\nDisallow: /\n";
|
||
}
|