jacy/dukang
1
0
Fork 0
Code Issues Pull Requests Actions 23 Packages Projects Releases Wiki Activity

fix(admin): 后台校验文件避免 301，并让搜索引擎读到 noindex

Browse Source 
serve -s 会把 .html 301 去后缀，百度站长验证失败；同时放开 robots 抓取，避免旧标题一直留在 SERP。
This commit is contained in:
jacy
2026-09-01 09:11:43 +08:00
parent f35e1a3ba5
commit eeee55e215
12 changed files with 42 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/nginx-no-crawler.conf
+6 -5
View File
@@ -1,11 +1,12 @@
# 禁止搜索引擎抓取（shop / partner / admin，含测试域）
# 合规爬虫读 robots.txt；X-Robots-Tag 覆盖不先读 robots.txt 的抓取器
# 禁止搜索引擎收录（shop / partner / admin，含测试域）
# 必须允许抓取页面：Disallow:/ 会让百度无法读到 noindex，旧标题会一直留在 SERP。
# X-Robots-Tag 覆盖不先读 robots.txt / 不执行 JS 的抓取器。
add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
add_header X-Robots-Tag "noindex, nofollow, noarchive, nosnippet" always;
location = /robots.txt {
default_type text/plain;
charset utf-8;
add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
return 200 "User-agent: *\nDisallow: /\n";
add_header X-Robots-Tag "noindex, nofollow, noarchive, nosnippet" always;
return 200 "User-agent: *\nAllow: /\n";
}