教程站的一篇文章，可能同时有正文地址、FAQ 锚点，以及带来源参数的购买入口。它们的用途不同：正文地址用于发现内容，锚点定位答案，购买链接记录来源。一次“清理重复 URL”，不应该顺手删除后两者。
我采用的边界是：只对发现清单生成规范地址，不改 HTML 原文，也不回写购买 href。本文的离线示例使用 Python 标准库，读取构建后的 HTML 与现有 sitemap，输出差异报告及一份候选地图；不请求线上页面，不提交索引，不影响购买流程。
一、先分清三类地址
原文：/guide/pro.html 答案：/guide/pro.html#renewal 购买：shop.example.test/item/8?utm_…
前两者通常对应同一份文档，因此地图只保留正文地址。第三条不是正文的重复页：它属于购买路径，参数可能被归因系统使用，不能因为“不够干净”就删掉。
二、只删已知追踪参数，不猜业务参数
示例显式列出 utm_source、utm_medium、utm_campaign 等常见追踪字段。lang、plan、version、ref 以及未知 utm_custom 都原样保留。保留参数的顺序和编码也不重排，避免把“看起来等价”误当作服务端行为完全一致。
因此，a.html?utm_source=x#faq 可以成为发现地址 a.html；a.html?lang=en 与 a.html?lang=zh 则不会被擅自合并。如果业务确实将两者视为同一页，应先明确 canonical，而不是扩大删除名单。
三、canonical 优先，但必须有边界
每篇页面优先采用唯一且合法的 canonical；缺失时使用页面自身地址。跨域、越过部署子目录的 canonical 会告警并回退；多个 canonical 不猜哪条正确。别名形成链或环时直接停止，留给维护者检查。
同域还不够。例如项目部署在 /guide/，/guide-other/ 不能因字符串前缀相似就进入地图。示例同时校验协议、主机、端口与完整目录边界，并拒绝编码后的上级路径和反斜杠。HTML 中若使用 base 标签，相对地址也必须先按它解析，再校验范围。
四、如何运行与验收
将下方完整实现保存为 tool.py，测试保存为 test_tool.py。运行测试后，再将 --html-dir 指向构建输出；--candidate 只允许创建新文件，不会覆盖原地图。输入应包含全部要发现的公开 HTML，不要把后台、预览稿或私有文件混进构建目录。
重点看四项：missing_urls 是缺少的正文，unexpected_urls 是地图中不在本次页面集合的地址，duplicate_sitemap_entries 是规范化后的重复数，warnings 是待核 canonical。候选地图不是上线指令：出现差异要先判断原因，不能看到“unexpected”就批量删线上页面。
我维护的 AIXiamo 教程目录也是这类“答案与办理入口并存”的使用场景：gpt-guide.aixiamo.com/knowledge 。这里只作为维护场景示例；下面工具是本文的独立离线实现，不表示已在该目录公开托管。
五、验收范围与上线检查
本地检查能证明候选地图的地址集合，以及读取过程没有改动原购买链接；不能证明线上全部返回 200、canonical 已被搜索引擎采用，更不能保证收录或 AI 引用。发布前仍需沿既有流程核对线上状态；已经成功提交的地图，不应只因生成了新报告就反复提交。
本文含AI辅助起草，代码与构造样例已在本地测试复核。
运行： python3 -m unittest -v test_tool python3 tool.py --base docs.example.test/guide/ --html-dir ./build --sitemap ./build/sitemap.xml --candidate ./sitemap.candidate.xml
完整实现（tool.py）：
"""Audit static discovery URLs without rewriting any HTML purchase links."""
import argparse
import json
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import unquote, unquote_plus, urljoin, urlsplit, urlunsplit
import xml.etree.ElementTree as ET
TRACKING_KEYS = frozenset({
"utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
"utm_id", "utm_source_platform", "utm_creative_format",
"utm_marketing_tactic", "gclid", "dclid", "fbclid", "msclkid",
})
NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
class URLPolicy:
def __init__(self, base_url):
base = urlsplit(base_url)
if (base.scheme not in {"https", "http"} or not base.hostname
or base.username or base.password or base.query or base.fragment
or not base.path.endswith("/") and base.path):
raise ValueError("base must be an HTTP(S) directory URL ending in /")
self.base = base
self.base_url = base_url
self.origin = self._origin(base)
self.scope = unquote(base.path or "/")
self._safe_path(base.path or "/")
@staticmethod
def _origin(parts):
return (parts.scheme.lower(), parts.hostname.lower(),
parts.port or (443 if parts.scheme == "https" else 80))
@staticmethod
def _safe_path(path):
decoded = path
for _ in range(5):
value = unquote(decoded)
if value == decoded:
break
decoded = value
else:
raise ValueError("excessively nested path encoding")
if "\\" in decoded or "\x00" in decoded or ".." in decoded.split("/"):
raise ValueError("unsafe path")
return decoded
def normalize(self, href, from_url=None):
parts = urlsplit(urljoin(from_url or self.base_url, href))
if (parts.scheme not in {"https", "http"} or not parts.hostname
or parts.username or parts.password or self._origin(parts) != self.origin):
raise ValueError("outside permitted origin")
path = parts.path or "/"
decoded = self._safe_path(path)
if not (decoded.startswith(self.scope) or decoded == self.scope.rstrip("/")):
raise ValueError("outside permitted base path")
# Retain each non-tracking query component byte for byte, including order.
query = "&".join(part for part in parts.query.split("&")
if unquote_plus(part.partition("=")[0]).lower()
not in TRACKING_KEYS)
return urlunsplit((self.base.scheme, self.base.netloc, path, query, ""))
class Page(HTMLParser):
def __init__(self, html):
super().__init__(convert_charrefs=True)
self.hrefs, self.canonicals, self.bases = [], [], []
self.feed(html)
def handle_starttag(self, tag, attrs):
attrs = dict(attrs)
href = attrs.get("href")
if href is None:
return
if tag == "a":
self.hrefs.append(href)
if tag == "base":
self.bases.append(href)
if tag == "link" and "canonical" in attrs.get("rel", "").lower().split():
self.canonicals.append(href)
def audit(pages, sitemap_xml, base_url):
"""pages maps public page URLs to HTML strings; returns a read-only report."""
if not pages:
raise ValueError("no public HTML pages supplied")
policy = URLPolicy(base_url)
aliases, originals, links, warnings = {}, {}, [], []
for page_url, html in pages.items():
source = policy.normalize(page_url)
page = Page(html)
originals[page_url] = page.hrefs.copy()
resolution_base = page_url
if len(page.bases) > 1:
raise ValueError("multiple base tags: " + page_url)
if page.bases:
resolution_base = policy.normalize(page.bases[0], page_url)
target = source
if len(page.canonicals) == 1:
try:
target = policy.normalize(page.canonicals[0], resolution_base)
except ValueError:
warnings.append({"page": page_url, "issue": "rejected canonical"})
elif len(page.canonicals) > 1:
warnings.append({"page": page_url, "issue": "ambiguous canonical"})
aliases[source] = target
for href in page.hrefs:
try:
links.append(policy.normalize(href, resolution_base))
except ValueError:
pass
# A -> B -> C and canonical loops need human review, not guessed precedence.
for source, target in aliases.items():
if target in aliases and aliases[target] != target:
raise ValueError("canonical chain or loop: " + source)
expected = sorted(set(aliases.values()))
root = ET.fromstring(sitemap_xml)
if root.tag not in {"urlset", "{" + NS + "}urlset"}:
raise ValueError("supply a urlset, not a sitemap index")
raw_locs = [node.text.strip() for node in root.iter()
if node.tag in {"loc", "{" + NS + "}loc"} and node.text]
accepted, rejected = [], []
for loc in raw_locs:
try:
normalized = policy.normalize(loc)
accepted.append(aliases.get(normalized, normalized))
except ValueError:
rejected.append(loc)
ET.register_namespace("", NS)
result = ET.Element("{" + NS + "}urlset")
for url in expected:
entry = ET.SubElement(result, "{" + NS + "}url")
ET.SubElement(entry, "{" + NS + "}loc").text = url
return {
"discovery_urls": expected,
"missing_urls": sorted(set(expected) - set(accepted)),
"unexpected_urls": sorted(set(accepted) - set(expected)),
"rejected_sitemap_urls": rejected,
"duplicate_sitemap_entries": len(accepted) - len(set(accepted)),
"internal_links": sorted(set(aliases.get(url, url) for url in links)),
"original_hrefs": originals,
"warnings": warnings,
"candidate_sitemap": ET.tostring(result, encoding="unicode"),
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base", required=True)
parser.add_argument("--html-dir", required=True, type=Path)
parser.add_argument("--sitemap", required=True, type=Path)
parser.add_argument("--candidate", type=Path,
help="optional NEW candidate file; never replaces input")
args = parser.parse_args()
folder = args.html_dir.resolve()
pages = {}
for path in sorted(folder.rglob("*.html")):
relative = path.resolve().relative_to(folder).as_posix()
public_path = relative[:-10] if path.name == "index.html" else relative
pages[urljoin(args.base, public_path)] = path.read_text(encoding="utf-8")
report = audit(pages, args.sitemap.read_text(encoding="utf-8"), args.base)
if args.candidate:
if args.candidate.resolve() == args.sitemap.resolve():
raise ValueError("candidate must not overwrite the original sitemap")
with args.candidate.open("x", encoding="utf-8") as stream:
stream.write(report["candidate_sitemap"] + "\n")
print(json.dumps(report, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
离线测试（test_tool.py）：
import contextlib
import io
import json
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
from tool import URLPolicy, audit, main
BASE = "https://docs.example.test/guide/"
EMPTY = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" />'
class DiscoveryTests(unittest.TestCase):
def setUp(self):
self.policy = URLPolicy(BASE)
def test_tracking_and_fragment_removed(self):
self.assertEqual(self.policy.normalize("a.html?utm_source=x#faq"), BASE + "a.html")
def test_functional_query_preserved_byte_for_byte(self):
self.assertEqual(self.policy.normalize("a.html?lang=zh%20CN&view=full+size&utm_campaign=q&n=2"),
BASE + "a.html?lang=zh%20CN&view=full+size&n=2")
def test_unknown_utm_is_not_assumed_tracking(self):
self.assertEqual(self.policy.normalize("a.html?utm_custom=keep&ref=keep"),
BASE + "a.html?utm_custom=keep&ref=keep")
def test_external_host_and_subdomain_rejected(self):
for url in ("https://evil.test/guide/", "https://sub.docs.example.test/guide/"):
with self.assertRaises(ValueError):
self.policy.normalize(url)
def test_basepath_prefix_is_not_scope(self):
with self.assertRaises(ValueError):
self.policy.normalize("/guide-other/a.html")
def test_literal_traversal_outside_scope_rejected(self):
with self.assertRaises(ValueError):
self.policy.normalize("../outside.html")
def test_encoded_traversal_and_backslash_rejected(self):
for path in ("%2e%2e/out.html", "%252e%252e/out.html", "x%5cy.html"):
with self.assertRaises(ValueError):
self.policy.normalize(path)
def test_non_http_scheme_and_credentials_rejected(self):
for url in ("javascript:alert(1)", "mailto:hello@example.test", "https://u:p@docs.example.test/guide/"):
with self.assertRaises(ValueError):
self.policy.normalize(url)
def test_samehost_scheme_and_nonstandard_port_rejected(self):
for url in ("http://docs.example.test/guide/", "https://docs.example.test:444/guide/"):
with self.assertRaises(ValueError):
self.policy.normalize(url)
def test_relative_canonical_takes_precedence(self):
page = '<link rel="canonical" href="current.html?utm_medium=x#top">'
result = audit({BASE + "old.html": page}, EMPTY, BASE)
self.assertEqual(result["discovery_urls"], [BASE + "current.html"])
def test_invalid_canonical_falls_back_with_warning(self):
result = audit({BASE + "a.html": '<link rel="canonical" href="https://outside.test/">'}, EMPTY, BASE)
self.assertEqual(result["discovery_urls"], [BASE + "a.html"])
self.assertEqual(result["warnings"][0]["issue"], "rejected canonical")
def test_ambiguous_canonical_warns_without_guessing(self):
html = '<link rel="canonical" href="a.html"><link rel="canonical" href="b.html">'
result = audit({BASE + "source.html": html}, EMPTY, BASE)
self.assertEqual(result["discovery_urls"], [BASE + "source.html"])
self.assertEqual(result["warnings"][0]["issue"], "ambiguous canonical")
def test_purchase_href_and_html_are_not_modified(self):
href = "https://shop.example.test/item/8?utm_source=github&utm_medium=guide&utm_campaign=original&utm_content=pro#buy"
html = '<a href="' + href.replace("&", "&") + '">购买</a>'
pages = {BASE + "a.html": html}
before = pages.copy()
result = audit(pages, EMPTY, BASE)
self.assertEqual(pages, before)
self.assertEqual(result["original_hrefs"][BASE + "a.html"], [href])
def test_sitemap_deduplicates_tracking_variants(self):
xml = '<urlset><url><loc>' + BASE + 'a.html?utm_source=x</loc></url><url><loc>' + BASE + 'a.html#faq</loc></url></urlset>'
result = audit({BASE + "a.html": ""}, xml, BASE)
self.assertEqual(result["duplicate_sitemap_entries"], 1)
self.assertEqual(result["missing_urls"], [])
self.assertEqual(result["discovery_urls"], [BASE + "a.html"])
def test_functional_query_is_not_merged(self):
pages = {BASE + "a.html?lang=en": "", BASE + "a.html?lang=zh": ""}
self.assertEqual(len(audit(pages, EMPTY, BASE)["discovery_urls"]), 2)
def test_html_base_tag_is_resolved_safely(self):
html = '<base href="sub/"><link rel="canonical" href="a.html"><a href="b.html#top">B</a>'
result = audit({BASE + "index.html": html}, EMPTY, BASE)
self.assertEqual(result["discovery_urls"], [BASE + "sub/a.html"])
self.assertEqual(result["internal_links"], [BASE + "sub/b.html"])
def test_external_base_tag_fails_closed(self):
with self.assertRaises(ValueError):
audit({BASE + "a.html": '<base href="https://other.test/">'}, EMPTY, BASE)
def test_canonical_chain_requires_manual_review(self):
pages = {BASE + "a.html": '<link rel="canonical" href="b.html">',
BASE + "b.html": '<link rel="canonical" href="c.html">'}
with self.assertRaises(ValueError):
audit(pages, EMPTY, BASE)
def test_wrong_xml_and_sitemap_index_rejected(self):
for xml in ("<urlset>", "<sitemapindex/>"):
with self.assertRaises(Exception):
audit({BASE + "a.html": ""}, xml, BASE)
def test_unknown_sitemap_page_is_reported_not_silently_kept(self):
xml = '<urlset><url><loc>' + BASE + 'missing.html</loc></url></urlset>'
result = audit({BASE + "a.html": ""}, xml, BASE)
self.assertEqual(result["unexpected_urls"], [BASE + "missing.html"])
self.assertEqual(result["missing_urls"], [BASE + "a.html"])
def test_empty_page_collection_fails_closed(self):
with self.assertRaises(ValueError):
audit({}, EMPTY, BASE)
def test_cli_index_mapping_and_original_files_unchanged(self):
with tempfile.TemporaryDirectory() as tmp:
folder = Path(tmp)
(folder / "index.html").write_text("<h1>Index</h1>", encoding="utf-8")
(folder / "myindex.html").write_text("<h1>Not directory</h1>", encoding="utf-8")
sitemap = folder / "sitemap.xml"
sitemap.write_text(EMPTY, encoding="utf-8")
before = {p.name: p.read_bytes() for p in folder.iterdir()}
output = io.StringIO()
argv = ["tool.py", "--base", BASE, "--html-dir", str(folder), "--sitemap", str(sitemap)]
with patch.object(sys, "argv", argv), contextlib.redirect_stdout(output):
main()
result = json.loads(output.getvalue())
self.assertEqual(result["discovery_urls"], [BASE, BASE + "myindex.html"])
self.assertEqual(before, {p.name: p.read_bytes() for p in folder.iterdir()})
def test_cli_refuses_overwriting_input_map(self):
with tempfile.TemporaryDirectory() as tmp:
folder = Path(tmp)
(folder / "index.html").write_text("", encoding="utf-8")
sitemap = folder / "sitemap.xml"
sitemap.write_text(EMPTY, encoding="utf-8")
argv = ["tool.py", "--base", BASE, "--html-dir", str(folder),
"--sitemap", str(sitemap), "--candidate", str(sitemap)]
with patch.object(sys, "argv", argv), self.assertRaises(ValueError):
main()
self.assertEqual(sitemap.read_text(encoding="utf-8"), EMPTY)
if __name__ == "__main__":
unittest.main(verbosity=2)
本次离线验收：23 项测试通过。测试结果只覆盖上述示例逻辑，不代表线上抓取或索引状态。
