Any tool that fetches a URL a stranger typed is an SSRF hole until proven otherwise. This is the seam every other package fetches through. Useful on its own to anyone building a URL-taking service, which is why it is published separately rather than buried in the crawler.
npm i @advance-labs/net-guardSSRF-guarded HTTP fetch seam — DNS-resolves and rejects private/loopback/link-local/CGNAT/cloud-metadata addresses, re-validates every redirect hop, caps body size and time, and host-pins to defeat DNS rebinding.
The engine behind our paid audit and the free Chrome extension. The rules are declarative data, not branching code, which is why the extension can run 40 of them client-side and the hosted audit can run 54 without two implementations drifting apart.
Sitemap first, links second, and it obeys robots.txt. Bounded on purpose: an audit crawler that hammers a prospect's site is a liability, not a feature.
Extraction with no network access at all. Give it a string of HTML, get back the metadata, heading tree, links and raw structured-data blocks. Keeping fetching out of it is what makes the rule tests deterministic.
Finds structured data in all three encodings, maps it to schema.org types, and checks the properties an answer engine actually needs rather than every property the spec allows.
The shared vocabulary. Every other toolkit package depends on it and nothing else depends on them, so a type change is one edit rather than six.