📚 Indexed Pages

Discover what Common Crawl — the public dataset used by AI models and search engines — knows about a domain. Useful for spotting pages you forgot existed and finding unindexed content.

Domain

Max results

💡 Tip: try skillsinsider.com to see what Common Crawl has indexed on this site.

Querying Common Crawl indexes…

This usually takes 5–15 seconds. First query per domain may be slower.

🕰️ Historical URLs

Query the Internet Archive's Wayback Machine for every URL ever captured on a domain. Great for finding old content, deleted pages, and expired URLs that may still have backlinks.

Domain

Max results

Latest snapshot per URL (dedupe)

💡 Wayback may be slow for very large sites. Try a smaller subdomain if you hit the max results quickly.

Querying Internet Archive…

Wayback CDX can take 5–20 seconds depending on domain size.

🔎 Brand Mentions

Search Common Crawl for URLs that contain a specific string — usually your brand name or domain slug. Finds blog posts, comparison pages, and reviews that mention you in the URL.

Brand slug or keyword

Max results

💡 Only alphanumeric characters and hyphens. 3–63 characters. Example: skillsinsider or your-brand.

Searching Common Crawl indexes…

Scanning recent crawl data for URL matches.

🗺️ What Is Backlink Discovery?

Traditional backlink tools (Ahrefs, Moz, Semrush) maintain private indexes of the web's link graph — who links to whom. They cost hundreds of dollars per month because that data is expensive to gather and maintain.

This tool uses two free public datasets to answer related questions:

Common Crawl — a nonprofit that crawls a large portion of the web every month and releases the data publicly. It's what many AI models and search engines are trained on.
Wayback Machine — the Internet Archive's snapshot system, which has been capturing website snapshots since 1996.

Neither dataset gives you a pure "who links to you" answer. But combined, they help you find:

Forgotten pages — URLs on your site that Common Crawl still has indexed but you may have moved or deleted
Historical URLs — old paths that used to exist and may still be linked to from external sites
Brand mentions — pages across the web that reference your brand name in their URL, often review posts or comparison articles

Honest limitations

Not a backlink database — Common Crawl publishes crawled content, not an inverted link index. We can't tell you the exact anchor text of a link from a page we don't control.
Historical, not real-time — Common Crawl indexes crawl the web every month, not every minute. Data may be several weeks old.
Coverage varies — small sites may be entirely missing from Common Crawl. Popular sites have good coverage.
Verify before acting — any candidate source should be verified manually. Use our Backlink Toolkit to confirm that a specific page actually links to you.

How to use the results

From the Pages tab: look for URLs you no longer serve. If they return 200 (page exists) but aren't linked from your navigation, they may be orphaned. If they 404, you may want to redirect them or bring the content back.
From the Wayback tab: note the earliest and latest snapshot per URL. Old URLs that were captured many times are the ones that external sites most likely link to.
From the Brand Mentions tab: each result is a candidate. Verify the page actually mentions you, then reach out to strengthen the mention or request a follow link.

❓ Frequently Asked Questions

Not exactly. This tool doesn't have a complete index of who links to you — no free tool does. What it can do is find candidates: URLs on your domain that Common Crawl knows about, historical URLs from the Wayback Machine, and pages across the web that mention your brand. To verify that a specific page links to you, use our Backlink Toolkit.

Common Crawl is a nonprofit that crawls the web monthly and releases the data publicly for free. It's what many AI models are trained on and what many free SEO tools rely on as a data source. Using it directly means no API keys, no rate limits beyond our own, and no cost.

Common Crawl releases one index per month. Our tool queries the latest 3 indexes by default, so results typically cover the last 1–3 months of crawl data. Wayback CDX data goes back to 1996 and includes snapshots up to the last few days.

No. Search queries are processed in memory and cached for a few hours on the server side (keyed by domain, not by user), but no user identifiers are attached. IP addresses are hashed only for rate limiting. Nothing is shared with third parties beyond the public API requests to Common Crawl and the Internet Archive.

The Wayback Machine only has data for domains that have been captured. Brand-new domains or niche sites may have no snapshots. If you're sure the site is older, try without the "latest only" option — some URLs are only captured once every few years.

Four practical uses: (1) find orphaned pages on your site that should be linked internally, (2) recover old content that used to have backlinks, (3) set up 301 redirects for historical URLs so old links don't break, and (4) reach out to pages that mention your brand to strengthen or convert mentions into links.

20 queries per hour per IP address. Each query may scan 3 Common Crawl indexes or hit the Wayback CDX API once. This is generous enough for normal use and keeps the service fast for everyone.

Any publicly accessible domain works. Sites behind authentication, intranet sites, and localhost will not work. Domains with very little public web presence may return empty results — that's expected.