How to Extract Links from a Website or a Single Page
When to fetch one page, when to crawl, how to set a scope that finishes, and what a crawler structurally cannot find — including the pages it will always miss.
The short answer: for one page, use the website tool in single-page mode. For a section of a site, switch to crawl mode and set a scope. If the site publishes a sitemap, read that instead — it is faster and more complete.
Now the detail that makes the difference between a crawl that finishes and one that does not.
| You want | Use | Why |
|---|---|---|
| Every link on one page | Single page | One request, immediate, no limits to think about |
| Links across a documentation section | Crawl with a path filter | Bounded and finishes |
| Every page a site publishes | [Sitemap tool](/url-extractor-from-sitemap/) | The site’s own list — faster and more complete than crawling |
| Links in markup you already have | [HTML tool](/url-extractor-from-html/) | No fetch at all; works for saved pages and email templates |
| Image addresses specifically | [Image URL extractor](/image-url-extractor/) | Handles srcset and picture properly |
What you get back
Each row is a link, not just an address:
- Found on — the page the link was on. On a crawl this is what makes the export useful: you can see which page points at a broken destination.
- URL — exactly as written in the markup.
- Anchor text — what a reader sees.
- Scope — internal or external.
- rel —
nofollow,ugc,sponsoredand anything else present.
That last one matters for audits. A link marked nofollow is being treated differently by search engines, and seeing it in the export is often the point of the exercise.
Defining "internal"
Two reasonable definitions, and the choice changes the size of your crawl:
- Exactly this hostname (default) —
blog.example.comis external toexample.com. - This domain and its subdomains —
blog.example.comis internal.
Only internal links are followed. Both kinds are always listed.
Scoping a crawl that actually finishes
- 1
Start in the right place
Start at
https://example.com/docs/rather than the homepage if the docs are what you want. Depth is counted from where you start, so starting close saves several levels. - 2
Set a path filter
Add
/docs/to Only follow paths starting with. Everything outside it is still listed as an external-to-scope link; it is simply not followed. - 3
Exclude the noise
Add
/tag/,/search/,/page/to Never follow paths. Tag archives and paginated listings multiply a crawl without adding distinct pages. - 4
Start small, then widen
Run 20 pages at depth 1 first. Look at what came back. If the scope is right, raise the limits; if you are collecting tag pages, fix the filters before spending the budget.
Why a crawl runs in steps
This site runs on serverless functions, which have a hard time limit per request. A crawl of fifty pages cannot be one request.
So it is a series of short steps, each claiming a small batch and saving its progress. The practical consequences:
- Pause and resume. Close the tab and the job pauses; come back in the same browser and resume it.
- Nothing runs in the background. With the tab closed, no pages are being fetched. Any tool that claims otherwise without a real background trigger is not telling you the truth.
- Two tabs cannot double-count. Each step takes a lock, so a duplicate request waits rather than burning your page limit twice.
- A dropped connection recovers. If a step dies, its lock expires and the next step picks the job back up.
Crawling considerately
Requests are paced, carry a URLExtractorBot/1.0 user agent with a link back here, are capped in size and time, and honour robots.txt including any Crawl-delay. Keep those defaults unless you own the site.
Frequently asked questions
Why did it find fewer pages than the site has?
Most often: the pages are not linked from within your scope, your depth limit stopped before reaching them, or they are built by JavaScript. Check the page and depth counters shown during the crawl, and compare against the site’s sitemap if it has one.
Can I crawl a site that needs a login?
No. We fetch anonymously and cannot sign in on your behalf — which is deliberate. A tool that took your credentials to crawl a private area would be a bad idea for everyone.
Does this check whether the links work?
No. Extraction tells you a link exists and where it points. It does not request the destination, so it cannot tell you whether it resolves. That is a link checker, which is a different tool.
What counts against the page limit?
Each page actually fetched. Links found on those pages count against a separate link limit. A crawl can stop because of either, and the result says which.