URL Extractor from Website
Fetch one public page and list its links, or crawl a bounded part of a site and collect them across pages — with the source page, the anchor text and an internal/external label on every row.
- You provide
- A public page address. Choose single page, or a bounded same-site crawl.
- You receive
- Source page, destination, anchor text, internal/external label, and any rel="nofollow", ugc or sponsored values.
- Use it when
- Use it to audit the links on a page, or to inventory a small, defined part of a site.
Fetching happens on our server because your browser cannot read another site directly. We record only the temporary crawl state needed to resume, and delete it after a short time.
What a crawl cannot find
- • Pages nothing links to — a crawl follows links, so an orphan page is invisible.
- • Pages behind a login or a paywall.
- • Navigation that only exists once JavaScript runs. We read the HTML the server sends and do not run scripts, so a link built in the browser is not there for us. Paste the rendered HTML into the HTML tool instead.
- • Anything beyond your page and depth limits, or excluded by robots.txt.
What this tool cannot do
- A crawl follows links. A page that nothing links to cannot be found by any crawler, including this one.
- Pages behind a login, a paywall or an IP restriction are not reachable.
- We read the HTML the server sends and do not run JavaScript. Navigation that only exists after scripts run is invisible to us — paste the rendered HTML into the HTML tool instead.
- Defaults are 50 pages and depth 2. You can raise them to 200 pages and depth 4 here; both are product limits we tuned, not platform guarantees.
- The page limit and the link limit are separate. A crawl can stop because of either, and the results say which.
- robots.txt is respected by default, and pages it disallows are skipped and listed as skipped.
Two modes, because they are different jobs
Single page fetches one address and lists every link on it. It is quick, it is bounded, and it is what you want when you are checking one page.
Website crawl starts at an address and follows links within a scope you define. It is the right tool for a small inventory — a documentation section, a blog category, a product range — and the wrong tool for "give me every URL on the internet's largest retailer".
What "internal" means here
By default, internal means exactly the same hostname. A link from example.com to blog.example.com is external under that rule, because it is a different host.
You can switch to "this domain and its subdomains", which treats blog.example.com and shop.example.com as internal to example.com. The choice matters: only internal links are followed, and widening the scope can multiply the size of a crawl.
Both internal and external links are always listed. The scope only controls what gets followed.
How a crawl actually runs
A crawl is not one long request. It runs as a series of short steps, each fetching a few pages and saving its progress. That has consequences worth knowing:
- You can pause and resume. If you close the tab, the job pauses. Come back in the same browser within its retention period and you can resume from where it stopped.
- Nothing continues in the background. When the tab is closed, no pages are being fetched. We do not claim otherwise.
- Two tabs cannot double-count. Each step takes a lock on the job, so a duplicate request waits rather than running the same batch twice and blowing through your page limit.
- A dropped connection is recoverable. If a step dies halfway, its lock expires and the next step picks the job up again.
Avoiding the traps
Some sites contain effectively infinite URL space — calendars that generate a page per day forever, faceted search that multiplies filters, comment-reply links that nest. A naive crawler falls into these and never comes out.
Links that match known trap patterns (deep date archives, ?date=, ?replytocom=, very high page offsets) are not queued. A page you actually want that happens to match one of these patterns can still be reached by starting the crawl there.
Scope filters
- Only follow paths starting with — restrict the crawl to
/blog/,/docs/and so on. Several prefixes can be given. - Never follow paths starting with — the opposite, useful for
/tag/,/search/and other low-value sections. - Treat /page and /page#section as the same page — on by default, because a fragment almost never means a different document.
Frequently asked questions
Will this find every page on a site?
No, and nothing can promise that. A crawler follows links, so it can only reach pages that something links to, within the depth and page limits you set, that are not excluded by robots.txt, and that do not need JavaScript to be discovered. For a site’s own declared list of pages, its sitemap is usually more complete — that is exactly what a sitemap is for.
Why did the crawl stop before reaching my page limit?
Either the queue ran out — it had followed everything in scope — or it hit the link limit, which is separate from the page limit. The results panel says which, and both counters are shown throughout.
Does closing the tab keep the crawl running?
No. The browser is what asks for each step, so closing the tab pauses the job. You can resume it from the same browser until it expires, and the expiry time is shown while it runs.
Can I crawl a site that requires a login?
No. We fetch pages anonymously and have no way to sign in on your behalf, which is deliberate — a tool that accepted your credentials to crawl a private area would be a bad idea for both of us.
Is the crawl data stored?
Only the temporary state a resumable job needs: the queue, the pages already visited, and the links found so far. It is tied to an unguessable token held by your browser, it is not readable by anyone else, and it is deleted automatically after the retention period shown while the job runs.
Related tools
- You provide:
- A public sitemap address, an uploaded .xml or .xml.gz file, or pasted XML.
- You receive:
- Page URLs separated from child sitemap files, with lastmod when the file provides it, ready to copy or export.
- Use it when:
- Use it before a migration, for a content inventory, or any time you need a site’s declared URL list as a spreadsheet.
- You provide:
- A public page address, or pasted HTML.
- You receive:
- Each image address with the attribute it came from, its width descriptor where srcset provides one, and its alt text.
- Use it when:
- Use it to inventory the images on a page, check which sizes are being offered, or collect asset URLs before a migration.
- You provide:
- Pasted HTML source, or a local .html file.
- You receive:
- Each link with its href, visible anchor text, rel and target — plus image and media references if you ask for them.
- Use it when:
- Use it when you already have the markup: a saved page, a "view source" copy, or an email template.
- You provide:
- A list of URLs, or mixed text containing them.
- You receive:
- The hostname, or the registrable root domain, for each entry — with counts and an option to keep only unique values.
- Use it when:
- Use it to see which sites a link list actually points at, or to group thousands of URLs by domain.
Guides for this tool
- How to Extract and Copy Links in ChromeWhat Chrome can do on its own, a bookmarklet you can read before you use it, and an honest look at when an extension is worth the permissions it asks for.
- How to Extract Webpage Links with JavaScriptTwo lines in the browser console give you every link on the page — including the ones built by JavaScript that a crawler cannot see. Plus how to get anchor text, rel values and a downloadable file.
- How to Extract Links from a Website or a Single PageWhen to fetch one page, when to crawl, how to set a scope that finishes, and what a crawler structurally cannot find — including the pages it will always miss.