Duplicate URL Remover
Remove repeated addresses from a list while keeping the first occurrence and the original order — and see exactly which rows were removed, and why.
- You provide
- A list of URLs, one per line.
- You receive
- The de-duplicated list, plus a preview of exactly which entries were removed and which entry they matched.
- Use it when
- Use it after combining exports, or before importing a list somewhere that rejects duplicates.
Your list is processed in this browser tab. Nothing is uploaded.
What this tool cannot do
- With no options on, two addresses match only when they are byte-for-byte identical. Every looser rule is opt-in and shown in the removal preview.
- Paths and query values are never lowercased. They are case-sensitive on most servers, so /Page and /page are different pages.
- The output keeps your original text unless you ask for the normalised form instead.
Conservative by default
Out of the box, two lines are duplicates only if they are exactly the same text. That is deliberately strict, because almost every "obvious" relaxation is wrong somewhere:
https://example.com/?id=1andhttps://example.com/?id=2are different pages.https://example.com/Pageandhttps://example.com/pageare different on any case-sensitive server, which is most of them.https://example.com/docs#installandhttps://example.com/docs#apimay show completely different content on a single-page application.
So each looser rule is a checkbox you turn on knowingly, and the removal preview names the rule that caused each removal.
| Rule | Treats as the same | Watch out for |
|---|---|---|
| Ignore host capitalisation | `EXAMPLE.com/a` and `example.com/a` | Safe — hostnames are case-insensitive by definition. |
| Ignore a leading www. | `www.example.com/a` and `example.com/a` | Usually safe; a few sites serve different content. |
| Ignore tracking parameters | `?utm_source=x&id=4` and `?id=4` | Safe for utm_ and click ids; other parameters still count. |
| Ignore the #fragment | `/docs#install` and `/docs#api` | Not safe on single-page apps, where the fragment is the page. |
| Ignore the whole query string | `?id=1` and `?id=2` | Rarely what you want. This merges genuinely different pages. |
| Ignore a trailing slash | `/docs` and `/docs/` | Usually the same page; occasionally not. |
The removal preview
The Removed tab lists every line that was dropped, the line number it duplicated, and the rule that matched them. A CSV export of the same thing is available, so you can audit the decision rather than trusting it.
That is the point of the tool: not just a shorter list, but a list you can defend.
Frequently asked questions
Does it change my URLs?
Not unless you ask it to. The matching rules affect comparison only; the output keeps your original text. There is a separate option to write the normalised form instead, and it is off by default.
Which copy is kept?
The first occurrence, in the original order — that is usually what you want when the list is in a meaningful sequence. You can switch to keeping the last occurrence instead.
Can I sort the result?
Yes. Sorting is applied after de-duplication, so it changes the output order without changing which rows survive.
Related tools
- You provide:
- A list of URLs, one per line.
- You receive:
- The cleaned list, with a before/after view of every address that changed.
- Use it when:
- Use it before sharing links, publishing a reference list, or importing URLs that would otherwise carry campaign parameters.
- You provide:
- A list of URLs, or mixed text containing them.
- You receive:
- The hostname, or the registrable root domain, for each entry — with counts and an option to keep only unique values.
- Use it when:
- Use it to see which sites a link list actually points at, or to group thousands of URLs by domain.
- You provide:
- Pasted text, or a local TXT, CSV, JSON or Markdown file.
- You receive:
- Every web address found, with its line number, in the order it appeared.
- Use it when:
- Use it when the links are sitting inside prose, an email, a chat log, a notes file or a data export.
- You provide:
- A public sitemap address, an uploaded .xml or .xml.gz file, or pasted XML.
- You receive:
- Page URLs separated from child sitemap files, with lastmod when the file provides it, ready to copy or export.
- Use it when:
- Use it before a migration, for a content inventory, or any time you need a site’s declared URL list as a spreadsheet.
Guides for this tool
- How to Extract URLs from Text Using PythonA working extractor in about thirty lines — find candidates, trim punctuation without breaking balanced brackets, then validate with urlparse rather than trusting the pattern.
- How to Extract URLs and Domains from Text in Google SheetsFormulas that pull addresses out of a text column, get the hostname, and — importantly — why the popular "root domain" formula is wrong for half the internet.
- How to Extract Hyperlink URLs in Google SheetsGoogle Sheets has two kinds of hyperlink too. One yields to a formula; the other needs Apps Script, because no formula can see it. Here is how to tell which you have.
- How to Extract URLs from Hyperlinks in ExcelA cell showing "Click here" stores its address somewhere you cannot reach with a formula. Three methods that work — VBA, FORMULATEXT and unzipping the file — and exactly when each one applies.