How to Extract URLs from Text Using Python
A working extractor in about thirty lines — find candidates, trim punctuation without breaking balanced brackets, then validate with urlparse rather than trusting the pattern.
Almost every answer to this question is one regular expression. That gets you 80% of the way and then fails on the cases you will actually hit: a link at the end of a sentence, a Wikipedia address with brackets in the path, a markdown link.
The fix is not a cleverer pattern. It is three small steps: match generously, trim carefully, then validate with a real parser.
Here is the whole thing.
import re
from urllib.parse import urlparse
# Step 1: find candidate runs. Deliberately generous — the checks come later.
CANDIDATE = re.compile(r"""
\b
(?:https?://|www\.) # an explicit scheme, or a www. label
[^\s<>"'`]+ # run until whitespace or a character a URL cannot hold
""", re.VERBOSE | re.IGNORECASE)
TRAILING = '.,;:!?\'"'
PAIRS = {')': '(', ']': '[', '}': '{'}
def trim(candidate: str) -> str:
"""Remove trailing punctuation, keeping balanced brackets."""
while candidate:
last = candidate[-1]
if last in TRAILING:
candidate = candidate[:-1]
continue
if last in PAIRS:
opener = PAIRS[last]
if candidate.count(last) > candidate.count(opener):
candidate = candidate[:-1]
continue
break
return candidate
def extract_urls(text: str, assume_https: bool = True) -> list[str]:
urls = []
seen = set()
for match in CANDIDATE.finditer(text):
url = trim(match.group(0))
if not url:
continue
if url.lower().startswith('www.'):
if not assume_https:
continue
url = 'https://' + url
# Step 3: let the parser decide, not the pattern.
parsed = urlparse(url)
if parsed.scheme not in ('http', 'https') or not parsed.netloc:
continue
if url not in seen:
seen.add(url)
urls.append(url)
return urlsRun against Python 3.11.15. The `list[str]` annotation needs Python 3.9 or newer; on older versions use `List[str]` from typing.
What it does on real input
Given this:
Docs at https://docs.example.com/start (see the Install section).
Mirror: www.example.net/status
Wikipedia-style: https://reference.example.org/wiki/Extraction_(data)
Ends a sentence: https://example.com/pricing.
In brackets: (https://example.com/terms)
Duplicate: https://docs.example.com/start
Angle: <https://example.com/privacy>
Not a link: javascript:alert(1)
Query kept: https://example.com/s?q=a+b&sort=Desc#Topit produces exactly:
https://docs.example.com/start
https://www.example.net/status
https://reference.example.org/wiki/Extraction_(data)
https://example.com/pricing
https://example.com/terms
https://example.com/privacy
https://example.com/s?q=a+b&sort=Desc#TopNote what happened: the trailing full stop went, the unbalanced closing bracket went, the balanced brackets survived, the duplicate collapsed, javascript: never matched, and the query and fragment are intact.
The three steps, and why each exists
Step 1 — match generously. The pattern stops at whitespace and at <, >, ", ' and backtick, because none of those appear in a URL written in prose. It deliberately does not try to validate structure; that is step 3's job.
Step 2 — trim carefully. This is the part single-regex solutions cannot do, because it requires counting. Sentence punctuation comes off unconditionally. A closing bracket comes off only when there is no matching opener inside the candidate — which is what keeps Foo_(bar) intact while stripping the bracket from (https://…/terms).
Step 3 — validate by parsing. urlparse tells you whether the result is actually usable. A pattern can tell you a string looks like a URL; only a parser tells you what it resolves to. This step is also what keeps javascript: and data: out of your results.
Variations
Get the line number too:
for line_number, line in enumerate(text.splitlines(), start=1):
for url in extract_urls(line):
print(line_number, url)Keep only certain domains:
from urllib.parse import urlparse
def on_domain(urls, domain):
domain = domain.lower().removeprefix('www.')
out = []
for url in urls:
host = (urlparse(url).hostname or '').lower()
if host == domain or host.endswith('.' + domain):
out.append(url)
return outRead a file:
from pathlib import Path
urls = extract_urls(Path('notes.md').read_text(encoding='utf-8'))Frequently asked questions
Why not use a single big URL regex?
Because the hard part is not recognising the shape of a URL, it is knowing where one ends in running text. That requires counting brackets, which regular expressions cannot do. Splitting match from trim from validate makes each part small enough to test.
Is there a library for this?
`urlextract` uses the public TLD list and handles bare domains, and `tldextract` is the standard choice for splitting a hostname into subdomain, domain and suffix. Both are good; the code above is useful when you want no dependency or need to control the trimming rules.
How do I handle internationalised domains?
`urlparse` keeps the Unicode form. To convert it to the punycode form that DNS uses, `hostname.encode("idna").decode("ascii")` does it, and raises on hostnames that cannot be encoded.
Sources
- Python docs — urllib.parse — Python Software Foundation
- Python docs — re, regular expression syntax — Python Software Foundation
- tldextract on PyPI — PyPI
- RFC 3986 — Uniform Resource Identifier syntax — IETF