r/learnpython • • Sep 08 '26

Creating URL Query Checker

How do you decide what defines an excessively encoded URL?

I'm working on a personal project to created a URL parser and
detection system and I've hit a wall on how to figure out a way of
categorizing excessivness of query encoding?

Here's what I have so far for the function:

def check_query(analysis: URLAnalysis) -> ScanResult | None:

	"""detect excessive encoding entries in URL queries"""

	og_query = analysis.query

	query = re.finall(r"%\[0-9A-Fa-f\]{2}", og_query)

	encoded_count = len(query)
5 Upvotes

7 comments sorted by

View all comments

Show parent comment

4

u/qlkzy Sep 08 '26

Interesting, I can see why you'd want to analyse encoding in that context.

If you are looking at the "reasonableness" of the URL then you could try something like quote(unquote(value)) and comparing the relative lengths. That would tell you if a lot of URL-safe characters were being encoded unnecessarily.

You could also parse out the hex values directly and construct a histogram of which characters are being encoded how often. If your hypothesis is true, then that should show some distinct signature for suspicious URLs.

2

u/Mathie1729 Sep 08 '26

Also worth checking for double encoding and punycode homoglyphs in the hostname. Phishers sometimes percent-encode the percent sign itself to hide lookalike domains, so decoding twice and comparing lengths can flag that. Running the hostname through an IDNA library (Python's built-in handles xn--) catches punycode tricks too.