For a deliberate case-insensitive comparison of human-readable labels, Python’s str.casefold() is usually a better starting point than lower(). Keep the original strings and derive a separate comparison key. If multiple originals produce one key, report the collision instead of silently deciding which record survives.
This tutorial builds an in-memory review tool for imported labels such as department names. It is not a recipe for changing account identities, passwords, filesystem names or database uniqueness rules.
Why lowercasing is not the same operation
The Python documentation describes casefolding as more aggressive than lowercasing because it aims to remove case distinctions. Its example is the German letter sharp s: lowercasing leaves it unchanged, while casefolding converts it to ss. Consequently, two labels that remain different after lower() can have equal casefolded keys.
Unicode also permits different code-point sequences for some visually equivalent text. An accented character can be precomposed or represented by a base character plus a combining mark. Casefolding is not a general replacement for Unicode normalization. Decide which equivalences your application intends before importing records.
Define and test one comparison policy
The example below uses canonical decomposition, NFD, before and after casefolding. The second normalization keeps the final key in the chosen normalized form after case transformations. It deliberately does not strip whitespace, remove punctuation or drop accents. Those would be additional matching policies, not harmless cleanup.
Run this with Python 3 in a disposable script. It only compares fixed strings and prints a short result; it does not edit files or send data anywhere.
import unicodedata
def label_key(value):
if not isinstance(value, str):
raise TypeError("label must be a string")
decomposed = unicodedata.normalize("NFD", value)
return unicodedata.normalize("NFD", decomposed.casefold())
assert "Straße".lower() != "STRASSE".lower()
assert label_key("Straße") == label_key("STRASSE")
assert label_key("Café") == label_key("Cafe\u0301")
assert label_key("cafe") != label_key("café")
assert label_key("Sales ") != label_key("Sales")
assert label_key("A-B") != label_key("AB")
labels = ["Straße", "STRASSE", "Café", "Cafe\u0301", "Sales"]
groups = {}
for original in labels:
groups.setdefault(label_key(original), []).append(original)
collisions = [values for values in groups.values() if len(values) > 1]
assert collisions == [["Straße", "STRASSE"], ["Café", "Cafe\u0301"]]
assert labels[0] == "Straße"
try:
label_key(None)
except TypeError:
pass
else:
raise AssertionError("non-string label was accepted")
print("Collision groups:", len(collisions))
print("Original labels:", len(labels))Expected output is Collision groups: 2 followed by Original labels: 5, on separate lines. The assertions also check that accent removal, whitespace trimming and punctuation removal have not accidentally become part of the policy. A failing assertion means you should investigate before using the function in an import.
Review collisions without destroying records
The dictionary stores a list of originals for each key. That matters: a simple assignment such as groups[key] = original would replace earlier entries with later ones. With the grouped approach, a reviewer can see both source spellings and decide whether they represent duplicates, legitimate separate entries or a data-entry issue.
For a real import, store the source row identifier alongside each label. Preserve the original input file, produce a separate review report, and leave the destination unchanged until the collision decisions are approved. Repeated identical labels also create a multi-entry group in this example; the code counts entries, not distinct spelling variants.
Validate the surrounding data format separately. For JSON inputs, rejecting duplicate JSON property names addresses a different overwrite risk. JSON field names and the label values inside those fields need separate policies.
Know what this key does not prove
Equal keys establish equality under this chosen transformation, not that two people, companies or assets are the same entity. Casefolding does not detect every visually confusable character, establish ownership or prevent impersonation. It also does not reproduce every language-specific collation rule.
NFD preserves canonical distinctions differently from compatibility normalization. Do not switch to NFKC or NFKD merely because more matches look convenient: compatibility transformations can intentionally merge additional characters. Document the desired behaviour and add examples from your actual data first.
Quick review checklist
- Original labels and row identifiers remain available.
- The normalization and case policy is documented.
- Positive matches and deliberate nonmatches have tests.
- Collisions are reviewed rather than automatically deleted.
- Authentication and storage rules are not silently changed.
Can I lowercase instead? For an explicitly ASCII-only contract that may be appropriate, but it is not interchangeable with the Unicode policy tested here.
Sources: Python’s casefold documentation and Unicode normalization reference.
Educational-use disclaimer: Test carefully before using these steps in production. WhileNetworking is not responsible for misuse, damage, data loss or production issues.
