Linux uniq does not search an entire file for every repeated value. It compares adjacent lines. If a hostname appears, another hostname follows, and the first hostname appears again, plain uniq keeps both separated appearances. That is expected behaviour, not evidence that the tool failed or that the lines necessarily contain hidden characters.
Choose the question first: are you investigating consecutive repeats in a sequence, or counting identical lines across the whole input? The correct pipeline depends on whether original order is evidence you need to preserve.
See the difference with a small fixture
This Bash example uses a temporary directory and an invented list of server names. It does not read production logs. Run the complete block together. The original fixture remains unchanged by both counting commands, and cleanup removes only the files created for this exercise.
work=$(mktemp -d) || exit 1
printf '%s\n' web-a web-a db-b web-a db-b db-b > "$work/hosts.txt"
printf '%s\n' 'Adjacent runs:'
LC_ALL=C uniq -c "$work/hosts.txt"
printf '%s\n' 'Whole-file counts, sorted by line:'
LC_ALL=C sort "$work/hosts.txt" | LC_ALL=C uniq -c
rm -- "$work/hosts.txt"
rmdir -- "$work"The first result has four groups: two web-a lines, one db-b line, one web-a line and two db-b lines. The second result has two groups: three db-b lines and three web-a lines. The spacing before counts is presentation; the important distinction is which lines belong to each group.
According to the GNU coreutils uniq manual, matching lines must be adjacent. The -c option prefixes each output line with its count. Sorting first makes identical lines neighbours, but changes their sequence. The result is ordered by the line text here, not ranked by frequency.
Use adjacent runs when sequence matters
Imagine repeated health-check messages separated by a recovery message. Grouping consecutive identical messages can help show bursts. Sorting first destroys the order that distinguishes those bursts from one continuous incident. Keep an untouched original and treat any grouped output as a derived view.
There is another limitation: uniq compares lines, not your application’s idea of an event. Two otherwise similar log messages with different timestamps are different full lines. Conversely, identical text does not prove the same event was recorded twice. Decide what the record represents before deleting anything.
For a run-oriented review, read the file with uniq -c and retain the original sequence separately. If you need durations, correlation identifiers or parsed fields, this simple full-line exercise is not enough. Use a documented parser rather than silently stripping information until lines look equal.
Count repeated values across the complete input
For a plain one-value-per-line inventory where order is irrelevant, sorting a stream before uniq is appropriate. Setting LC_ALL=C for both commands makes the example’s comparison environment explicit. It does not trim spaces, remove carriage returns or standardise spelling.
The -d option reports one line per repeated group, while -u reports groups containing only one line. These are still adjacent-group rules. On unsorted input, a value that appears elsewhere can be printed by uniq -u because its current group has one member. Do not interpret that as a whole-file uniqueness guarantee.
If you only want one copy of each distinct complete line, the manual also suggests sorting with sort -u. That is a different output from frequency counts. Choose it only when losing order and counts is acceptable.
Avoid overwriting the evidence you are inspecting
Never redirect a pipeline back into the same input filename. A shell can truncate that destination before the reader consumes it. Write any retained analysis to a new, deliberately chosen path and check it before considering a separate replacement operation.
For large files, sorting may require temporary storage and noticeable processing time even though the source file is not modified. Start with a small representative sample in a lab, then plan resources for the full input. This tutorial demonstrates behaviour, not production capacity.
If two different filenames might actually refer to the same underlying file, consult the separate hard-link identity check with stat. Filename identity is a different problem from duplicate line contents.
Quick decision checklist
- Need consecutive bursts? Keep order and inspect adjacent counts.
- Need total counts of identical full lines? Sort a stream before counting.
- Need case-insensitive or field-based matching? Define those rules explicitly first.
- Need to retain evidence? Keep the original and write derived output elsewhere.
Safety: This tutorial is for educational purposes. Test carefully before production use. WhileNetworking is not responsible for misuse, damage, data loss or production issues.
