Linux uniq Command: Remove or Count Adjacent Duplicate Lines

uniq reduces consecutive identical lines to a single line or shows the number of repetitions. To remove duplicate values from the entire file, you usually need to sort first.

What is the uniq command?

Identical lines that are apart from each other cannot be removed using uniq alone. If you need to preserve the input order, do not sort blindly; first decide which duplicates matter. Using uniq after sorting will change the original line order.

Basic syntax

uniq [option] [input file [output file]]

The installed implementation and options may vary depending on the distribution. Check the description on your current system with man uniq.

Examples

Removing adjacent duplicates

When the same line occurs consecutively, only the first line is printed.

uniq events.txt

Counting all duplicates

After grouping identical values with sorting, use -c to display the number of repetitions.

sort names.txt | uniq -c

Check only duplicate values

Print the types of values that occur two or more times consecutively.

sort names.txt | uniq -d

Distinguishing adjacent duplicates from overall duplicates

uniq only combines consecutive identical lines. In the example below, the last apple is separate because it is not next to the first two lines.

printf 'apple\napple\nbanana\napple\n' | uniq -c

Example output (spaces before numbers may vary depending on implementation):

2 apple
1 banana
1 apple

To combine identical lines across the entire file into one, use sort file | uniq after sorting. However, sorting changes the original line order, so do not apply it blindly to logs or event records where the input order is important. -d shows only duplicate groups, and -u shows only groups that appear once.

Main Options and Format

Precautions when using

The results of sorting and comparison with sort | uniq may vary depending on the locale. If you need an exact byte-based aggregation, apply the same LC_ALL=C environment to both commands. uniq -u refers to lines that appear only once, not all unique values.

Frequently Asked Questions

Why doesn’t uniq remove duplicate rows that are not consecutive?

Because it compares the current row with the immediately preceding row. To remove all duplicates, sort the input so that identical rows are grouped together.

Official Documentation

You can check the exact behavior of the options and implementation differences in the official uniq documentation.