Uniq command on Linux (remove duplicates)

The uniq command on Linux removes duplicate lines from an ordered file. That’s why it’s often used in conjunction with the sort command.

The most common options are:

• -c: Indicates the number of occurrences at the beginning of the lines;
• -d: Prints only duplicate lines;
• -i: Ignores the difference between uppercase and lowercase letters;
• -u: Prints only single lines, which have no duplicates.

Without sorting, uniq is unable to eliminate duplicates. See the example below:

$ uniq supermarket rice beans meat potato lettuce tomato rice meat

$ sort supermarket | Uniq lettuce rice potato meat beans tomato

The uniq command is very useful for removing duplicates from files with a data source, especially text files, CSV files, etc.

The “-i” option causes uniq to ignore whether items are capitalized or lowercase.

The “-D” option shows only duplicate lines:

$ sort supermarket | Uniq -D rice rice meat meat