dark

uniq – Show or ignore duplicate lines

The uniq command is used to display or ignore duplicate lines in a file. It is a Linux command that is useful for filtering out duplicate lines from a file, especially when working with large datasets. The command is simple to use and can be used in combination with other commands to perform complex operations.

Overview

The basic syntax of the uniq command is as follows:

uniq [option(s)] [input_file [output_file]]

The uniq command reads input from a file or standard input, removes duplicate lines, and writes the output to standard output or a specified file.

By default, uniq only removes adjacent duplicate lines. This means that if a file has duplicate lines that are not adjacent, they will not be removed. However, this behavior can be changed using some options.

Examples

  1. Display unique lines in a file:
uniq file.txt
  1. Ignore case while comparing lines:
uniq -i file.txt
  1. Display only duplicate lines:
uniq -d file.txt
  1. Count the number of occurrences of each unique line:
uniq -c file.txt
  1. Ignore the first ‘n’ characters while comparing lines:
uniq -s n file.txt
  1. Ignore the first ‘n’ fields while comparing lines:
uniq -f n file.txt

Options

The following table lists the available options for the uniq command:

Option Description
-c Prefix lines with the number of occurrences
-d Only print duplicate lines
-f n Ignore the first ‘n’ fields while comparing lines
-i Ignore case while comparing lines
-s n Ignore the first ‘n’ characters while comparing lines
-u Only print unique lines
-z Use null characters instead of newlines as separators

Troubleshooting tips

  • If you get an error message that says “No such file or directory”, make sure that the file you are trying to read exists in the specified location.
  • If you are not getting the expected output, try using some of the available options to modify the behavior of the command.

Notes

  • The uniq command only removes adjacent duplicate lines by default.
  • The uniq command is case-sensitive by default.
  • The uniq command can be used in combination with other commands to perform complex operations.

Related commands