We can use the uniq, comm command provided by linux to deduplicate, or compare text file contents.

Let’s begin the second part of the intermediate level of word processing.

Unique Result

Let’s prepare some repeated and unique contents for the uniq command:

1
2
3
4
5
6
7
8
9


echo 'I am duplicated' >> uniq_file.txt
echo 'I am duplicated' >> uniq_file.txt
echo 'I am duplicated' >> uniq_file.txt
echo 'I am Duplicated' >> uniq_file.txt
echo 'I am Duplicated' >> uniq_file.txt
echo 'I am Duplicated' >> uniq_file.txt
echo 'Line unique 1' >> uniq_file.txt
echo 'Line unique 2' >> uniq_file.txt
echo 'Line unique 3' >> uniq_file.txt

Prepared some repeated and unique contents

By default, the uniq command output does not include subsequently repeated contents:

1

uniq uniq_file.txt

Repeated Contents

We can add the -d or –repeated parameter to output the file’s repeated contents:

1
2


uniq -d uniq_file.txt
uniq --repeated uniq_file.txt

Output uniq_file.txt file's repeated contents

Unique Contents

We can add the -u or –unique parameter to output the file’s unique contents:

1
2


uniq -u uniq_file.txt
uniq --unique uniq_file.txt

Output uniq_file.txt file's unique contents

Repeated Contents Without Case Sensitivity

We can add the -d or –repeated plus -i or –ignore-case parameters to output the file’s repeated contents without case sensitivity:

1
2


uniq -d -i uniq_file.txt
uniq --repeated --ignore-case uniq_file.txt

Output uniq_file.txt file's repeated contents without case sensitivity

Contents With Their Occurrence Numbers

We can add the -c or –count to output the file’s repeated contents without case sensitivity:

1
2


uniq -c uniq_file.txt
uniq --count uniq_file.txt

Output uniq_file.txt file's contents with their occurrence numbers

Compared Result

We need to prepare two files to demonstrate the comm command:

1
2
3
4


printf '%s\n' a b c d e     > file1
printf '%s\n'   b c d e f g > file2
cat file1
cat file2

By default, comm outputs three columns of data, the first column is unique to the first file, the second column is unique to the second file, and the third column is the co-existent contents of the two files: