How to Remove Duplicate Lines from Any List

What deduplication does

The tool compares each line to the lines already seen. The first time a line appears, it is kept; every later identical line is dropped. Order is preserved — the surviving lines stay in the order of their first appearance, which is usually what you want for a list that has a logical sequence.

Deduplication is exact by default: two lines must match character-for-character to be considered duplicates. That precision is what makes the two options below so important.

Case sensitivity and whitespace

Two options control what counts as a duplicate:

• Case sensitivity — when on, "Apple" and "apple" are different lines and both are kept. When off, they are treated as the same line and only the first survives. Turn it off for names and labels where capitalisation is cosmetic; keep it on for code and data where case matters.
• Trim whitespace — when on, leading and trailing spaces are ignored, so "Apple" and "Apple " count as the same line. Turn it on when your list came from a source that adds stray spacing; keep it off when indentation is meaningful.

Strategies for cleaning lists

A short workflow handles almost every list-cleaning job:

• Clean first. Run the list through the Text Cleaner to normalise spacing and line breaks before deduplicating, so cosmetic differences do not create false duplicates.
• Choose your options deliberately. Decide whether case and trailing whitespace should matter before you run the tool, not after.
• Dedupe, then sort. Remove duplicates first, then order the survivors with Sort Lines — sorting before deduping just rearranges the repeats.
• Review the count. Note how many lines were removed; a surprisingly large number can reveal a data-entry bug or a copy-paste error worth investigating.

When to remove duplicate lines

Deduplication is the right move whenever a list has grown by accretion: email and contact exports that double up, keyword lists built from several sources, log files that repeat entries, brainstormed idea lists with repeats, and CSV or TSV data where rows have been appended more than once. After deduping, pass the result to Sort Lines or Text to List to finish the job.