Remove Duplicate Lines
Strips repeated lines from a list and tells you how many went.
Results
What this tool does
Lists arrive with repeats in them: email addresses pasted from three places, product codes exported twice, log lines that say the same thing. Removing the repeats by hand means sorting first and then reading down the column, which is slow and easy to get wrong. This keeps the first occurrence of every line and drops the rest, leaves the order untouched, and tells you how many lines it removed so you can see at a glance whether there was a problem worth looking into.
Formula
keeps the first occurrence of each line and throws away the repeats
Variables
| Symbol | Meaning | Unit |
|---|---|---|
text | Your text | — |
OUT | Cleaned text | — |
LB | Lines before | — |
LA | Lines after | — |
LR | Lines removed | — |
Worked example
- Your textapple banana apple cherry banana date
- Cleaned textapple banana cherry date
- Lines before6
- Lines after4
- Lines removed2
Limitations
- The calculation runs entirely in your browser. The values you type are never sent to a server.
Frequently asked questions
Which copy is kept?
The first one. The order of what remains is exactly the order it had, so a list that was already sorted stays sorted and a list that was not keeps its original sequence. Lines are compared exactly as written: a trailing space makes two otherwise identical lines count as different, which is why it often helps to remove extra spaces first.