Remove Duplicate Words
Keep the first appearance of every word and throw the repeats away, ignoring capitals and punctuation.
Results
What this tool does
Keyword lists, tag sets and brainstormed notes fill up with the same word written three times, often in different cases so a plain sort will not catch them. This page keeps the first of each and drops the rest, treating "Red", "red" and "red," as one word while preserving whichever form came first. The count of characters removed tells you how much of the list was repetition.
Formula
keeps the first time each word appears and throws away the repeats, ignoring case and punctuation at the edges
Variables
| Symbol | Meaning | Unit |
|---|---|---|
text | Your text | — |
OUT | Result | — |
CB | Characters | — |
CA | Characters after | — |
CR | Characters removed | — |
Worked example
- Your textblue green red blue yellow green blue orange red purple
- Resultblue green red yellow orange purple
- Characters55
- Characters after35
- Characters removed20
Limitations
- The calculation runs entirely in your browser. The values you type are never sent to a server.
Frequently asked questions
Does it ignore capitals and punctuation?
When deciding what counts as a repeat, yes: "The", "the" and "the," are all the same word for that purpose. When writing the result, no: the first time a word appears it is kept exactly as you typed it, with its capital letter and its comma. So "The, the THE cat." comes back as "The, cat." — the first form survives and the rest go.
What happens to the line breaks?
They go. The result comes back as one run of words separated by single spaces, because a word list with the repeats taken out no longer has the shape it had. If you need to keep a line-per-entry layout, the tool for that is the one that removes duplicate lines, which works on whole lines rather than on the words inside them.