Remove Duplicate Lines
Paste your list below — duplicate lines disappear instantly, keeping the first occurrence of each one.
Paste a list above to remove duplicate lines.
Cleaning a list without opening a spreadsheet
Mailing lists, keyword exports, log files and copy-pasted spreadsheet columns almost always end up with repeated entries. Scrolling through by eye to find and delete them is slow and error-prone, and a spreadsheet's own dedupe feature usually forces you to load the whole file just to fix a short list. This tool takes whatever you paste, one item per line, and removes every repeat while keeping the very first time each line appeared — instantly, entirely inside the browser tab, with nothing uploaded.
How duplicate detection works
By default, two lines count as duplicates only if they match exactly, character for character. Three options change how "the same" is decided, and they can be combined:
- Case-insensitive treats
Apple,appleandAPPLEas one value — only the first spelling you pasted is kept. - Trim whitespace before comparing ignores leading and trailing spaces, so
bananaandbanana(with a trailing space from a copy-paste) are recognized as duplicates — the visible text of the surviving line is left exactly as pasted. - Also sort the output alphabetically re-orders the cleaned list A→Z once duplicates are gone, instead of preserving the order lines first appeared in.
Turning on Show only the duplicate lines flips the result: instead of the cleaned list, you get a short report of just the values that showed up more than once, each listed a single time — handy for auditing what was repeated rather than only getting a cleaned copy.
Example 1 — exact duplicates. Input: apple, banana, apple, cherry, banana. With every option off, the result is apple, banana, cherry — 3 unique lines, 2 duplicates removed.
Example 2 — case and whitespace variants. Input: Apple, apple, APPLE (one has a trailing space). With every option off, all three survive as distinct lines. Turn on both Case-insensitive and Trim whitespace and they collapse to a single Apple — 1 unique line, 2 duplicates removed.
Example 3 — duplicates-only view. Input: a 200-line email export where 12 addresses each appear twice. Turning on Show only the duplicate lines outputs just those 12 addresses — one line each — instead of the full 188-line cleaned list, so you can see at a glance exactly what was repeated.
Common use cases
| Task | Recommended options |
|---|---|
| Clean a mailing or contact list | Case-insensitive + Trim whitespace |
| Deduplicate keyword research exports | Case-insensitive |
| Find which values repeat in a large list | Show only the duplicate lines |
| Deduplicate case-sensitive identifiers (codes, SKUs) | Leave case-insensitive off |
| Produce a clean, alphabetized reference list | Also sort the output alphabetically |
Things to know
- The cleaned list preserves the original order lines appeared in — the tool removes repeats, it doesn't shuffle anything, unless you turn sorting on.
- Blank lines are treated like any other line: several blank lines in a row collapse into the one that appeared first, same as any other repeated value.
- Comparison options only affect which lines are treated as matches — the text of every surviving line is kept exactly as you pasted it, never altered.
- Everything runs locally in plain JavaScript; nothing you paste is uploaded, logged, or sent to a server.
Sources & further reading
- MDN Web Docs — Set, the JavaScript structure used to detect duplicate values
- MDN Web Docs — String.normalize(), why visually identical text can compare as different
- Unicode Consortium — UAX #15 Normalization Forms, the standard behind text equivalence
- W3C Internationalization — case folding and the limits of case-insensitive matching
Frequently asked questions
How do I remove duplicate lines from a list while keeping the first occurrence?
Paste your list into the box, one item per line, and click "Remove duplicates." The tool scans top to bottom and keeps the first time each line appears, deleting every later repeat — so the surviving lines stay in their original order instead of being resorted. This is the standard behavior for cleaning email lists, keyword lists, or log lines where the first entry usually matters most. Turn on "Also sort the output alphabetically" afterward if you want the cleaned list ordered too.
What's the difference between case-sensitive and case-insensitive duplicate removal?
With "Case-insensitive" on, "Apple", "apple" and "APPLE" are treated as the same line, so only the first one you pasted survives and the rest are removed as duplicates. With it off, capitalization matters, so "Apple" and "apple" are kept as two separate, distinct lines even though a human reader would call them the same word. Turn case-insensitive matching on for names, emails, and everyday word lists, and leave it off for anything where capitalization is meaningful, like code identifiers or product SKUs.
Does trimming whitespace affect which lines count as duplicates?
Yes. Pasted lists — especially from spreadsheets or emails — often carry invisible leading or trailing spaces, so "banana" and "banana " look identical but are technically different strings. Turning on "Trim whitespace before comparing" strips leading and trailing spaces before checking for matches, so those two lines are correctly recognized as duplicates. The trimming only affects the comparison — your original line text in the result is left exactly as you pasted it.
Can I see only the lines that were duplicates, instead of the deduplicated list?
Yes. Turn on "Show only the duplicate lines" to flip the output from the cleaned, unique list to a report of just the values that appeared more than once — each one listed a single time. This is useful when you want to audit what was repeated in a list, like finding which email addresses or SKUs show up twice, rather than just getting a cleaned copy. The unique-count and removed-count stats above the result stay accurate either way.
Is my pasted list uploaded to a server?
No. This tool runs entirely in your browser using plain JavaScript — the list you paste, the options you choose, and the result are never sent to a server or logged anywhere. Your last input and settings are optionally kept in this browser's local storage only, so they're still there next time you open the tool; clearing your browser data removes them. There's no sign-up, no upload, and no size limit beyond what your own device can comfortably handle.