Overview The Line Sort/Dedupe tool sorts and deduplicates text lines. Supports ascending/descending sort, case-insensitive deduplication with first-seen casing preservation, and combined sort+dedupe operations.
Use Cases
Text Cleanup : Remove duplicate lines from listsData Preparation : Sort lines alphabetically for diff/mergeLog Analysis : Deduplicate and sort log entriesList Management : Clean up email lists, usernames, IDsConfiguration Files : Sort and dedupe config entriesSQL Output : Clean up query results
Plain text with one item per line:
banana apple cherry apple date
user123 user456 USER123 user789 user456
ERROR: Connection failed WARNING: Low memory ERROR: Connection failed INFO: Process started WARNING: Low memory
Actions Sort Ascending (default) Sort lines alphabetically A→Z:
apple apple banana cherry date
Sort Descending Sort lines alphabetically Z→A:
date cherry banana apple apple
Dedupe Remove duplicates, preserve first-seen casing, maintain original order:
Dedupe + Sort Remove duplicates, then sort alphabetically:
Processed text with one item per line:
Metadata:
Deduplication Rules Case Insensitive Comparison ignores case:
→ Treated as same value
First-Seen Casing Preserves casing of first occurrence:
→ Output: Apple (first-seen)
Whitespace Handling Lines are trimmed before comparison:
→ Treated as same value
→ Output: apple (first trimmed version)
Empty Lines Empty lines (after trimming) are ignored:
→ Empty lines removed
Examples Input: Sort Ascending
Output (Sort Ascending)
Input: Sort Descending
Output (Sort Descending)
Input: Dedupe (Preserve Order)
Output (Dedupe)
banana apple cherry apple date banana
Input: Dedupe + Sort
Output (Dedupe + Sort)
banana apple cherry apple date banana
Input: Case-Insensitive Dedupe
Output (Dedupe)
Apple banana APPLE Cherry Banana apple
Input: With Empty Lines
Output (Dedupe + Sort)
apple banana cherry apple
Input: Log Lines
Output (Dedupe + Sort)
ERROR: Connection timeout WARNING: Low disk space ERROR: Connection timeout INFO: Service started WARNING: Low disk space ERROR: Connection timeout
Implementation Details From lib/tools/engine.ts:968-1004:
case 'line-sort' : { if (! input ) return { output : '' , meta : '0 lines | 0 duplicate(s)' }; const lines = input . split ( ' \n ' ); // Build a first-seen Map iteratively const buildFirstSeenMap = ( src : string []) : Map < string , string > => { const seen = new Map < string , string >(); for ( const l of src ) { const key = l . trim (). toLowerCase (); if ( key && ! seen . has ( key )) seen . set ( key , l . trim ()); } return seen ; }; // Apply dedupe option before sorting when explicitly requested const workingLines = options . dedupe === true ? [ ... buildFirstSeenMap (lines). values ()] : lines ; let result : string []; switch ( action ) { case 'sort-asc' : case 'default' : result = [ ... workingLines ]. sort (( a , b ) => a . localeCompare ( b )); break ; case 'sort-desc' : result = [ ... workingLines ]. sort (( a , b ) => b . localeCompare ( a )); break ; case 'dedupe' : result = [ ... buildFirstSeenMap ( lines ). values ()]; break ; case 'dedupe-sort' : result = [ ... buildFirstSeenMap ( lines ). values ()]. sort (( a , b ) => a . localeCompare ( b )); break ; default : result = [ ... workingLines ]. sort (( a , b ) => a . localeCompare ( b )); } const nonEmpty = lines . filter (( l ) => l . trim ()). length ; const uniqueCount = new Set ( lines . map (( l ) => l . trim (). toLowerCase ()). filter ( Boolean )). size ; const dupes = nonEmpty - uniqueCount ; return { output: result . join ( ' \n ' ), meta: ` ${ result . length } lines | ${ dupes } duplicate(s)` }; }
First-Seen Casing :const buildFirstSeenMap = ( src : string []) : Map < string , string > => { const seen = new Map < string , string >(); for ( const l of src ) { const key = l . trim (). toLowerCase (); if ( key && ! seen . has ( key )) seen . set ( key , l . trim ()); } return seen ; };
Normalize key: trim + lowercase
Only store first occurrence
Preserve original casing in value
Sorting :lines . sort (( a , b ) => a . localeCompare ( b )) // Ascending lines . sort (( a , b ) => b . localeCompare ( a )) // Descending
Uses localeCompare for proper Unicode sorting
Supports accented characters and non-English alphabets
Edge Cases Single Line → Output: apple
→ Meta: 1 lines | 0 duplicate(s)
All Duplicates → Output: apple
→ Meta: 1 lines | 2 duplicate(s)
→ Output: (empty)
→ Meta: 0 lines | 0 duplicate(s)
Only Whitespace → Output: (empty)
→ Meta: 0 lines | 0 duplicate(s)
Lines : Handles up to 100K lines efficientlyAlgorithm : O(n log n) for sorting, O(n) for deduplicationMemory : Stores all lines in memory (approx 1MB per 10K lines)
sort + uniq = Line Sort/Dedupe with “Dedupe + Sort” action
sort -u = Line Sort/Dedupe with “Dedupe + Sort” action
sort -r = Line Sort/Dedupe with “Sort Descending” action
The Line Sort/Dedupe tool uses first-seen casing preservation for deduplication, which differs from Unix uniq (which requires sorted input and preserves last-seen casing).
For maximum duplicate removal, use the “Dedupe + Sort” action. For preserving original line order, use the “Dedupe” action.
Very large inputs (over 1 million lines) may cause performance issues or browser memory limits. For massive datasets, use command-line tools like sort and uniq instead.
