Invisible characters and where they come from
| Code point | Name | Usually comes from |
|---|---|---|
U+00A0 |
No-break space | Word, Google Docs, HTML , macOS Option+Space |
U+200B |
Zero-width space | Web pages, Slack, chat apps, AI-generated text |
U+FEFF |
Byte order mark / zero-width no-break space | Start of files saved as “UTF-8 with BOM” by Windows tools and Excel |
U+00AD |
Soft hyphen | Hyphenation hints from web pages and word processors |
U+202F |
Narrow no-break space | French punctuation spacing; macOS and iOS date and time formats |
U+3000 |
Ideographic space | Chinese and Japanese input methods |
Cleaning whitespace in code
// \s in JavaScript already matches U+00A0, U+FEFF and the other Unicode spaces
const collapsed = text.replace(/[^\S\n]+/g, ' '); // collapse, keep line breaks
const trimmed = text.split('\n').map((l) => l.trim()).join('\n');
// \s does NOT match zero-width spaces or soft hyphens; remove them explicitly
const visible = text.replace(/[\u200B\u2060\u00AD\uFEFF]/g, '');
import re
collapsed = re.sub(r"[^\S\n]+", " ", text) # str patterns are Unicode-aware
no_zero_width = re.sub("[\u200b\u2060\u00ad\ufeff]", "", text)
# Read a CSV saved by Excel without the BOM ending up in the first header
open("data.csv", encoding="utf-8-sig")
sed -E 's/[[:space:]]+$//' file.txt # trailing whitespace
tr -s ' ' < file.txt # squeeze repeated spaces
grep -nP '[\x{00A0}\x{200B}\x{FEFF}]' file.txt # find invisible characters (GNU grep)
Removing blank lines instead? Use remove empty lines. Want to see how long the text is after cleaning? The word counter shows characters, bytes and words.
Frequently asked questions
Why does my code or CSV import break on text that looks fine?
Often because of a character you can’t see. A no-break space (U+00A0) looks like a space but isn’t matched by " " or a split on spaces; a zero-width space (U+200B) makes two strings that look identical compare unequal; a byte order mark (U+FEFF) at the start of a CSV turns the first column name into "\uFEFFid". Paste the text here and the table under the tool lists each one with its line.
What does “Fix invisible characters” do?
Every kind of space that isn’t a normal space (no-break, narrow, thin, em, ideographic…) becomes a normal space, and zero-width spaces, word joiners, byte order marks, soft hyphens and direction marks are deleted. Zero-width joiners are kept, because they hold emoji like 👩💻 together and are needed in scripts like Arabic and Hindi.
Will collapsing spaces break my code indentation?
Not if you untick “Trim lines”: the leading indentation of each line is then kept, and only runs of spaces inside the line are collapsed. Trailing spaces at the end of lines are always removed. Don’t collapse spaces in Python or YAML if alignment inside lines matters, or in Markdown, where two trailing spaces mean a line break.
How do I remove all whitespace, including line breaks?
Tick “Remove all whitespace”. Every space, tab, line break and other Unicode whitespace is deleted, which is handy for cleaning up a hex string, a Base64 blob or a phone number split over several lines.