How to remove non UTF-8 characters from text file
I have a bunch of Arabic, English, Russian files which are encoded in utf-8. Trying to process these files using a Perl script, I get this error:
Malformed UTF-8 character (fatal)
Manually checking the content of these files, I found some strange characters in them. Now I'm looking for a way to automatically remove these characters from the files.
Is there anyway to do it?
This command:
iconv -f utf-8 -t utf-8 -c file.txt
will clean up your UTF-8 file, skipping all the invalid characters.
-f is the source format
-t the target format
-c skips any invalid sequence
Your method must read byte by byte and fully understand and appreciate the byte wise construction of characters. The simplest method is to use an editor which will read anything but only output UTF-8 characters. Textpad is one choice.
None of the methods here or on any other similar questions worked for me. In the end what worked was simply opening the file in Sublime Text 2. Go to File > Reopen with Encoding > UTF-8. Copy the entire content of the file into a new file and save it.
May not be the expected solution but putting this out here in case it helps anyone, since I've been struggling for hours with this.
cat foo.txt | strings -n 8 > bar.txt
will do the job.
Follow WeChat
Success story sharing
Want to stay one step ahead of the latest teleworks?
Subscribe Now相似问题
- How to redirect output to a file and stdout
- How can I check if a program exists from a Bash script?
- How do I tell if a regular file does not exist in Bash?
- How to redirect and append both standard output and standard error to a file with Bash
- How do I remove  from the beginning of a file?
- Print string to text file
- How to remove spaces from a string using JavaScript?
- How do I find all files containing specific text on Linux?
pbpaste | iconv -f utf-8 -t -utf-8 -c | pbcopy. I also created an Alfred workflow with a global shortcut for stripping all special characters by targetingascii.iconv -f utf-8 -t ascii//TRANSLITsolved my problem. It converts curly quotes to straight quotes.-ofor different output file