What is the simplest way to remove all the carriage returns \r
from a file in Unix?
tr -d
command is the simplest method, but I am wondering how to remove just the last carriage return?
I'm going to assume you mean carriage returns (CR, "\r"
, 0x0d
) at the ends of lines rather than just blindly within a file (you may have them in the middle of strings for all I know). Using this test file with a CR at the end of the first line only:
$ cat infile
hello
goodbye
$ cat infile | od -c
0000000 h e l l o \r \n g o o d b y e \n
0000017
dos2unix
is the way to go if it's installed on your system:
$ cat infile | dos2unix -U | od -c
0000000 h e l l o \n g o o d b y e \n
0000016
If for some reason dos2unix
is not available to you, then sed
will do it:
$ cat infile | sed 's/\r$//' | od -c
0000000 h e l l o \n g o o d b y e \n
0000016
If for some reason sed
is not available to you, then ed
will do it, in a complicated way:
$ echo ',s/\r\n/\n/
> w !cat
> Q' | ed infile 2>/dev/null | od -c
0000000 h e l l o \n g o o d b y e \n
0000016
If you don't have any of those tools installed on your box, you've got bigger problems than trying to convert files :-)
tr -d '\r' < infile > outfile
See tr(1)
tr
does not support the \r
escape, try '\015'
or perhaps a literal '^M'
(in many shells on many terminals, ctrl-V ctrl-M will produce a literal ctrl-M character).
outfile = infile
?
someProg <in >out && mv out in
.
The simplest way on Linux is, in my humble opinion,
sed -i.bak 's/\r$//g' <filename>
-i will edit the file in place, while the .bak will create a backup of the original file by making a copy of your file and adding the extension .bak at the end. (You can specify what ever you want after the -i
, or specify only -i
to not create a backup.)
The strong quotes around the substitution operator 's/\r//'
are essential. Without them the shell will interpret \r
as an escape+r and reduce it to a plain r
, and remove all lower case r
. That's why the answer given above in 2009 by Rob doesn't work.
And adding the /g
modifier ensures that even multiple \r
will be removed, and not only the first one.
-i
flag since it modifies the original file, and it might be dangerous if you wish to keep it unchanged
Old School:
tr -d '\r' < filewithcarriagereturns > filewithoutcarriagereturns
There's a utility called dos2unix that exists on many systems, and can be easily installed on most.
sed -i s/\r// <filename>
or somesuch; see man sed
or the wealth of information available on the web regarding use of sed
.
One thing to point out is the precise meaning of "carriage return" in the above; if you truly mean the single control character "carriage return", then the pattern above is correct. If you meant, more generally, CRLF (carriage return and a line feed, which is how line feeds are implemented under Windows), then you probably want to replace \r\n
instead. Bare line feeds (newline) in Linux/Unix are \n
.
s/\r//
doesn't seem to remove carriage returns on OS X, it seems to remove literal r
chars instead. I'm not sure why that is yet. Maybe it has something to do with the way the string is quoted? As a workaround, using CTRL-V + CTRL-M
in place of \r
seems to work.
If you are a Vi user, you may open the file and remove the carriage return with:
:%s/\r//g
or with
:1,$ s/^M//
Note that you should type ^M by pressing ctrl-v and then ctrl-m.
^M
-s at all. Getting around this is a ton of keystrokes, which is not what vim is made for ;). I'd just go for sed -i
, and then `-e 's/\r$//g' to limit the removal to CRs at EOL.
Someone else recommend dos2unix
and I strongly recommend it as well. I'm just providing more details.
If installed, jump to the next step. If not already installed, I would recommend installing it via yum
like:
yum install dos2unix
Then you can use it like:
dos2unix fileIWantToRemoveWindowsReturnsFrom.txt
Once more a solution... Because there's always one more:
perl -i -pe 's/\r//' filename
It's nice because it's in place and works in every flavor of unix/linux I've worked with.
-i.bak
if you want a backup of your original file.
Removing \r on any UNIX® system:
Most existing solutions in this question are GNU-specific, and wouldn't work on OS X or BSD; the solutions below should work on many more UNIX systems, and in any shell, from tcsh
to sh
, yet still work even on GNU/Linux, too.
Tested on OS X, OpenBSD and NetBSD in tcsh
, and on Debian GNU/Linux in bash
.
With sed:
In tcsh
on an OS X, the following sed
snippet could be used together with printf
, as neither sed
nor echo
handle \r
in the special way like the GNU does:
sed `printf 's/\r$//g'` input > output
With tr:
Another option is tr
:
tr -d '\r' < input > output
Difference between sed and tr:
It would appear that tr
preserves a lack of a trailing newline from the input file, whereas sed
on OS X and NetBSD (but not on OpenBSD or GNU/Linux) inserts a trailing newline at the very end of the file even if the input is missing any trailing \r
or \n
at the very end of the file.
Testing:
Here's some sample testing that could be used to ensure this works on your system, using printf
and hexdump -C
; alternatively, od -c
could also be used if your system is missing hexdump
:
% printf 'a\r\nb\r\nc' | hexdump -C
00000000 61 0d 0a 62 0d 0a 63 |a..b..c|
00000007
% printf 'a\r\nb\r\nc' | ( sed `printf 's/\r$//g'` /dev/stdin > /dev/stdout ) | hexdump -C
00000000 61 0a 62 0a 63 0a |a.b.c.|
00000006
% printf 'a\r\nb\r\nc' | ( tr -d '\r' < /dev/stdin > /dev/stdout ) | hexdump -C
00000000 61 0a 62 0a 63 |a.b.c|
00000005
%
If you're using an OS (like OS X) that doesn't have the dos2unix
command but does have a Python interpreter (version 2.5+), this command is equivalent to the dos2unix
command:
python -c "import sys; import fileinput; sys.stdout.writelines(line.replace('\r', '\n') for line in fileinput.input(mode='rU'))"
This handles both named files on the command line as well as pipes and redirects, just like dos2unix
. If you add this line to your ~/.bashrc file (or equivalent profile file for other shells):
alias dos2unix="python -c \"import sys; import fileinput; sys.stdout.writelines(line.replace('\r', '\n') for line in fileinput.input(mode='rU'))\""
... the next time you log in (or run source ~/.bashrc
in the current session) you will be able to use the dos2unix
name on the command line in the same manner as in the other examples.
you can simply do this :
$ echo $(cat input) > output
a * b
...
Here is the thing,
%0d
is the carriage return character. To make it compatabile with Unix. We need to use the below command.
dos2unix fileName.extension fileName.extension
try this to convert dos file into unix file:
fromdos file
For UNIX... I've noticed dos2unix removed Unicode headers form my UTF-8 file. Under git bash (Windows), the following script seems to work nicely. It uses sed. Note it only removes carriage-returns at the ends of lines, and preserves Unicode headers.
#!/bin/bash
inOutFile="$1"
backupFile="${inOutFile}~"
mv --verbose "$inOutFile" "$backupFile"
sed -e 's/\015$//g' <"$backupFile" >"$inOutFile"
If you are running an X environment and have a proper editor (visual studio code), then I would follow the reccomendation:
Visual Studio Code: How to show line endings
Just go to the bottom right corner of your screen, visual studio code will show you both the file encoding and the end of line convention followed by the file, an just with a simple click you can switch that around.
Just use visual code as your replacement for notepad++ on a linux environment and you are set to go.
Notepad++
's command to Edit / EOL Conversion / Unix (LF)
on your Windows system before copying the file to your Linux system.
Using sed
sed $'s/\r//' infile > outfile
Using sed
on Git Bash for Windows
sed '' infile > outfile
The first version uses ANSI-C quoting and may require escaping \
if the command runs from a script. The second version exploits the fact that sed
reads the input file line by line by removing \r
and \n
characters. When writing lines to the output file, however, it only appends a \n
character. A more general and cross-platform solution can be devised by simply modifying IFS
IFS=$'\r\n' # or IFS+=$'\r' if the lines do not contain whitespace
printf "%s\n" $(cat infile) > outfile
IFS=$' \t\n' # not necessary if IFS+=$'\r' is used
Warning: This solution performs filename expansion (*
, ?
, [...]
and more if extglob
is set). Use it only if you are sure that the file does not contain special characters or you want the expansion.
Warning: None of the solutions can handle \
in the input file.
cat input.csv | sed 's/\r/\n/g' > output.csv
worked for me
I've used python for it, here my code;
end1='/home/.../file1.txt'
end2='/home/.../file2.txt'
with open(end1, "rb") as inf:
with open(end2, "w") as fixed:
for line in inf:
line = line.replace("\n", "")
line = line.replace("\r", "")
fixed.write(line)
Though it's a older post, recently I came across with same problem. As I had all the files to rename inside /tmp/blah_dir/ as each file in this directory had "/r" trailing character ( showing "?" at end of file), so doing it script way was only I could think of.
I wanted to save final file with same name (without trailing any character). With sed, problem was the output filename which I was needed to mention something else ( which I didn't want).
I tried other options as suggested here (not considered dos2unix because of some limitations) but didn't work.
I tried with "awk" finally which worked where I used "\r" as delimiter and taken the first part:
trick is:
echo ${filename}|awk -F"\r" '{print $1}'
Below script snippet I used ( where I had all file had "\r" as trailing character at path /tmp/blah_dir/) to fix my issue:
cd /tmp/blah_dir/
for i in `ls`
do
mv $i $(echo $i | awk -F"\r" '{print $1}')
done
Note: This example is not very exact though close to what I worked (Mentioning here just to give the better idea about what I did)
I made this shell-script to remove the \r character. It works in solaris and red-hat:
#!/bin/ksh
LOCALPATH=/Any_PATH
for File in `ls ${LOCALPATH}`
do
ARCACT=${LOCALPATH}/${File}
od -bc ${ARCACT}|sed -n 'p;n'|sed 's/015/012/g'|awk '{$1=""; print $0}'|sed 's/ /\\/g'|awk '{printf $0;}'>${ARCACT}.TMP
printf "`cat ${ARCACT}.TMP`"|sed '/^$/d'>${ARCACT}
rm ${ARCACT}.TMP
done
exit 0
Success story sharing
\r
works only with GNU sed, else you can do this:sed `echo "s/\r//"`
sed
norecho
recognise\r
on MacOs. In this case onlyprintf "\r"
appears to work.sed "s/$(printf '\r')\$//"
$
like so:sed $'s@\r@@g' |od -c
(but if you would replace with\n
you would need to escape it)CTRL-V + CTRL-M
in place of\r
looks like it might work.