I need to do a regex find and replace on all the files in a folder (and its subfolders). What would be the linux shell command to do that?
For example, I want to run this over all the files and overwrite the old file with the new, replaced text.
sed 's/old text/new text/g'
There is no way to do it using only sed. You'll need to use at least the find utility together:
find . -type f -exec sed -i.bak "s/foo/bar/g" {} \;
This command will create a .bak
file for each changed file.
Notes:
The -i argument for sed command is a GNU extension, so, if you are running this command with the BSD's sed you will need to redirect the output to a new file then rename it.
The find utility does not implement the -exec argument in old UNIX boxes, so, you will need to use a | xargs instead.
I prefer to use find | xargs cmd
over find -exec
because it's easier to remember.
This example globally replaces "foo" with "bar" in .txt files at or below your current directory:
find . -type f -name "*.txt" -print0 | xargs -0 sed -i "s/foo/bar/g"
The -print0
and -0
options can be left out if your filenames do not contain funky characters such as spaces.
find . -type f -name "*.txt" -print0 | xargs -0 sed -i '' "s/foo/bar/g"
(note providing an empty string to the -i
argument).
sed -i.bak
instead of sed -i
. I think as mentioned by @JakubKukul , sed -i ''
also works.
For portability, I don't rely on features of sed that are specific to linux or BSD. Instead I use the overwrite
script from Kernighan and Pike's book on the Unix Programming Environment.
The command is then
find /the/folder -type f -exec overwrite '{}' sed 's/old/new/g' {} ';'
And the overwrite
script (which I use all over the place) is
#!/bin/sh
# overwrite: copy standard input to output after EOF
# (final version)
# set -x
case $# in
0|1) echo 'Usage: overwrite file cmd [args]' 1>&2; exit 2
esac
file=$1; shift
new=/tmp/$$.new; old=/tmp/$$.old
trap 'rm -f $new; exit 1' 1 2 15 # clean up files
if "$@" >$new # collect input
then
cp $file $old # save original file
trap 'trap "" 1 2 15; cp $old $file # ignore signals
rm -f $new $old; exit 1' 1 2 15 # during restore
cp $new $file
else
echo "overwrite: $1 failed, $file unchanged" 1>&2
exit 1
fi
rm -f $new $old
The idea is that it overwrites a file only if a command succeeds. Useful in find
and also where you would not want to use
sed 's/old/new/g' file > file # THIS CODE DOES NOT WORK
because the shell truncates the file before sed
can read it.
Might I suggest (after backing up your files):
find /the/folder -type f -exec sed -ibak 's/old/new/g' {} ';'
Example: inline replace {AutoStart} with 1 for all of the ini files under the /app/config/ folder and its child folders:
sed -i 's/{AutoStart}/1/g' /app/config/**/*.ini
sed -i 's/{AutoStart}/1/g' /app/config/**/*.ini
This worked for me (on mac terminal, on Linux you don't need '' -e
):
sed -i '' -e 's/old text/new text/g' `grep 'old text' -rl *`
the command grep 'old text' -rl *
lists all files in the working directory (and subdirectories) where "old text" exists. This then is passed in sed.
If you are worried about clobbering files that you accidentally haven't considered, you can run grep
with the recursive option first to see which files will be potentially changed:
grep -r 'searchstring' *
It's then not hard to put together something that will run the replacement on each of these files:
for f in $(grep -r 'searchstring' *)
do
sed -i -e 's/searchstring/replacement/g' "$f"
done
(Only valid with GNU sed - adjustments would be needed for POSIX compatibility as the -i
inplace option is a GNU extension):
Might want to try my mass search/replace Perl script. Has some advantages over chained-utility solutions (like not having to deal with multiple levels of shell metacharacter interpretation).
#!/usr/bin/perl
use strict;
use Fcntl qw( :DEFAULT :flock :seek );
use File::Spec;
use IO::Handle;
die "Usage: $0 startdir search replace\n"
unless scalar @ARGV == 3;
my $startdir = shift @ARGV || '.';
my $search = shift @ARGV or
die "Search parameter cannot be empty.\n";
my $replace = shift @ARGV;
$search = qr/\Q$search\E/o;
my @stack;
sub process_file($) {
my $file = shift;
my $fh = new IO::Handle;
sysopen $fh, $file, O_RDONLY or
die "Cannot read $file: $!\n";
my $found;
while(my $line = <$fh>) {
if($line =~ /$search/) {
$found = 1;
last;
}
}
if($found) {
print " Processing in $file\n";
seek $fh, 0, SEEK_SET;
my @file = <$fh>;
foreach my $line (@file) {
$line =~ s/$search/$replace/g;
}
close $fh;
sysopen $fh, $file, O_WRONLY | O_TRUNC or
die "Cannot write $file: $!\n";
print $fh @file;
}
close $fh;
}
sub process_dir($) {
my $dir = shift;
my $dh = new IO::Handle;
print "Entering $dir\n";
opendir $dh, $dir or
die "Cannot open $dir: $!\n";
while(defined(my $cont = readdir($dh))) {
next
if $cont eq '.' || $cont eq '..';
# Skip .swap files
next
if $cont =~ /^\.swap\./o;
my $fullpath = File::Spec->catfile($dir, $cont);
if($cont =~ /$search/) {
my $newcont = $cont;
$newcont =~ s/$search/$replace/g;
print " Renaming $cont to $newcont\n";
rename $fullpath, File::Spec->catfile($dir, $newcont);
$cont = $newcont;
$fullpath = File::Spec->catfile($dir, $cont);
}
if(-l $fullpath) {
my $link = readlink($fullpath);
if($link =~ /$search/) {
my $newlink = $link;
$newlink =~ s/$search/$replace/g;
print " Relinking $cont from $link to $newlink\n";
unlink $fullpath;
my $res = symlink($newlink, $fullpath);
warn "Symlink of $newlink to $fullpath failed\n"
unless $res;
}
}
next
unless -r $fullpath && -w $fullpath;
if(-d $fullpath) {
push @stack, $fullpath;
} elsif(-f $fullpath) {
process_file($fullpath);
}
}
closedir($dh);
}
if(-f $startdir) {
process_file($startdir);
} elsif(-d $startdir) {
@stack = ($startdir);
while(scalar(@stack)) {
process_dir(shift(@stack));
}
} else {
die "$startdir is not a file or directory\n";
}
for i in $(ls);do sed -i 's/old_text/new_text/g' $i;done
In case the name of files in folder has some regular names (like file1, file2...) I have used for cycle.
for i in {1..10000..100}; do sed 'old\new\g' 'file'$i.xml > 'cfile'$i.xml; done
Success story sharing
\;
for?-i
by itself does not create a backup file, and is what causes sed to perform the operation on the file in place.{}
for ?{}
will be replaced by each filename found byfind
and\;
tells to find that the command that he needs to execute finish at this point.