ChatGPT解决这个技术问题 Extra ChatGPT

How to loop over files in directory and change path and add suffix to filename

I need to write a script that starts my program with different arguments, but I'm new to Bash. I start my program with:

./MyProgram.exe Data/data1.txt [Logs/data1_Log.txt].

Here is the pseudocode for what I want to do:

for each filename in /Data do
  for int i = 0, i = 3, i++
    ./MyProgram.exe Data/filename.txt Logs/filename_Log{i}.txt
  end for
end for

So I'm really puzzled how to create second argument from the first one, so it looks like dataABCD_Log1.txt and start my program.

@LéaGris The proposed duplicate seems less stellar, especially as one of the answers there still advocates looping over ls output. These seem different enough that I have not nominated that as a duplicate of this, either.

J
Jack G

A couple of notes first: when you use Data/data1.txt as an argument, should it really be /Data/data1.txt (with a leading slash)? Also, should the outer loop scan only for .txt files, or all files in /Data? Here's an answer, assuming /Data/data1.txt and .txt files only:

#!/bin/bash
for filename in /Data/*.txt; do
    for ((i=0; i<=3; i++)); do
        ./MyProgram.exe "$filename" "Logs/$(basename "$filename" .txt)_Log$i.txt"
    done
done

Notes:

/Data/*.txt expands to the paths of the text files in /Data (including the /Data/ part)

$( ... ) runs a shell command and inserts its output at that point in the command line

basename somepath .txt outputs the base part of somepath, with .txt removed from the end (e.g. /Data/file.txt -> file)

If you needed to run MyProgram with Data/file.txt instead of /Data/file.txt, use "${filename#/}" to remove the leading slash. On the other hand, if it's really Data not /Data you want to scan, just use for filename in Data/*.txt.


If no files are found/match the wildcard I'm finding the for loops execute block is still entered once with filename = "/Data/*.txt". How can I avoid this?
@OliverPearmain Either use shopt -s nullglob before the loop (and shopt -u nullglob after to avoid problems later on), or add if [[ ! -e "$filename ]]; then continue; fi at the beginning of the loop, so it'll skip nonexistent files.
Perfect shopt -s nullglob works great. Thanks for the info and prompt response.
This doesn't work when there are files which contain whitespace in their name.
@Isa It should work with whitespace, as long as all of the double-quotes are in place. Leave any of the double-quotes out, and you'll have problems with whitespace.
J
Joshua Goldberg

Sorry for necromancing the thread, but whenever you iterate over files by globbing, it's good practice to avoid the corner case where the glob does not match (which makes the loop variable expand to the (un-matching) glob pattern string itself).

For example:

for filename in Data/*.txt; do
    [ -e "$filename" ] || continue
    # ... rest of the loop body
done

Reference: Bash Pitfalls


This is still a timely warning. I thought I had created my script incorrectly, but I had my file extension lower case instead of upper case, it found no files, and returned the glob pattern. ugh.
Since the Bash tag is used: that's what shopt nullglob is for! (or shopt failglob can be used too, depending on the behavior you want).
Besides, this also checks for files deleted during processing, before the loop reaches them.
Also handles the case where you have a directory called dir.txt
Don't be sorry; the whole purpose of Stack Overflow is to collect and curate canonical questions rather than have new users reask the same old questions.
J
Jonathan Leffler
for file in Data/*.txt
do
    for ((i = 0; i < 3; i++))
    do
        name=${file##*/}
        base=${name%.txt}
        ./MyProgram.exe "$file" Logs/"${base}_Log$i.txt"
    done
done

The name=${file##*/} substitution (shell parameter expansion) removes the leading pathname up to the last /.

The base=${name%.txt} substitution removes the trailing .txt. It's a bit trickier if the extensions can vary.


I believe there's an error in your code. The one line should be base=${name%.txt}, instead of base=${base%.txt}.
@CaseyKlimkowsky: Yes; when the code and the comments disagree, at least one of them is wrong. In this case, I think it is only the one — the code; often, it is actually both that are wrong. Thanks for pointing that out; I've fixed it.
Rather than name=${file##*/} you can use name=`basename $file`
@Tk421 — Sorta. You should use $(…) instead of back ticks `…`. And using basename runs a process whereas the ${file##*/} notation is run by the shell without an external process. Functionally, the two are equivalent except perhaps in extreme edge cases. But the shell variable expansion should be more efficient ff, and possibly even measurably more efficient.
J
James Andino

You can use finds null separated output option with read to iterate over directory structures safely.

#!/bin/bash
find . -type f -print0 | while IFS= read -r -d $'\0' file; 
  do echo "$file" ;
done

So for your case

#!/bin/bash
find . -maxdepth 1 -type f  -print0 | while IFS= read -r -d $'\0' file; do
  for ((i=0; i<=3; i++)); do
    ./MyProgram.exe "$file" 'Logs/'"`basename "$file"`""$i"'.txt'
  done
done

additionally

#!/bin/bash
while IFS= read -r -d $'\0' file; do
  for ((i=0; i<=3; i++)); do
    ./MyProgram.exe "$file" 'Logs/'"`basename "$file"`""$i"'.txt'
  done
done < <(find . -maxdepth 1 -type f  -print0)

will run the while loop in the current scope of the script ( process ) and allow the output of find to be used in setting variables if needed


$'\0' is a weird way of writing ''. You're missing IFS= and the -r switch to read: your read statement should be: IFS= read -rd '' file.
I figure some would need search $'\0' and spread some stack points around. Going to make the edits you pointed out. What are the ill effects of not having IFS= trying echo -e "ok \nok\0" | while read -d '' line; do echo -e "$line"; done there seem not be any. Also -r I see is often default, but could not find an example for what it prevents happening.
IFS= is needed in case a filename ends with a space: try is with touch 'Prospero ' (note the trailing space). Also you need the -r switch in case a file name has a backslash: try it with touch 'Prospero\n'.
Underrated answer. Try the most upvoted ones on a directory containing more than 100k files. Good luck if you're on a low-end machine.
S
Sebastian Korotkiewicz

Run a command on each file

do something (echo) with all .txt files

for f in *.txt;  do echo ${f}; done;

u
user3183111

Looks like you're trying to execute a windows file (.exe) Surely you ought to be using powershell. Anyway on a Linux bash shell a simple one-liner will suffice.

[/home/$] for filename in /Data/*.txt; do for i in {0..3}; do ./MyProgam.exe  Data/filenameLogs/$filename_log$i.txt; done done

Or in a bash

#!/bin/bash

for filename in /Data/*.txt; 
   do
     for i in {0..3}; 
       do ./MyProgam.exe Data/filename.txt Logs/$filename_log$i.txt; 
     done 
 done