I needed to write a script to enter multi-line input to a program (psql
).
After a bit of googling, I found the following syntax works:
cat << EOF | psql ---params
BEGIN;
`pg_dump ----something`
update table .... statement ...;
END;
EOF
This correctly constructs the multi-line string (from BEGIN;
to END;
, inclusive) and pipes it as an input to psql
.
But I have no idea how/why it works, can some one please explain?
I'm referring mainly to cat << EOF
, I know >
outputs to a file, >>
appends to a file, <
reads input from file.
What does <<
exactly do?
And is there a man page for it?
cat
. Try psql ... << EOF ...
See also "here strings". mywiki.wooledge.org/BashGuide/InputAndOutput?#Here_Strings
cat
reads stding(when piped to it) or reads a file that corresponds to it's command line args
The cat <<EOF
syntax is very useful when working with multi-line text in Bash, eg. when assigning multi-line string to a shell variable, file or a pipe.
Examples of cat <
1. Assign multi-line string to a shell variable
The
2. Pass multi-line string to a file in Bash
The
3. Pass multi-line string to a pipe in Bash
The $ sql=$(cat <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
EOF
)
$sql
variable now holds the new-line characters too. You can verify with echo -e "$sql"
.
$ cat <<EOF > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
EOF
print.sh
file now contains:
#!/bin/bash
echo $PWD
echo /home/user
$ cat <<EOF | grep 'b' | tee b.txt
foo
bar
baz
EOF
b.txt
file contains bar
and baz
lines. The same output is printed to stdout
.
This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.
From man bash
:
Here Documents This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input for a command. The format of here-documents is: <<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \
<<-
please note that only leading tab characters are stripped -- not soft tab characters. This is one of those rare case when you actually need the tab character. If the rest of your document uses soft tabs, make sure to show invisible characters and (e.g.) copy and paste a tab character. If you do it right, your syntax highlighting should correctly catch the ending delimiter.
In your case, "EOF" is known as a "Here Tag". Basically <<Here
tells the shell that you are going to enter a multiline string until the "tag" Here
. You can name this tag as you want, it's often EOF
or STOP
.
Some rules about the Here tags:
The tag can be any string, uppercase or lowercase, though most people use uppercase by convention. The tag will not be considered as a Here tag if there are other words in that line. In this case, it will merely be considered part of the string. The tag should be by itself on a separate line, to be considered a tag. The tag should have no leading or trailing spaces in that line to be considered a tag. Otherwise it will be considered as part of the string.
example:
$ cat >> test <<HERE
> Hello world HERE <-- Not by itself on a separate line -> not considered end of string
> This is a test
> HERE <-- Leading space, so not considered end of string
> and a new line
> HERE <-- Now we have the end of the string
<<-
is used leading tab will not prevent the tag from being recognized
POSIX 7
kennytm quoted man bash
, but most of that is also POSIX 7: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 :
The redirection operators "<<" and "<<-" both allow redirection of lines contained in a shell input file, known as a "here-document", to the input of a command. The here-document shall be treated as a single word that begins after the next
Examples
Some examples not yet given.
Quotes prevent parameter expansion
Without quotes:
a=0
cat <<EOF
$a
EOF
Output:
0
With quotes:
a=0
cat <<'EOF'
$a
EOF
or (ugly but valid):
a=0
cat <<E"O"F
$a
EOF
Outputs:
$a
Hyphen removes leading tabs
Without hyphen:
cat <<EOF
<tab>a
EOF
where <tab>
is a literal tab, and can be inserted with Ctrl + V <tab>
Output:
<tab>a
With hyphen:
cat <<-EOF
<tab>a
<tab>EOF
Output:
a
This exists of course so that you can indent your cat
like the surrounding code, which is easier to read and maintain. E.g.:
if true; then
cat <<-EOF
a
EOF
fi
Unfortunately, this does not work for space characters: POSIX favored tab
indentation here. Yikes.
<<-
and <tab>a
, it should be noted that the purpose was to allow normal indentation of code within the script while allowing heredoc text presented to the receiving process to begin in column 0. It is a not too commonly seen feature and a bit more context may prevent a good deal of head-scratching...
$
"$a"'$b'"$c"
, but there is no analogue here AFAIK.
Using tee instead of cat
Not exactly as an answer to the original question, but I wanted to share this anyway: I had the need to create a config file in a directory that required root rights.
The following does not work for that case:
$ sudo cat <<EOF >/etc/somedir/foo.conf
# my config file
foo=bar
EOF
because the redirection is handled outside of the sudo context.
I ended up using this instead:
$ sudo tee <<EOF /etc/somedir/foo.conf >/dev/null
# my config file
foo=bar
EOF
A little extension to the above answers. The trailing >
directs the input into the file, overwriting existing content. However, one particularly convenient use is the double arrow >>
that appends, adding your new content to the end of the file, as in:
cat <<EOF >> /etc/fstab
data_server:/var/sharedServer/authority/cert /var/sharedFolder/sometin/authority/cert nfs
data_server:/var/sharedServer/cert /var/sharedFolder/sometin/vsdc/cert nfs
EOF
This extends your fstab
without you having to worry about accidentally modifying any of its contents.
<< EoF
basically means:
<<
- "read the multi-line input that begins from the next line onward, and treat it as if it's code in a separate file"
EoF
- "stop reading immediately after the word EoF
is found in the multi-line input"
As other answers have explained, the multi-line input is called a Here Document
A Here Document is often used to generate output to be passed to a subsequent process. For example cat << EoF
can be used to generate a desired output, using a Here Document.
Here's an example of using a Here Document to create a text document on the fly:
cat << EoF > ./my-document.txt
Hello world
Have a nice day
EoF
Long story short, EOF
marker(but a different literal can be used as well) is a heredoc format that allows you to provide your input as multiline. A lot of confusion comes from how cat
actually works it seems. You can use cat
with >>
or >
as follows:
$ cat >> temp.txt
line 1
line 2
While cat
can be used this way when writing manually into console, it's not convenient if I want to provide the input in a more declarative way so that it can be reused by tools and also to keep indentations, whitespaces, etc.
Heredoc allows to define your entire input as if you are not working with stdin
but typing in a separate text editor. This is what Wikipedia article means by:
it is a section of a source code file that is treated as if it were a separate file.
This isn't necessarily an answer to the original question, but a sharing of some results from my own testing. This:
<<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test
will produce the same file as:
cat <<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test
So, I don't see the point of using the cat command.
<<test
without cat <<test
.
note to mention that cat << \EOT (see the backslash) will not expand any variables inside, while cat << EOT will do.
examples:
FOO="bar"
cat << \EOT > foobar.txt
echo "$FOO"
EOT
will output: echo $FOO
while:
FOO="bar"
cat << EOT > foobar.txt
echo "$FOO"
EOT
will output: echo "bar"
Example to create a json file:
cat << EoF > ./allaccess.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
EoF
As a result:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
Worth noting that here docs work in bash loops too. This example shows how-to get the column list of table:
export postgres_db_name='my_db'
export table_name='my_table_name'
# start copy
while read -r c; do test -z "$c" || echo $table_name.$c , ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
# stop copy , now paste straight into the bash shell ...
output:
my_table_name.guid ,
my_table_name.id ,
my_table_name.level ,
my_table_name.seq ,
or even without the new line
while read -r c; do test -z "$c" || echo $table_name.$c , | perl -ne
's/\n//gm;print' ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
# output: daily_issues.guid ,daily_issues.id ,daily_issues.level ,daily_issues.seq ,daily_issues.prio ,daily_issues.weight ,daily_issues.status ,daily_issues.category ,daily_issues.name ,daily_issues.description ,daily_issues.type ,daily_issues.owner
Success story sharing
tee
instead ofcat
,sudo
can be used to write to a file at a restricted location. Like e.g.sudo tee /etc/somepath/file > /dev/null <<EOF ...
cat
is only for complete bash command with here-document, if withoutcat
or other commands, the here-document started with symbol<<
can not echo to thestdout
, and the rest of the first line "grep 'b' | tee b.txt" could not get input.