I have a text file with the following format. The first line is the "KEY" and the second line is the "VALUE".
KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:1349 string
1
KEY 7329:2407 string
2
KEY 0:1774 string
1
I need the value in the same line as of the key. So the output should look like this...
KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1
It will be better if I could use some delimiter like $
or ,
:
KEY 4048:1736 string , 3
How do I merge two lines into one?
pr
, paste
, awk
, xargs
, sed
and pure bash
! (xargs
is the slower, slower than bash!)
paste
is good for this job:
paste -d " " - - < filename
awk:
awk 'NR%2{printf "%s ",$0;next;}1' yourFile
note, there is an empty line at the end of output.
sed:
sed 'N;s/\n/ /' yourFile
Alternative to sed, awk, grep:
xargs -n2 -d'\n'
This is best when you want to join N lines and you only need space delimited output.
My original answer was xargs -n2
which separates on words rather than lines. -d
(GNU xargs option) can be used to split the input by any singular character.
There are more ways to kill a dog than hanging. [1]
awk '{key=$0; getline; print key ", " $0;}'
Put whatever delimiter you like inside the quotes.
References:
Originally "Plenty of ways to skin the cat", reverted to an older, potentially originating expression that also has nothing to do with pets.
Here is my solution in bash:
while read line1; do read line2; echo "$line1, $line2"; done < data.txt
Here is another way with awk
:
awk 'ORS=NR%2?FS:RS' file
$ cat file
KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:1349 string
1
KEY 7329:2407 string
2
KEY 0:1774 string
1
$ awk 'ORS=NR%2?FS:RS' file
KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1
As indicated by Ed Morton in the comments, it is better to add braces for safety and parens for portability.
awk '{ ORS = (NR%2 ? FS : RS) } 1' file
ORS
stands for Output Record Separator. What we are doing here is testing a condition using the NR
which stores the line number. If the modulo of NR
is a true value (>0) then we set the Output Field Separator to the value of FS
(Field Separator) which by default is space, else we assign the value of RS
(Record Separator) which is newline.
If you wish to add ,
as the separator then use the following:
awk '{ ORS = (NR%2 ? "," : RS) } 1' file
ORS
and that's being treated as true
since ORS gets a value thats not zero or a null string and awks guessing correctly that it should be a sting instead of numeric comparison? Is it something else? I'm really not sure and so I'd have written it as awk '{ORS=(NR%2?FS:RS)}1' file
. I parenthesized the ternary expression to ensure portability too.
Although it seems the previous solutions would work, if a single anomaly occurs in the document the output would go to pieces. Below is a bit safer.
sed -n '/KEY/{
N
s/\n/ /p
}' somefile.txt
/KEY/
do? What does the p
do at the end?
/KEY/
searches for the line with the KEY
. the p
prints the result out. it's safer because it only applies the operation on lines with a KEY
in it.
A slight variation on glenn jackman's answer using paste
: if the value for the -d
delimiter option contains more than one character, paste
cycles through the characters one by one, and combined with the -s
options keeps doing that while processing the same input file.
This means that we can use whatever we want to have as the separator plus the escape sequence \n
to merge two lines at a time.
Using a comma:
$ paste -s -d ',\n' infile
KEY 4048:1736 string,3
KEY 0:1772 string,1
KEY 4192:1349 string,1
KEY 7329:2407 string,2
KEY 0:1774 string,1
and the dollar sign:
$ paste -s -d '$\n' infile
KEY 4048:1736 string$3
KEY 0:1772 string$1
KEY 4192:1349 string$1
KEY 7329:2407 string$2
KEY 0:1774 string$1
What this cannot do is use a separator consisting of multiple characters.
As a bonus, if the paste
is POSIX compliant, this won't modify the newline of the last line in the file, so for an input file with an odd number of lines like
KEY 4048:1736 string
3
KEY 0:1772 string
paste
won't tack on the separation character on the last line:
$ paste -s -d ',\n' infile
KEY 4048:1736 string,3
KEY 0:1772 string
"ex" is a scriptable line editor that is in the same family as sed, awk, grep, etc. I think it might be what you are looking for. Many modern vi clone/successors also have a vi mode.
ex -c "%g/KEY/j" -c "wq" data.txt
This says for each line, if it matches "KEY" perform a j oin of the following line. After that command completes (against all lines), issue a w rite and q uit.
You can use awk like this to combine ever 2 pair of lines:
awk '{ if (NR%2 != 0) line=$0; else {printf("%s %s\n", line, $0); line="";} } \
END {if (length(line)) print line;}' flle
Another solutions using vim (just for reference).
Solution 1:
Open file in vim vim filename
, then execute command :% normal Jj
This command is very easy to understand:
% : for all the lines,
normal : execute normal command
Jj : execute Join command, then jump to below line
After that, save the file and exit with :wq
Solution 2:
Execute the command in shell, vim -c ":% normal Jj" filename
, then save the file and exit with :wq
.
norm!
more robust that normal
in case J
been remapped. +1 for vim solution.
If Perl is an option, you can try:
perl -0pe 's/(.*)\n(.*)\n/$1 $2\n/g' file.txt
-0
tell perl to set the record separator ($/)
to null, so that we can span multiple lines in our matching pattern. The manpages are a bit too technical for me to figure out what it means in practice.
You can also use the following vi command:
:%g/.*/j
:%g//j
since all you need is a match for the join to be executed, and a null string is still a valid regex.
//
, the previous search pattern will be used instead. If there is no previous pattern, Vim simply reports an error and do nothing. Jdamian's solution works all the time.
Another approach using vim would be:
:g/KEY/join
This applies a join
(to the line below it) to all lines that have the word KEY
in it. Result:
KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1
nawk '$0 ~ /string$/ {printf "%s ",$0; getline; printf "%s\n", $0}' filename
This reads as
$0 ~ /string$/ ## matches any lines that end with the word string
printf ## so print the first line without newline
getline ## get the next line
printf "%s\n" ## print the whole line and carriage return
In the case where I needed to combine two lines (for easier processing), but allow the data past the specific, I found this to be useful
data.txt
string1=x
string2=y
string3
string4
cat data.txt | nawk '$0 ~ /string1=/ { printf "%s ", $0; getline; printf "%s\n", $0; getline } { print }' > converted_data.txt
output then looks like:
converted_data.txt
string1=x string2=y
string3
string4
cat input.txt
KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:1349 string
1
KEY 7329:2407 string
2
KEY 0:1774 string
1
paste -sd ' \n' input.txt
KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1
paste -sd ' \n' input.txt | rev | sed 's/ / , /' | rev
KEY 4048:1736 string , 3
KEY 0:1772 string , 1
KEY 4192:1349 string , 1
KEY 7329:2407 string , 2
KEY 0:1774 string , 1
Simplest way is here:
Remove even lines and write it in some temp file 1. Remove odd lines and write it in some temp file 2. Combine two files in one by using paste command with -d (means delete space)
sed '0~2d' file > 1 && sed '1~2d' file > 2 && paste -d " " 1 2
perl -0pE 's{^KEY.*?\K\s+(\d+)$}{ $1}msg;' data.txt > data_merged-lines.txt
-0
gobbles the whole file instead of reading it line-by-line;
pE
wraps code with loop and prints the output, see details in http://perldoc.perl.org/perlrun.html;
^KEY
match "KEY" in the beginning of line, followed by non-greedy match of anything (.*?
) before sequence of
one or more spaces \s+ of any kind including line breaks; one or more digit (\d+) which we capture and later re-insert as $1;
followed by the end of line $
.
\K
conveniently excludes everything on its left hand side from substitution so { $1}
replaces only 1-2 sequence, see http://perldoc.perl.org/perlre.html.
A more-general solution (allows for more than one follow-up line to be joined) as a shell script. This adds a line between each, because I needed visibility, but that is easily remedied. This example is where the "key" line ended in : and no other lines did.
#!/bin/bash
#
# join "The rest of the story" when the first line of each story
# matches $PATTERN
# Nice for looking for specific changes in bart output
#
PATTERN='*:';
LINEOUT=""
while read line; do
case $line in
$PATTERN)
echo ""
echo $LINEOUT
LINEOUT="$line"
;;
"")
LINEOUT=""
echo ""
;;
*) LINEOUT="$LINEOUT $line"
;;
esac
done
Try the following line:
while read line1; do read line2; echo "$line1 $line2"; done <old.txt>new_file
Put delimiter in-between
"$line1 $line2";
e.g. if the delimiter is |
, then:
"$line1|$line2";
Success story sharing
paste
takes always one line from each input file and outputs a single line with those lines combined by the separator character defined with-d
. The trick here is to list magic input file-
twice. By GNU traditions, filename-
means read from standard input and listing that magic file twice forpaste
means it will read one line from "first" input file and then one line from "second" input file resulting in total of 2 lines read from standard input. Those lines will then be concatenated with a single space in between. Note that-d
only takes one byte.