Insert, append, change lines using sed :
--------------------------------------------------------
Input file:
$ cat order789.txt
title:
PO for vessel unit1 const.
items:
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar
# Add the line "heater:M21789" after the line "items"
$ sed '
/items/ a\
heater:M21789
' order789.txt
Output:
title:
PO for vessel unit1 const.
items:
heater:M21789
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar
# Add the line "heater:M21789" after line number 3 (output would be same as above)
$ sed '
3 a\
heater:M21789
' order789.txt
# Insert a line "heater:M21789" before the line beginning with "fan"
$ sed '
/^fan/ i\
heater:M21789
' order789.txt
Output:
title:
PO for vessel unit1 const.
items:
heater:M21789
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar
# We can insert or add more than one line also.
$ sed '
4 i\
heater:M21789\
newitem:YYYYY
' order789.txt
Output:
title:
PO for vessel unit1 const.
items:
heater:M21789
newitem:YYYYY
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar
# One can change a line as well, e.g. to change the line beginning with "fan" with the line "BigFAN:F5757"
$ sed '
/^fan/ c\
BigFAN:F5757
' order789.txt
Output:
title:
PO for vessel unit1 const.
items:
BigFAN:F5757
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar
Embedding shell command in sed - bash:
-----------------------------------------------------------
Input file:
$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active
The content of '/etc/lsb-release' file on my ubuntu desktop:
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.04
DISTRIB_CODENAME=hardy
DISTRIB_DESCRIPTION="Ubuntu 8.04.3 LTS"
Lets extract the 'DISTRIB_RELEASE' version from the above file:
$ awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release
8.04
Required: Replace the text 'VERSION' in the input file 'file.txt' with the output of the above command (i.e. 'DISTRIB_RELEASE' version)
First way of doing this:
$ myvar=$(awk '/DISTRIB_RELEASE/ {print $2}' FS=\= /etc/lsb-release)
$ sed "s/VERSION/$myvar/g" file.txt
Output
port:9903
os-version:8.04
codename:hardy
status:active
** Important to see that we have used double quotes (instead of regular single quote) in the above sed statement. Using single quote is not going to expand the content of the variable 'myvar'.
Other ways :
$ sed "s/VERSION/`awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release`/g" file.txt
Which is same as:
$ sed "s/VERSION/$(awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release)/g" file.txt
And another way of quoting (This will allow you to use the single quote with sed statement):
$ sed 's/VERSION/'"$(awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release)"'/g' file.txt
same as
$ sed 's/VERSION/'"`awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release`"'/g' file.txt
Extract range of lines using sed awk bash:
-------------------------------------------------------------
Below are few different ways to print or extract a section of a file based on line numbers.
Lets try to extract lines between line number 27 and line number 99 of input file 'file.txt'
Using sed editor:
$ sed -n '27,99 p' file.txt > /tmp/file1
Which is same as:
$ sed '27,99 !d' file.txt > /tmp/file2
Awk alternative : you can make use of awk NR variable
$ awk 'NR >= 27 && NR <= 99' file.txt > /tmp/file3
Using Linux/UNIX 'head' and 'tail' command:
$ head -99 file.txt | tail -73 > /tmp/file4
Which is basically:
$ head -99 file.txt | tail -$(((99-27)+1)) > /tmp/file5
In vi editor, we can use the following command in ex mode (open the main file 'file.txt' in vi):
:27,99 w! /tmp/file6
i.e. Write lines between line number 27 and line number 99 of main file 'file.txt' to file '/tmp/file6'
Perl alternative would be:
$ perl -ne 'print if 27..99' file.txt > /tmp/file7
And the solution using python:
$ python
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
>>> fp = open("/tmp/file8","w")
>>> for i,line in enumerate(open("file.txt")):
... if i >= 26 and i < 99 :
... fp.write(line)
...
>>>
So the contents of all the output files produced (i.e /tmp/file[1-8]) will be the same (i.e. line number 27 to line number 99 of 'file.txt')
Sed to replace a part of file : bash scripting :
-----------------------------------------------------------
$ cat idfile
id='1' dsadsad adsad
id='31' dsadsad adsad 32432
id='231' dsadsad adsad 3234
id='123' 2332 dsadsad adsad
id='124' 2332
Output required:
Subtract 1 from the id value from each line of the above file. i.e. final output should like this:
id='0' dsadsad adsad
id='30' dsadsad adsad 32432
id='230' dsadsad adsad 3234
id='122' 2332 dsadsad adsad
id='123' 2332
The script:
$ cat idfile | while read line
> do
> R=`echo $line | sed "s/id='\([0-9].*\)'.*/\1/"`
> ((R-=1))
> echo $line | sed "s/\(id='\)\([0-9].*\)\('.*\)/\1$R\3/"
> done
Note:
If
((R-=1))
complians, then use
R=`expr $R - 1`
Some references:
$ echo "id='1' dsadsad adsad" | sed "s/\(id='\)\([0-9].*\)\('.*\)/\1/"
id='
$ echo "id='1' dsadsad adsad" | sed "s/\(id='\)\([0-9].*\)\('.*\)/\2/"
1
$ echo "id='1' dsadsad adsad" | sed "s/\(id='\)\([0-9].*\)\('.*\)/\3/"
' dsadsad adsad
Merge previous line using sed :
----------------------------------------------
Input file:
$ cat file.txt
1 AA 2
2 BB 3
3 DD 4
5 CC 12
7 ZZ 12
Required output:
If I search for line with first field as "3" , th line with first field 3 should be merged with the previous line to it.
i.e
1 AA 2
2 BB 3 3 DD 4
5 CC 12
7 ZZ 12
Find the character immediately after a word - sed:
------------------------------------------------------------------------
$ echo "unix:bash/scripting:12"
unix:bash/scripting:12
Purpose: find the character after the word "bash"
$ echo "unix:bash/scripting:12" | sed 's/.*bash\(.\).*/\1/'
Delete one or more space using sed- bash:
------------------------------------------------------------
Input file:
$ cat test.txt
one two three four
1 2 3 4
i ii iii iv
Sed code to delete/remove one or more space from a file. (Basically combine one or more space to a single space)
$ sed 's/[ \t]\+/ /g' test.txt > test.txt.1
$ cat test.txt.1
one two three four
1 2 3 4
i ii iii iv
Remove or replace newlines using sed,awk,tr - BASH:
----------------------------------------------------------------------------
$ cat week.txt
Su
Mo
Tu
We
Th
Fr
Sa
Output Required:
- a) Remove the newlines. i.e. required output:
SuMoTuWeThFrSa
- b) Replace the newlines with "|", i.e.
Su|Mo|Tu|We|Th|Fr|Sa
Remove/Replace newlines with sed
a)
$ sed -e :a -e '$!N;s/\n//;ta' week.txt
SuMoTuWeThFrSa
b)
$ sed -e :a -e '$!N;s/\n/|/;ta' week.txt
Su|Mo|Tu|We|Th|Fr|Sa
One more way of doing. But not suitable for files with large number of records, as you see the number of N's is just 1 less than number of lines in the file.
a)
$ sed 'N;N;N;N;N;N;s/\n//g' week.txt
SuMoTuWeThFrSa
b)
$ sed 'N;N;N;N;N;N;s/\n/|/g' week.txt
Su|Mo|Tu|We|Th|Fr|Sa
Remove/Replace newlines with awk
a)
$ awk '{printf "%s",$0} END {print ""}' week.txt
SuMoTuWeThFrSa
b)
$ awk '{printf "%s|",$0} END {print ""}' week.txt
Su|Mo|Tu|We|Th|Fr|Sa|
So we need to remove the last "|" in the above output.
$ awk '{printf "%s|",$0} END {print ""}' week.txt | awk '{sub(/\|$/,"");print}'
Su|Mo|Tu|We|Th|Fr|Sa
Remove/Replace newlines with tr
a)
$ tr -d '\n' < week.txt
SuMoTuWeThFrSa
b)
$ tr '\n' '|' < week.txt
Su|Mo|Tu|We|Th|Fr|Sa|
Similarly we need to remove the last "|" from the above output:
$ tr '\n' '|' < week.txt | sed 's/|$//'
Su|Mo|Tu|We|Th|Fr|Sa
