UNIX time command output redirect to file:
---------------------------------------------------------------
$ time ./prog.sh
Initiating merge
Merge completed
real 0m8.803s
user 0m0.010s
sys 0m0.000s
#Trying to redirect the output to a file
$ time ./prog.sh > out.txt
real 0m8.804s
user 0m0.020s
sys 0m0.000s
#out.txt content:
$ cat out.txt
Initiating merge
Merge completed
#So it only redirected the STDOUT of the script executed, but the 'time' command outut is not redirected.
#This is becuase the command time sends it's output to STDERR (instead of STDOUT)
#To capture output of 'time' command:
$ { time ./prog.sh ; } 2> out.txt
Initiating merge
Merge completed
#Now out.txt content:
$ cat out.txt
real 0m8.303s
user 0m0.010s
sys 0m0.000s
#And to capture output of script as well as time command:
$ { time ./prog.sh ; } &> out.txt
#'out.txt' now has both the outputs.
$ cat out.txt
Initiating merge
Merge completed
real 0m8.303s
user 0m0.020s
sys 0m0.000s
there's two types of time command available:
1) Shell's in-build time: Gives only scheduler information
2) /usr/bin/time: Gives more information, also allows formatting the output
The second (/usr/bin/time) one accepts output redirection without code block:
$ /usr/bin/time ./prog.sh &> newout.txt
$ cat newout.txt
Initiating merge
Merge completed
0.00user 0.01system 0:08.30elapsed 0%CPU (0avgtext+0avgdata 5728maxresident)k
0inputs+8outputs (0major+719minor)pagefaults 0swaps
Bash insert newline after every 3 lines:
--------------------------------------------------------
-----------
Input file:
-----------
$ cat file.txt
FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67
EE 94
AA 80
NK 80
---------
Required:
---------
Insert a newline after every 3 lines of the above file. i.e. required output:
FR 24
AA 33
EE 34
EE 46
BE 30
AA 31
DE 90
AL 10
AA 50
FR 67
EE 94
AA 80
NK 80
----------
Solutions:
----------
a) Using awk:
$ awk '!( NR % 3 ) {$0 = $0"\n"} 1' file.txt
Which is same as:
$ awk '!( NR % 3 ) {$0 = $0"\n"} {print}' file.txt
b) A UNIX bash script to achieve this:
#!/bin/sh
c=0
while read line
do
((s=c%3))
if [ "$s" -eq 0 ]; then
echo -e "\n$line"
else
echo "$line"
fi
((c=c+1))
done < file.txt | sed '1d'
Following if-else block
if [ "$s" -eq 0 ]; then
echo -e "\n$line"
else
echo "$line"
fi
can also be written as:
[ "$s" -eq 0 ] && echo -e "\n$line" || echo "$line"
Unix Bash script - Check if integer or not:
-------------------------------------------------------------
Please find below two UNIX bash scripts to test if the entered input is integer or not. Request all to suggest if any other alternatives to check this. Thanks in advance.
1) Script 1:
$ cat check-integer.sh
#!/bin/sh
#Check if input is integer or not
[ -z $1 ] && echo "No input, exiting .." && exit
[[ $1 = *[![:digit:]]* ]] && echo "Not Integer" || echo "Integer"
Intersection between two files:
grep -Fx -f 51101 100002568
Bash - Concatenate with underscore in between:
--------------------------------------------------------------------
Input file:
$ cat file.txt
X 2 10
X 2 11
Y 5 12
Z 3 11
X 6 78
Required: For each of the lines of the above file, we are required to create sub directories of the following naming convention inside the directory outdir/.
i.e. for line
X 2 10
we need to create the sub directory:
outdir/b_config_X_2
The bash script I wrote:
$ cat createdir.sh
#!/bin/sh
while read line
do
eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
echo "$AID,$BID,$CID"
dir=b_config_$AID_$BID
mkdir -p outdir/$dir
done < file.txt
Executing it:
$ ./createdir.sh
X,2,10
X,2,11
Y,5,12
Z,3,11
X,6,78
Lets see what got created under outdir/
$ ls -l outdir/
total 16
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_2
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_3
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_5
drwxr-xr-x 2 root root 4096 Dec 28 04:54 b_config_6
To make it work, I did the following change in the above script:
dir=b_config_$AID\_$BID
So the modified script:
$ cat createdir.sh
#!/bin/sh
while read line
do
eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
echo "$AID,$BID,$CID"
dir=b_config_$AID\_$BID
mkdir -p outdir/$dir
done < file.txt
After execution, it created the correct sub directories:
$ ls -l outdir/
total 16
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_X_2
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_X_6
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_Y_5
drwxr-xr-x 2 root root 4096 Dec 28 04:56 b_config_Z_3
The other alternative to concatenate values of two variables with underscore in between is:
dir=b_config_${AID}_${BID}
What is the use of the following Awk eval function ?
eval $(echo "$line" | awk '{print "AID="$1";BID="$2";CID="$3}')
is a replacement of:
AID=$(echo $line | awk '{print $1}')
BID=$(echo $line | awk '{print $2}')
CID=$(echo $line | awk '{print $3}')
or
AID=$(echo $line | cut -d " " -f1)
BID=$(echo $line | cut -d " " -f2)
CID=$(echo $line | cut -d " " -f3)
Unix - Append 0 to single digit date:
-----------------------------------------------------
Input file file.txt has dates in month/day/year format.
$ cat file.txt
3/4/2013
3/10/2013
10/4/2013
12/10/2012
Required: Add prefix 0 to first and second field if its a single digit.
Awk solution:
$ awk 'BEGIN {FS=OFS="/"}
{
if (length($1) == 1) $1="0"$1
if (length($2) == 1) $2="0"$2
{ print }
}' file.txt
Output:
03/04/2013
03/10/2013
10/04/2013
12/10/2012
Sum of numbers in file - UNIX alternatives:
---------------------------------------------------------------
Input file:
$ cat /tmp/file.txt
286
255564800
609
146
671290
Required: Add (Sum) all the numbers present in the above file.
Way#1: This is supposed to be the most popular way of doing an addition of numbers present in a particular field of a file.
$ awk '{s+=$0} END {print s}' /tmp/file.txt
256237131
Way#2: Using UNIX/Linux 'paste' command and 'bc'
$ paste -sd+ /tmp/file.txt
286+255564800+609+146+671290
$ paste -sd+ /tmp/file.txt | bc
256237131
Way#3: Using UNIX/Linux 'tr' command and 'bc'
$ tr -s '\n' '+' < /tmp/file.txt
286+255564800+609+146+671290+
$ echo $(tr -s '\n' '+' < /tmp/file.txt)
286+255564800+609+146+671290+
#Since there's an extra '+' at end of above output, echo an additional '0' like this
$ echo $(tr -s '\n' '+' < /tmp/file.txt)0
286+255564800+609+146+671290+0
$ echo $(tr -s '\n' '+' < /tmp/file.txt)0 | bc
256237131
Way#4: Same as above but doing the arithmetic without using 'bc'
$ printf "%d\n" $(( $(tr -s '\n' '+' < /tmp/file.txt) 0 ))
256237131
Way#5: Using sed and 'bc'
$ sed 's/$/+/' /tmp/file.txt
286+
255564800+
609+
146+
671290+
$ echo $(sed 's/$/+/' /tmp/file.txt) 0
286+ 255564800+ 609+ 146+ 671290+ 0
$ echo $(sed 's/$/+/' /tmp/file.txt) 0 | bc
256237131
Way#6 : or a basic bash script using for loop
sum=0
for num in $(cat /tmp/file.txt)
do
((sum+=num))
done
echo $sum
Way#7: Using python
>>> sum = 0
>>> lines = open("/tmp/file.txt", "r").readlines()
>>> lines
['286\n', '255564800\n', '609\n', '146\n', '671290\n']
>>> for line in lines:
... sum+=eval(line)
...
>>> sum
256237131
Bash while loop sum issue explained :
-------------------------------------------------------
On one of my directory I had a lot of log files and I had to find the count of the total number of lines which starts with 's' (i.e. ^s).
My first approach was:
$ ls | xargs -i grep -c ^s {} | awk '{sum+=$0} END {print sum}'
190978
And I got my result. Then I thought of performing the same using bash scripting for and while loop and this is what I tried.
#!/bin/sh
sum=0
DIR=~/original
for file in $(ls $DIR)
do
Slines=$(grep -c ^s $DIR/$file)
((sum+=Slines))
#You can also use
#sum=$(expr $sum + $Slines)
#sum=`expr $sum + $Slines`
done
echo $sum
Executing it:
$ ./usingfor.sh
190978
Cool, correct result.
And then I modified the above script for bash while loop:
#!/bin/sh
sum=0
DIR=~/original
ls $DIR | while read file
do
Slines=$(grep -c ^s $DIR/$file)
((sum+=Slines))
done
echo $sum
Executing it:
$ ./usingwhile.sh
0
Oops!!! what went wrong ?
In Bash shell, piping directly to bash while loop causes the bash shell to function in a sub shell.
So in the above example the scope of the 'sum' variable is limited to the sub-shell of the while loop and so the modified value of 'sum' is not reflected when we exit the loop. Value of sum is still 0 (local value) as we initialized it to 0 at the beginning of the script.
The solution of this variable scoping problem with while and direct piping will be:
Remove the direct pipe and feed the list of file names under '~/original' directory as stdin to the while loop as shown below (Basically create a temp file with the file names of the directory '~/original')
#!/bin/sh
sum=0
DIR=~/original
ls $DIR > /tmp/filelist
while read file
do
Slines=$(grep -c ^s $DIR/$file)
((sum+=Slines))
done < /tmp/filelist
echo $sum
Executing it:
$ ./usingwhile_1.sh
190978
Extract sub-string from variable in bash:
------------------------------------------------------------
Suppose:
$ mypath=/dir1/dir2/dir3/dir4
$ echo $mypath
/dir1/dir2/dir3/dir4
Now, if you need to print the parent path from the above path (i.e. print '/dir1/dir2/dir3')
$ dirname $mypath
/dir1/dir2/dir3
$ parentpath=$(dirname $mypath)
$ echo $parentpath
/dir1/dir2/dir3
Using Sub-string Removal ways in Bash shell
${string%substring}
It deletes shortest match of $substring from 'back' of $string.
$ echo ${mypath%/*}
/dir1/dir2/dir3
or
$ printf '%s\n' "${mypath%/*}"
/dir1/dir2/dir3
If you need to print the last directory name from the above mypath, here are few ways:
Using Sub-string Removal ways in Bash shell
${string##substring}
It deletes the "longest" match of $substring from 'front' of $string.
$ echo ${mypath##*/}
dir4
Another way using awk:
$ echo $mypath | awk '{print $NF}' FS=\/
dir4
Bash script to copy required files:
---------------------------------------------------
Contents of my "inputdir" is a set of files with filename like this:
$ ls -1 inputdir/
log.10.16.1253168140.txt
log.11.5.1253168345.txt
log.11.9.1253168347.txt
log.12.1.1253168347.txt
log.19.1.1253168140.txt
Directory "testcfgs" contains a set of config xmls.
$ ls -1 testcfgs/
cfg_10_16.xml
cfg_10_5.xml
cfg_11_5.xml
cfg_11_9.xml
cfg_12_1.xml
cfg_19_1.xml
cfg_19_2.xml
cfg_91_9.xml
Required:
For each file of name "log.X.Y.timestamp.txt" in "inputdir", copy the corresponding "cfg_X_Y.xml" config file from "testcfgs" to a directory say "requiredcfgs".
A simple practical bash one liner script:
$ for filename in $(ls -1 inputdir/)
> do
> X=$(echo "$filename" | cut -d"." -f2)
> Y=$(echo "$filename" | cut -d"." -f3)
> cp testcfgs/cfg_$X\_$Y.xml requiredcfgs/
> done
The two lines above for finding X and Y value can be replaced by a single line using 'eval with awk', like this:
$ for filename in $(ls -1 inputdir/)
> do
> eval $(echo "$filename" | awk -F "." '{print "X="$2";Y="$3}')
> cp testcfgs/cfg_$X\_$Y.xml requiredcfgs/
> done
Contents of "requiredcfgs" directory after execution of the above bash script.
$ ls -1 requiredcfgs/
cfg_10_16.xml
cfg_11_5.xml
cfg_11_9.xml
cfg_12_1.xml
cfg_19_1.xml
Rename file to uppercase except extension - Bash:
-----------------------------------------------------------------------
My present working directory got the following files.
$ ls -1
new.py
readme.txt
sl.pl
test.py
Required: Make all file in this directory uppercase but not their extension (ex: image.jpg becoming IMAGE.jpg)
The shell script on the command prompt:
$ ls | while read file
> do
> name=${file%%.*}
> extn=${file##*.}
> newfilename=$(echo $(echo $name | tr 'a-z' 'A-Z').$extn)
> echo "Moving $file to $newfilename"
> mv $file $newfilename
> done
Note: Linux secondary prompt (PS2)
The > (greater than sign) is the Linux secondary prompt (PS2).
When one issues command that is incomplete, the shell will display this prompt and will wait for the user to complete the command and hit Enter again.
Output of the above bash script:
Moving new.py to NEW.py
Moving readme.txt to README.txt
Moving sl.pl to SL.pl
Moving test.py to TEST.py
Now
$ ls -1
NEW.py
README.txt
SL.pl
TEST.py
Bash script for sequential subtraction of numbers :
---------------------------------------------------------------------------
Input file epoch.txt contains UNIX epoch time in the following format.
$ cat epoch.txt
1249887102
1249887121
1249887181
1249887241
1249887433
1249887481
1249887541
1249887601
1249887661
Required:
Subtract 1st number from 2nd number(i.e. 2nd-1st),3rd number minus 2nd number and so on ...
The bash script:
#!/bin/sh
FILE=epoch.txt
N=1
total=$(sed -n '$=' $FILE)
until [ "$N" -eq $total ]
do
S1=$N
((S2=N+1))
N=$S2
VAL1=$(sed -n "$S1 p" $FILE)
VAL2=$(sed -n "$S2 p" $FILE)
echo "$VAL2 - $VAL1 = $((VAL2 - VAL1))"
done
$ sh difcal.sh
1249887121 - 1249887102 = 19
1249887181 - 1249887121 = 60
1249887241 - 1249887181 = 60
1249887433 - 1249887241 = 192
1249887481 - 1249887433 = 48
1249887541 - 1249887481 = 60
1249887601 - 1249887541 = 60
1249887661 - 1249887601 = 60
Padding zeros in file name using bash:
-----------------------------------------------------------
Example 1
---------
Files in my current directory are:
$ ls | paste -
a.10.done
a.2.done
m.100.done
n.1.done
Required: Rename the above files to
a.0010.done
a.0002.done
m.0100.done
n.0001.done
i.e. make the "number part" of the filename equal width by padding the number with leading zeroes
The solution using awk and bash parameter substitution
for file in $(ls)
do
eval $(echo "$file"|awk -F'.' '{print "prefix="$1";sl="$2}')
mv $file $(printf "%s.%04d.%s\n" "$prefix" "$sl" "done")
echo "Renamed $file”
done
Example 2
---------
Files in my current directory are:
$ ls | paste -
apl.1
apl.10
apl.100
apl.2
apl.5
Required: Rename the above files to
apl.0001
apl.0010
apl.0100
apl.0002
apl.0005
The solution:
for file in apl.*
do
psl='0000'${file#apl.}
psl=${psl:(-4)}
mv $file apl.$psl
done
Another way would be:
for file in apl.*
do
mv $file $(printf apl.%04d ${file#apl.})
done
Remove a path from $PATH variable in bash:
-------------------------------------------------------------------
$PATH on my ubuntu:
$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/org/bin:/opt/appls
Now to remove any path which contain "/org/" from PATH variable:
$ echo $PATH | tr ':' '\n' | awk '$0 !~ "/org/"' | paste -sd:
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/appls
Export the above to PATH
$ export PATH=$(echo $PATH | tr ':' '\n' | awk '$0 !~ "/org/"' | paste -sd:)
$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/appls
Use
awk '$0 != "/usr/local/sbin"'
in the above export awk part if you exactly want to remove the path "/usr/local/sbin" from PATH.
And using bash array:
$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/org/bin:/opt/appl
Remove any path which contain "/org/" from PATH
The one liner:
$ PATH=$(IFS=':';p=($PATH);unset IFS;p=(${p[@]%%*/org/*});IFS=':';echo "${p[*]}";unset IFS)
$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/appl
Parsing env variable PATH in bash:
-----------------------------------------------------
Requirement: Check for the existence of the directories mentioned in PATH env variable.
The bash script:
#!/bin/sh
IFS=:
read dirs <<END
$PATH
END
for dir in $dirs
do
[ ! -e "$dir" ] && echo "$dir not exist" \
|| echo "$dir exist"
done
Executing the above bash script:
$ ./check-PATH.sh
/usr/local/bin exist
/usr/bin exist
/bin exist
/usr/X11R6/bin exist
/cygdrive/c/WINDOWS/system32 exist
/cygdrive/c/WINDOWS exist
/usr/bin exist
/usr/lib/sanity not exist
Test for empty directory - bash scripting:
------------------------------------------------------------
Lets see how we can check whether a directory is empty or not in bash scripting.
#Create a directory
$ mkdir testdir
#Usual way to check if a directory is empty or not
$ [ -z $(ls testdir/) ] && echo "empty" || echo "NOT"
empty
#Lets create a file in testdir
$ touch testdir/file1
#Oh, it works, great
$ [ -z $(ls testdir/) ] && echo "empty" || echo "NOT"
NOT
#Create one more file, file2
$ touch testdir/file2
#Now see, ohh!!! it has faced some problem
$ [ -z $(ls testdir/) ] && echo "empty" || echo "NOT"
-bash: [: file1: binary operator expected
NOT
#Solution to the above:
$ [ -z "$(ls testdir/)" ] && echo "empty" || echo "NOT"
NOT
#Clear the testdir
$ rm testdir/file*
#So that
$ [ -z "$(ls testdir/)" ] && echo "empty" || echo "NOT"
empty
#Create a hidden file in the testdir
$ touch testdir/.hden
#But the above test condition is not going to take account of hidden files (as we are doing only ls)
$ [ -z "$(ls testdir/)" ] && echo "empty" || echo "NOT"
empty
#So use -A (do not list implied . and ..) option with ls
$ [ -z "$(ls -A testdir/)" ] && echo "empty" || echo "NOT"
NOT
#Create a file
$ touch testdir/file1
#Working ...
$ [ -z "$(ls -A testdir/)" ] && echo "empty" || echo "NOT"
NOT
#Remove the files (including hidden file) from testdir
$ rm testdir/.hden ; rm testdir/file1
#So the above is working.
$ [ -z "$(ls -A testdir/)" ] && echo "empty" || echo "NOT"
empty
But if we use -a option instead of -A
$ [ -z "$(ls -a testdir/)" ] && echo "empty" || echo "NOT"
NOT
This is because:
$ ls -a testdir/
. ..
But
$ ls -A testdir/
Handling argument list too long - bash :
--------------------------------------------------------------
I have nearly 200,000 files in one of my log directory out of which number of files of the name format "ka.log.*" is 120,000. So whenever I try to do apply some command such as rm, ls or cp etc on those big set of "ka.log.*" files, I used to get
$ ls ka.log.*
bash: /bin/ls: Argument list too long
$ cp ka.log.* new/
bash: /bin/cp: Argument list too long
$ mv ka.log.* new/
bash: /bin/mv: Argument list too long
$ rm ka.log.*
bash: /bin/rm: Argument list too long
"Argument list too long" error for the above commands is due to the limitation of the command (rm, mv, ls, cp) to handle large number of files(arguments).
Linux 'find' command is useful to perform these operations (ls, cp, mv or rm etc) on such big set of files/arguments.
e.g.
To copy those "ka.log.*" to directory /somedir
$ find . -name "ka.log.*" -exec cp {} /somedir/ \;
Looping through while:
find . -name "ka.log.*" | while read FILE
do
...
<some operation on $FILE>
...
done
Another way is to assign the file names to a variable, e.g.
FILES=$(echo /mydir/ka.log.*)
for FILE in $FILES
do
...
<some operation on $FILE>
…
done.
