Tuesday, January 6, 2009

Merge log files and sort the records in Linux

Recently I need to do merging of log files and sort the records according to timestamp. This is what I did, use the bulk replace file extension way to change the log files to .txt files, then applied the following command through Shell script.

#!/bin/sh

export MFILE=merge_file.txt
export SFILE=complete_file.txt

for file in `ls *.txt`
do

cat $file >> $MFILE

done
sort -k4,4 $MFILE > $SFILE
rm $MFILE


Note:
Log files are expected to have same format, otherwise the sorting will not be possible.

-k, --key = POS1[,POS2], means sorting by start a key at POS1, end it at POS2 (which is where the timestamp position in my log files)