Tuesday, January 6, 2009

Merge log files and sort the records in Linux

Recently I need to do merging of log files and sort the records according to timestamp. This is what I did, use the bulk replace file extension way to change the log files to .txt files, then applied the following command through Shell script.

#!/bin/sh



export MFILE=merge_file.txt

export SFILE=complete_file.txt



for file in `ls *.txt`

do



cat $file >> $MFILE



done

sort -k4,4 $MFILE > $SFILE

rm $MFILE




Note:
Log files are expected to have same format, otherwise the sorting will not be possible.


-k, --key = POS1[,POS2], means sorting by start a key at POS1, end it at POS2 (which is where the timestamp position in my log files)