When dealing with processing really large files (300files~15MBeach), it quickly becomes a process that you have to wait a long time for it to finish. I wrote a Java program that manipulated a flat text P1/90 (Seismic Vessel’s Positions) file and outputed a corrected version in another directory. Thus I wanted to check how far along it had gone and when I should expect it to finish.
Mind you I could have done this in the Java code – keep a record of all the files and how many had been processed – but this project was time-constrained so I didn’t have the luxury of developing it the way I wanted; it had to be done quickly. I therefore delegated the monitoring to the command line tools. Since I new how many files I had originally and how many were being created in the new directory, all I had to do is count each of them and compare them. The following one-liner does exactly that:
watch 'ls *190 | echo -n "`wc -l`"; echo -n "/"; ls ../*90 | wc -l'
The watch command executes a command periodically and prints the output on the terminal. It could be parametrised further by defining the time interval of execution with the -n flag, e.g. watch -n 60 would execute the command every 60 seconds. The above watch command is enclosed in single quotes and does the following:
- ls *90 | echo -n “`wc -l`” : This will enumerate the files ending in “90″ in the current directory (the one I was creating the output files) and pipe that into a print command. The print command is the echo. The -n flag removes the newline character at the end, so that we can keep the printout in the same line. The print output of the echo is the word count (wc) of how many filenames the ls command return. This gives us the total number of files found in this directory.
- echo -n “/” : This will append in the same line a forward slash. This is also inline with the previous output and creates the appearance of a ratio.
- ls ../*90 | wc -l :The final command will print, with a newline at the end, the number of files in the original – above – directory.
The output of this command looked like this:
Every 2.0s: ls *190 | echo -n "`wc -l`"; echo -n "/"; ls ../*90 | ... Tue Apr 10 16:51:00 2012 288/336
Do you have similar examples you can contribute?