Parsing large log files quickly

Timegrep is a fantastic utility to parse through massive log files quickly.  It does a binary search for a time range based on a specified time format.

The utility is available off github: https://github.com/linux-wizard/timegrep

Here is an example of how I can use it to go through and grep through dozens of log files each of which can be several GBs in size:

This example is for an NGINX server's errors.

find /var/log/nginx/ -type f -name '*.log-20170730' -exec ~/bin/timegrep.py -d 2017-07-29 --start-time=19:30:00 --end-time=19:45:00 '{}' \; | grep '\[error\]' > ./errors-list.txt

Another example to get some stats from Apache, combined with some piping and grepping from: https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/

Run this command from /var/log/httpd on a CentOS system:


find . -type f -name '*.access.log' -exec /root/bin/timegrep.py -d 2017-07-31 --start-time=10:05:00 --end-time=10:06:00 '{}' \; | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

This will go through all of the .access.log files in /var/log/httpd and parse all of the entries during the 10:05 to 10:06 minute, and print the top 20 IPs.

Basically, if you combine timegrep with the find command, you've got yourself some serious log parsing firepower.

Of course, if you've got this quantity of logs to parse through, sometimes tools like splunk are a bit more appropriate.  However, as they are not always available, the above technique can get you out of a serious bind.

Block access to files by IP using X-Forwarded-For

What is the purpose of blocking by X-Forwarded-For IP, instead of the REMOTE_ADDR?

Sometimes a site may be behind a reverse proxy and it may not be possible to add a rule to block a file by IP at the reverse proxy level.  If the reverse proxy is passing the remote client IP in a header like X-Forwarded-For, you can still block by client IP.

Match the header to an IP address and assign it to an environment variable in Apache.  Here is an example of a complete configuration to block remote access to a wordpress login page, except for a certain range of IPs:

<files wp-login.php>
order deny,allow
deny from all
SetEnvIf X-Forwarded-For "192\.168\..*" LocalAccess
SetEnvIf X-Forwarded-For "10\..*" LocalAccess
Allow from env=LocalAccess
</files>

If the IP contained in the X-Forwarded-For header matches one of the regular expressions, it will populate the "LocalAccess" environment variable.

Parse Apache Logs by Date Range

Parsing apache logs by date and by date ranges can be fairly simple with a bit of awk scripting.

We use AWK to compare date fields in order to retrieve specific rows.

The date fields between access logs and error logs can vary, so some adjustments are needed:

Note that the date field is contained within a single column in the access_log file, therefore we can do a comparison against a single column.  Typically column #4.

AWK Date Range for access logs:

$ awk '$4>"[09/Aug/2012:15:00:" && $4<"[09/Aug/2012:15:59:"' ./access_log | less

The date field in the error log is in separate columns.  Example: [Thu Aug 09 15:30:...  That in itself is four columns.  They must be combined in order to be compared effectively.  To do this, we assign a combination of those four columns to two variables: $from and $two.  We then use these two variables for the comparison.  See below:

AWK Date Range for error logs:

$ awk '$from>"[Thu Aug 09 15:30:00" && $to<"[Thu Aug 09 15:59:00"' from='$1 " " $2 " " $3 " " $4' to='$1 " " $2 " " $3 " " $4' ./error_log | less