Logs are nice and all that, but sometimes certain entries are there just to fill up the logs or are cluttering them. Here’s a few ways to exclude requests – by URL or visitor IP – from the Nginx access log.
[bjornad]
Exclude specific URLs
In my Nginx configs I usually have a location block like this for static resources. It makes sure correct caching headers are sent, but also turns off logging for static resources – both the access log and the error log when a 404 is returned:
location ~* \.(png|jpg|jpeg|gif|ico|woff|otf|ttf|eot|svg|txt|pdf|docx?|xlsx?)$ {
access_log off;
log_not_found off;
expires max;
add_header Pragma public;
add_header Cache-Control "public";
add_header Vary "Accept-Encoding";
}
(I handle JS and CSS files in a separate block. Also note that you can very well use a map instead of the awkwardly long regexp)
Exclude specific user agents
If you’re not interested in logging requests from certain bots/crawlers, or you have a monitoring service, like Pingdom.
map $http_user_agent $log_ua {
~Pingdom 0;
~Googlebot 0;
~Baiduspider 0;
default 1;
}
server {
[…]
access_log /var/log/nginx/access.log main if=$log_ua;
}
Note that maps must be created outside of the server block.
Exclude specific IP addresses
If you run a cronjob with curl/wget on your localhost polling your website at certain intervals (e.g. you run WordPress but use a real cronjob to poll wp-cron.php), or you have health checks from downstream.
map $remote_addr $log_ip {
"127.0.0.1" 0;
"10.0.0.2" 0;
"10.0.0.3" 0;
default 1;
}
server {
[…]
access_log /var/log/nginx/access.log main if=$log_ip;
}
Combining tests
If you want to match only specific user agents coming for specific IP addresses, it is possible to combine the tests. However, the Nginx config does not support a logical and operator, so we have to resort to a “clever hack”. Here’s a config to check for certain user agents from certain IPs:
map $http_user_agent $log_ua {
~Pingdom 0;
~Googlebot 0;
~Baiduspider 0;
default 1;
}
map $remote_addr $log_ip {
"127.0.0.1" 0;
"10.0.0.2" 0;
"10.0.0.3" 0;
default 1;
}
server {
[…]
set $logging 1;
set $logtest '';
if ( $log_ua = 0 ) {
set $logtest "${logtest}A";
}
if ( $log_ip = 0 ) {
set $logtest "${logtest}B";
}
if ( $logtest = "AB" ) {
set $logging 0;
}
access_log /var/log/nginx/access.log main if=$logging;
}
You can of course add a “test C” checking for the URL too.