Today I was looking at a web server’s log, more specifically the Nginx access.log. I was a bit surprised, and confused, because every line started with the address ::1, and the actual IP address was at the very end of each line.

Reading the Nginx documentation, reveals that the default logging format is the “combined” format, which looks like this:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

On this particular server, it looked like this:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

In short, the format is the same except that we add the X-Forwarded-For header value to the end of each line. Meaning that a proxy is actually specifying an IP address and passing it to the server. In this case, this proxy is Cloudflare.

However, this setup is neither neat nor secure. First of all, because we redundantly store ::1 on every line and have the actual IP on the last part, which is not a standard logging format and could cause issues with any form of standard log analyzer (admittedly, you can write rules, but… why make it harder on yourself?), and second, because anyone on the network could in theory pass this header and falsify the source IP.

We can do better.

Now, usually, we’d get the Cloudflare IP ranges and create an Nginx config to set the real IP based on Cloudflare’s header, while only trusting Cloudflare’s IPs, so that nobody else can just set this header and falsify the IP.

However, in this case, this is an LXC container which does not actually ever see any remote IPs, because it’s not even directly connected to the internet. Instead, the LXC container is running a Cloudflare tunnel and only speaks to Cloudflare over this tunnel, ergo all external connections come through this tunnel. And since this tunnel then connects to our Nginx, the IP will of course always be that of localhost.

How to fix it

We have to tell Nginx to get the real IP from the HTTP header CF-Connecting-IP while *only* trusting localhost to set this header.

Create the file /etc/nginx/conf.d/cloudflare-realip.conf with the following contents:

set_real_ip_from ::1;
set_real_ip_from 127.0.0.0/8;
real_ip_header CF-Connecting-IP;

Then check that the config has no errors by running:

nginx -t

And finally reload the Nginx config:

systemctl reload nginx

Now access the server’s website to generate some traffic and then let’s check our log again:

tail /var/log/nginx/access.log

See the actual IPs now? Cool!

Now some optional perfectionism at the end.

In theory we’re done here, but, you probably also want to edit the Nginx log format to restore the default, or if you’re already using a heavily modified log format, at least remove the "$http_x_forwarded_for" part.

In my case, since the only difference was this one HTTP header, I’m going to restore the default log_format by editing /etc/nginx/nginx.conf:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

and replacing this part simply with:

access_log  /var/log/nginx/access.log;

Then again nginx -t and systemctl reload nginx. However, if you want to be extra safe, what you actually wanna do after nginx -t is to run:

systemctl stop nginx
logrotate --force /etc/logrotate.d/nginx
systemctl start nginx

To make sure you’ll have consistent formats across your log files. This will make it easier later on if you want to correct the old logfiles, to not have a sudden format change in the middle of the file which could break reformatting if done with simple scripts.

Conclusion

The fix is basically 3 lines, and quite straightforward. But the fun comes in documenting this, uh… case-study? While also giving advice on how to do it *right*. :3

I run this blog in my spare time, if I helped you out, consider donating a cup of coffee. <3