Securing Prometheus with Caddy

The below explains how to secure access to a Prometheus server. By default, a Prometheus server instance uses insecure HTTP protocol on port 9090.

To secure a Prometheus server instance, do the the following:

• Block off-box access to TCP/9090
• Allow HTTPS access to port TCP/8443
• Install caddy web server
• Configure caddy to run on TCP/8443
• Configure caddy to proxy requests coming to port 8443 to the Prometheus server running on port 9090
• Configure authentication of the requests using basic authentication

Firewall Rules

First, check whether there is an existing rule allowing access to the Prometheus server. Here, such rule exists. It is rule 5.

# iptables -L -n --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:9090
6    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:9093
7    REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Delete the rule:

iptables -D INPUT 5

Second, check whether there is a rule allowing communication on port 8443. Here, it does not exist. Add it:

iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT

Caddy Configuration File

Use the following caddy configuration file:

mkdir -p /etc/caddy
cat <<'EOF' > /etc/caddy/Caddyfile
*:8443 {
    root /var/lib/caddy
    index index.html
    errors stdout
    log stdout
    tls /etc/caddy/myServerBundle.crt /etc/caddy/myServer.key
    basicauth /prometheus admin CaddyPrometheus
    proxy /prometheus localhost:9090 {
      header_upstream X-Real-IP {remote}
      header_upstream X-Forwarded-For {remote}
    }
}
EOF

The following configuration defines the root directory for caddy, i.e. /var/lib/caddy, the default file, i.e. index.html. It also instructs caddy to output logs and errors to standard output. Further, it provides TLS configuration via a certificate bundle file, followed by a server encryption key file.

root /var/lib/caddy
    index index.html
    errors stdout
    log stdout
    tls /etc/caddy/myServerBundle.crt /etc/caddy/myServer.key

The following configuration line creates admin user with CaddyPrometheus password.

basicauth /prometheus admin LoveCaddyPrometheus

Finally, the below instructs caddy to proxy all requests with the PATH beginning with /prometheus to the Prometheus server running on the same host on port 9090. When doing so, two new headers are being added, X-Real-IP and X-Forwarded-For.

proxy /prometheus localhost:9090 {
      header_upstream X-Real-IP {remote}
      header_upstream X-Forwarded-For {remote}
    }

Start caddy:

/bin/caddy -conf /etc/caddy/Caddyfile

Start Prometheus server instance with this configuration option:

--web.external-url https://<HOSTNAME>:8443/prometheus

The Prometheus instance will be accessible via https://<HOSTNAME>:8443/prometheus URL.