How to enable external Caddy Plugins

The below explains how to enabled external caddy plugins. Here, caddyserver/forwardproxy plugin is being enabled.

First, get Golang:

cd /usr/local &&
curl -s -L -O https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz
tar -xvzf go1.10.1.linux-amd64.tar.gz

Next, add the following ~/.bash_profile:

export GOPATH=$HOME/go
export GOBIN=$HOME/go/bin
export GOROOT=/usr/local/go

Also, add GOBIN to the PATH environment variable:

PATH=$GOBIN:$PATH
export PATH

After this step, you have Golang compiler ready in /usr/local/go/bin/ directory:

# /usr/local/go/bin/go version
go version go1.10.1 linux/amd64

Download the source code of caddy and caddyserver/forwardproxy plugin:

go get github.com/mholt/caddy/caddy
go get github.com/caddyserver/builds
go get github.com/caddyserver/forwardproxy

Add a reference to caddyserver/forwardproxy plugin in caddy source code. Find "This is where other plugins get plugged in" in $GOPATH/src/github.com/mholt/caddy/caddy/caddymain/run.go file and add a reference to the plugin.

_ "github.com/caddyserver/forwardproxy"

Finally, build caddy binary:

cd $GOPATH/src/github.com/mholt/caddy/caddy
go run build.go
caddy --version

The http.forwardproxy plugin is in the list of enabled plugins:

# caddy -plugins
Server types:
  http

Caddyfile loaders:
  short
  flag
  default

Other plugins:
  http.forwardproxy

Securing Prometheus with Caddy

The below explains how to secure access to a Prometheus server. By default, a Prometheus server instance uses insecure HTTP protocol on port 9090.

To secure a Prometheus server instance, do the the following:

• Block off-box access to TCP/9090
• Allow HTTPS access to port TCP/8443
• Install caddy web server
• Configure caddy to run on TCP/8443
• Configure caddy to proxy requests coming to port 8443 to the Prometheus server running on port 9090
• Configure authentication of the requests using basic authentication

Firewall Rules

First, check whether there is an existing rule allowing access to the Prometheus server. Here, such rule exists. It is rule 5.

# iptables -L -n --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:9090
6    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:9093
7    REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Delete the rule:

iptables -D INPUT 5

Second, check whether there is a rule allowing communication on port 8443. Here, it does not exist. Add it:

iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT

Caddy Configuration File

Use the following caddy configuration file:

mkdir -p /etc/caddy
cat <<'EOF' > /etc/caddy/Caddyfile
*:8443 {
    root /var/lib/caddy
    index index.html
    errors stdout
    log stdout
    tls /etc/caddy/myServerBundle.crt /etc/caddy/myServer.key
    basicauth /prometheus admin CaddyPrometheus
    proxy /prometheus localhost:9090 {
      header_upstream X-Real-IP {remote}
      header_upstream X-Forwarded-For {remote}
    }
}
EOF

The following configuration defines the root directory for caddy, i.e. /var/lib/caddy, the default file, i.e. index.html. It also instructs caddy to output logs and errors to standard output. Further, it provides TLS configuration via a certificate bundle file, followed by a server encryption key file.

root /var/lib/caddy
    index index.html
    errors stdout
    log stdout
    tls /etc/caddy/myServerBundle.crt /etc/caddy/myServer.key

The following configuration line creates admin user with CaddyPrometheus password.

basicauth /prometheus admin LoveCaddyPrometheus

Finally, the below instructs caddy to proxy all requests with the PATH beginning with /prometheus to the Prometheus server running on the same host on port 9090. When doing so, two new headers are being added, X-Real-IP and X-Forwarded-For.

proxy /prometheus localhost:9090 {
      header_upstream X-Real-IP {remote}
      header_upstream X-Forwarded-For {remote}
    }

Start caddy:

/bin/caddy -conf /etc/caddy/Caddyfile

Start Prometheus server instance with this configuration option:

--web.external-url https://<HOSTNAME>:8443/prometheus

The Prometheus instance will be accessible via https://<HOSTNAME>:8443/prometheus URL.