Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far.
edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.
Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy.
It's not tightly coupled any more than it would be with certbot. The issue I filed is an issue because it's unexpected behaviour (despite early closure, it has actually now been fixed).
> I think this thread is pretty much proof that that approach will bite you in the ass.
What bit me here is the fact that I'm running alpha software instead of a battle-tested web server; I'm doing so willingly, with full awareness of the risks that that entails.
Drawing the conclusion you did from the variables at play is shortsighted. If anything bites people in the ass, it's prejudice and shortsightedness. I wouldn't want you handling my ops/infrastructure.
> What bit me here is the fact that I'm running alpha software
This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were <= 30 days away from expiring and the CA can't be contacted.
> Drawing the conclusion you did from the variables at play is shortsighted
Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't working the way you want, you have to either a) replace it completely or b) work out how to disable the bit that's not working how you want, and replace that part of it.
> Drawing the conclusion you did from the variables at play is shortsighted
- People use a piece of software that serves as both ACME TLS certificate client and web server
- Said software by design won't start if the CA can't be contacted 30-days out from expiry
The conclusion I drew is that such integration leave the operator with less control than if they followed a separation-of-concerns approach, and left web serving to a web server, and TLS certificate renewal to an ACME client. The former doesn't need to care about how old the certificates are, just use what it's given.
I still think that refusing to start if the cert expires in 7 days or less is still an issue if Let's Encrypt is down.
There should be at most a warning but it should start. Otherwise you end up with an external dependency that can cause your web server to not start through no fault of your own.
The simple config syntax and sane defaults is nice in Caddy. For example, 3 lines of config nets a https server with http/2 support and an A rating with Qualsys for ssl setup.
Yes, that's exactly what I meant. You can tell because I didn't present any examples. And because one issue invalidates everything else. And I didn't express my opinion on this issue up thread.
I switched from nginx to caddy for my (extremely low-traffic) server because I was tired of having to copy & paste a bunch of SSL setup any time I was configuring a new domain.
I moved from Caddy to Traefik (https://traefik.io/) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.
Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1
If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes.
This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?
> If I am hosting 5 sites on caddy, and add a 6th one, I restart the server.
No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)
The idea here is that the server can't be scaled up or down. I suggest googling "cattle vs pets". If you know how to make a single process scale horizontally across additional hosts or scale up in alternative datacenters with a chef service notification, I'd pay you money to tell me how.
So, um: devops and cloud architecture is my job. I think I know what "cattle vs pets" is referring to. Nothing in this thread has anything to do with one versus many processes or one versus many nodes, nor does it have anything to do with manual configuration of anything. Rewrite the configuration based in orchestration data or, in extremity, upload a new version of a cookbook that handles that new site; the hypothetical sixth site is added and the service kicked over without human interaction. Scaling is an orthogonal concern.
I submitted the issue for it (https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.
This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this.
Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered.
What exactly do you do when your look-ma-no-hands server won't even start?
Edit: maybe "all-things-to-all-people" was the wrong term to use here.
I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims.
You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.
Indeed, the 0.10.3 fix compared to the situation with OCSP stapling in Apache is illustrative
This fix gets almost everybody where they should be, the next time the same thing happens (and it will) Caddy isn't a problem for three weeks, which is definitely enough time. Meanwhile we're going to see the same Apache crappiness for OCSP again each time until someone over there finally snaps out of it and asks someone who actually knows how OCSP stapling was supposed to work.
Wow @ that close comment:
> So, this is not a bug and all is working as intended.
Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!
I clicked the link, saw it was closed, and thought, "Wow, these guys are fast". Then I read it....
That's going to be a limiter for adoption. Hopefully @mholt reconsiders.
Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx.
Having the server be unable to start through circumstances outside of the system's control is just such a huge no.
Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far.
edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.
Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
Why did you switch away from nginx?
Not having to deal with certificate renewal is a big deal.
I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy.
Am I missing something?
And by running Caddy instead, I have one less piece to monitor and worry about.
The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
> The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
That's an odd thing to say in this particular conversation thread...
Why would you want to tightly couple your webserver to the availability of another service provider?
It's not tightly coupled any more than it would be with certbot. The issue I filed is an issue because it's unexpected behaviour (despite early closure, it has actually now been fixed).
> by running Caddy instead, I have one less piece to monitor and worry about
No, with Caddy you have several vaguely related pieces glued together with superglue.
> The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
I think this thread is pretty much proof that that approach will bite you in the ass.
> I think this thread is pretty much proof that that approach will bite you in the ass.
What bit me here is the fact that I'm running alpha software instead of a battle-tested web server; I'm doing so willingly, with full awareness of the risks that that entails.
Drawing the conclusion you did from the variables at play is shortsighted. If anything bites people in the ass, it's prejudice and shortsightedness. I wouldn't want you handling my ops/infrastructure.
> What bit me here is the fact that I'm running alpha software
This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were <= 30 days away from expiring and the CA can't be contacted.
> Drawing the conclusion you did from the variables at play is shortsighted
Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't working the way you want, you have to either a) replace it completely or b) work out how to disable the bit that's not working how you want, and replace that part of it.
> Drawing the conclusion you did from the variables at play is shortsighted
- People use a piece of software that serves as both ACME TLS certificate client and web server
- Said software by design won't start if the CA can't be contacted 30-days out from expiry
The conclusion I drew is that such integration leave the operator with less control than if they followed a separation-of-concerns approach, and left web serving to a web server, and TLS certificate renewal to an ACME client. The former doesn't need to care about how old the certificates are, just use what it's given.
You're drawing conclusions from unintended behaviour, which has now changed (and a release has been issued).
I still think that refusing to start if the cert expires in 7 days or less is still an issue if Let's Encrypt is down.
There should be at most a warning but it should start. Otherwise you end up with an external dependency that can cause your web server to not start through no fault of your own.
> unintended behaviour
Ahem. https://github.com/mholt/caddy/issues/1680#issuecomment-3026...
Emphasis mine:
> So, this is not a bug and all is working as intended.
Have you noticed that the bug has been fixed?
why do you keep calling it a bug?
The simple config syntax and sane defaults is nice in Caddy. For example, 3 lines of config nets a https server with http/2 support and an A rating with Qualsys for ssl setup.
> sane defaults
Failure to start if a CA is down is a "sane default" ?
Yes, that's exactly what I meant. You can tell because I didn't present any examples. And because one issue invalidates everything else. And I didn't express my opinion on this issue up thread.
Is it? I spent one day getting certbot up and running, and ever since then it's been pretty much a done deal.
That's 1 day that I didn't have to spend, even better when dealing with many instances.
Okay, I agree, with two caveats:
1. The cost does not scale with the amount of instances since it is the one-time cost to create the configuration package.
2. If you decide to go for Caddy instead, you'll have to spend the same time, if not more, learning Caddy.
I switched from nginx to caddy for my (extremely low-traffic) server because I was tired of having to copy & paste a bunch of SSL setup any time I was configuring a new domain.
I'm done with Caddy - this is security theater at its finest. Back to nginx we go. Why did I pick Caddy, because it was simple/easy/fast-to-setup.
I moved from Caddy to Traefik (https://traefik.io/) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.
You're not bit by this? https://github.com/containous/traefik/issues/791
Nope.
Or someone forks it
Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1
What if there's a reason to restart the server or spawn a new one?
Scaling, upgrading, disk issues, just migrating to new place... and foremost https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_com... Also nowadays with Docker around I rarely see server/app bundles supporting graceful reload.
This misses the overall point.
If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes.
This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?
> If I am hosting 5 sites on caddy, and add a 6th one, I restart the server.
No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)
This goes wholly against most cattle-not-pets devops philosophy though, right?
How so? In Chef:
The idea here is that the server can't be scaled up or down. I suggest googling "cattle vs pets". If you know how to make a single process scale horizontally across additional hosts or scale up in alternative datacenters with a chef service notification, I'd pay you money to tell me how.
So, um: devops and cloud architecture is my job. I think I know what "cattle vs pets" is referring to. Nothing in this thread has anything to do with one versus many processes or one versus many nodes, nor does it have anything to do with manual configuration of anything. Rewrite the configuration based in orchestration data or, in extremity, upload a new version of a cookbook that handles that new site; the hypothetical sixth site is added and the service kicked over without human interaction. Scaling is an orthogonal concern.
OK, there's a 0-day patch for OpenSSL, without it my users are vulnerable to RCE. Why can't I restart if LE's ACME is down?
Then wait until you see how caddy handles fully qualified domain names!
Caddy will just refuse to even handle them.
Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended.
Try out: https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ redirects to the relative domain
and then https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ )
Is there a bug for this? That's out of spec
Yes there is, @mholt said it’s a WONTFIX. You're supposed to script it yourself.
https://github.com/mholt/caddy/issues/1632
I submitted the issue for it (https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.
This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this.
Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered.
What exactly do you do when your look-ma-no-hands server won't even start?
Edit: maybe "all-things-to-all-people" was the wrong term to use here.
That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.
> That's a bit of a straw man
From https://caddyserver.com:
> The Most Beloved Server
They started the hyperbolic claims, not me.
I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims.
You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.
> You're claiming "caddy does everything". As opposed to what?
Serving content over http(s), and obtaining TLS certificates are two very different tasks.
> If you're running apache or nginx, your server does far more than caddy
Far more, that is directly related to serving content over http/https.
> Serving content over http(s), and obtaining TLS certificates are two very different tasks.
Except that with let's encrypt one actually needs the other.
And best part, according to the developer this is working as intended. A webserver with perfectly valid cached certificates refusing to start.
Updating to v0.10.3 should fix this problem and allow Caddy to start, provided your cert isn't less than 7 days from expiring: https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
Indeed, the 0.10.3 fix compared to the situation with OCSP stapling in Apache is illustrative
This fix gets almost everybody where they should be, the next time the same thing happens (and it will) Caddy isn't a problem for three weeks, which is definitely enough time. Meanwhile we're going to see the same Apache crappiness for OCSP again each time until someone over there finally snaps out of it and asks someone who actually knows how OCSP stapling was supposed to work.
Same with traefik: https://github.com/containous/traefik/issues/791