This afternoon on September 30th 2021 at 16:00 CEST, the root certificate that signed Letsencrypt, named X3, expired. The consequences should have been minor, since most if not all browsers were already using the new root, named X1.
On the serverside as a server
Sadly, some customers of Octopuce were having big issues : some website crontabs (regular tasks run on servers) stopped working, and some web shops card payment processor stopped working suddenly.
We debugged that situation, and as of now (2021-09-30 20:00 CEST) we think it was caused by a few simultaneous issues that, together, created that major problem.
The situation was as follow:
- We have a server certificate created by a not so recent certbot (latest 0.x). In this process (named ACME) the ACME server of Letsencrypt is sending us a certificate CHAIN, which contains BOTH the intermediate cert (R3) AND the other intermediate (X1) signed by the former root (X3).
- This intermediate certificate is now signed by an EXPIRED root (X3)
- Some of our providers (here a payment card company) still have X3 in their certificate store
- Those providers are using cURL to access our server using HTTPS.
cURL, the HTTP client used by people to automate connections, have a very thorough certificate validation process: if any certificate path is expired or invalid, it refuses to go through.
Our client’s payment provider was therefore unable to connect to our client’s website to send payment callbacks !
- Our solution is currently to remove the X1 certificate signed by the X3 expired root from our certificate chain in the web server. (This breaks compatibility with some older computers like androids <7.1.1)
- Another solution would be to update certbot to 1.6 minimum, and use the –preferred-chain option to ask for a chain containing the R3 cert or the X1 signed by X1, and not the X1 signed by X3.
- The best way to fix this would be that our client’s payment provider should remove the X3 expired certificate from its servers (but of course, we have no leverage in that)
- A last solution would be that letsencrypt should NOT send a certificate chain with the X1 signed by (now expired) X3, and only the intermediate R3 certificate. They did chose to do that because they wanted to keep the compatibility with Android <7.1.1 for as long as possible.
As a managed hosting provider, those last few hours were… challenging ;) finding a solution that could work both for our customer (keeping their compatibility as high as possible) and for their (problematic) payment providers, were a challenge. We found a solution within a few hours, and fixed this (temporarily) via our puppet deployment system. A definitive solution will be looked at in the next few days.
On the server side, as a client
Another consequence we had, since some of our servers are acting as HTTPS client, we had to remove the DST-X3 root certificate from our old ca-certificate files, since it’s expired. To do that, we had to :
- Edit
/etc/ca-certificates.confand add a!at the beginning of the line for theDST-X3 Root. - launch
update-ca-certificatesas root to update the system-wide list. - restart any service that uses those certificates, like php7.4-fpm, postfix …
This worked like a charm until …
… we tried to update a matomo instance for one of our customers… Matomo refused to do it, saying their website https certificate were expired. We found that Matomo have its own ca-certificate list from https://curl.haxx.se/ca/cacert.pem and sadly, this cacert.pem STILL HAVE the DST-X3 expired root :( So we removed it manually from our copy of this file and the update went fine after that…
We think we are not out of trouble … #MoreToComeSoon
PS: since then, we decided to Package ACME.SH for Debian …