You can run a service for years without any problems, and then you add TLS, and then everything changes.
At my job, we operate an internal PKI. We have a Root CA, which signs Intermediate certificates, which in turn sign all the hosts certs in use. This week, one of the Intermediate certificates expired, which in itself could be a major disaster, but hey - we were prepared!
- we knew it would happen on that day
- two years ago, a new Intermediate certificate was generated
- all host certificates in use can be verified with the new Intermediate cert
So, the expiration of the old Intermediate cert was expected to be a non-issue. The expire time came and went, and very soon services left and right started to fail with certificate verification errors - WHAT?!?
It was clear that it had to have something to do with the expired cert, but what and how?
As we scrambled to remove the expired cert from countless systems, certificate bundles, trust stores, container images etc. - which is a lot harder to do than to add a certificate - the affected services recovered.
When everything had cooled down a bit I tried to find out what went wrong, and soon I found the culprit:
A certificate - in general - is a public key that, among other metadata, is signed by the private key of a certificate authority. You can then use the CA’s own certificate to verify this signature.
What happened in our case was that two years ago the new Intermediate certificate was created from the same key that was used for the old Intermediate certificate. So any host certificate that was signed by the Intermediate key could by verified with the old Intermediate certificate - the signing key information is the same, just the validity period is different.
As both Intermediate certificates were distributed and deployed to all systems, any TLS verification could use either one - whichever happened to be found first. This worked great, until it didn’t.
Lesson learned: don’t recycle your private keys. Generate a new one, so that any certificate you sign with it will have to be verified with the correct Intermediate certificate.