The new address fails and the old one works

Can I prove the server is refusing the name rather than the caller?

The ticket

CUSTOMER TICKET: the address you migrated us to fails on every call

Account: Ardent Logistics (enterprise) Impact: nightly export failing again, one night so far Started: the evening we switched to the new address

Your migration note asked us to point our export at the new address. We did that yesterday afternoon and last night's run failed on every attempt.

Before you ask: we checked. Your team fixed something like this for us in April and told us the date to watch was in 2035. We looked, and it is still 2035, so it is not that again.

We also pointed the job back at the old address as a test and it went through immediately, first try, no changes to anything else. So the service is up, our credentials are fine, and our job works. It is only the address you asked us to move to.

We would rather not stay on the old address if you are retiring it. What do you need from us?

Your job

1. Confirm what the customer already told you, rather than assuming it. Both addresses, same evidence, side by side.
2. Read the error properly. It is not the same failure as April, and one line is the only thing that says so.
3. Fix the new address without breaking the old one. They are both in use.

Working notes

The customer's integration runs in the client service, and GATEWAY_URL controls which address it calls:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client /app/upload.sh

Both the gateway's configuration and the address the client uses are in labs/networking/_stack/compose.override.yaml. After editing, run tse apply, then tse check.

Moving the customer back to the old address is not a fix. The grader checks the new one, and the customer has already asked you not to.

Track

Networking, DNS, TLS

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Do these first: The nightly upload stopped and nothing changed

Start it

In a Codespace or a local clone:

tse start networking/02-the-new-address-fails-and-the-old-one-works

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer has done half your job. Do the other half before theorizing.

They told you three things: the new address fails, the old address works, and nothing else changed between the two attempts. All of that is checkable in about a minute, and checking it is worth more than believing it, because if any one of those is wrong you are solving a different problem.

Run the upload as it is configured now. Then run it again against the old address, changing nothing else. GATEWAY_URL is what the client reads, so you can override it for a single run without editing anything.

One fails, one succeeds, and they are the same client, the same credentials, the same payload, and the same server. That last part is the one to sit with. There is one gateway. Both addresses arrive at it.

So nothing about the service can explain this. Whatever is different has to be something that depends on which name you asked for, and there are not many things in a connection that do.

Now read the failure. You have seen a failure from this service before and this is not it. Compare the two errors line by line rather than by shape. They print the same code and the same closing paragraph, and they are different faults.

Hint 2 of 3

The two failures, side by side. April's:

curl: (60) SSL certificate OpenSSL verify result: certificate has expired (10)

And this one:

curl: (60) SSL: no alternative certificate subject name matches target hostname 'reports'

Same exit code. Same paragraph underneath. Completely different fault, and the only thing that says so is that line. This is why "it is the certificate again" is a guess rather than a diagnosis, and why the customer was right to push back.

April's failure was about when. This one is about who.

A certificate is a claim that a particular server owns particular names, signed by an authority the caller trusts. The caller does two things with it: checks the dates, and checks that the name it asked for is one of the names on the certificate. The second check is the one failing, and it is failing because of the name in the request rather than anything about the server.

So ask what names the certificate actually carries:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client sh -c \
  'echo | openssl s_client -connect reports:8443 -servername reports 2>/dev/null \
     | openssl x509 -noout -subject -dates -ext subjectAltName'

The dates are years away, exactly as the customer said. Read the last line.

Then ask the question that makes this an incident rather than a typo: the service answers to both names, and somebody made that true. Was anything else updated when they did?

Hint 3 of 3

The evidence:

subject=C=US, O=Prove It Lab, CN=gateway
notBefore=Apr  1 00:00:00 2025 GMT
notAfter=Dec 31 23:59:59 2035 GMT
X509v3 Subject Alternative Name:
    DNS:gateway

One name. The customer was told to call reports, and the certificate proves ownership of gateway and nothing else. Their client asked for a name the server cannot prove it owns, so it stopped, which is exactly what it is for.

Now the part worth remembering, because this is how it actually happens. Look at what the CA has issued:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec gateway sh -c \
  'for c in /certs/v*.pem; do echo "$c"; openssl x509 -in "$c" -noout -dates -ext subjectAltName; done'

v1, the one that expired in April, covered both names. v2, the reissue that replaced it, covers one. The rotation that fixed April's incident silently narrowed what the certificate proved, and nothing failed at the time because nobody was using the second name yet. The migration is what made it visible, a week later, looking like a migration problem.

v3 covers both names and runs to 2035. In compose.override.yaml:

services:
  gateway:
    environment:
      GATEWAY_CERT: /certs/v3.pem

Then tse apply and tse check.

Do not fix this by reissuing for reports alone. It would pass the first check and break every caller still on gateway, which is most of them, and you would have moved the outage rather than ended it. The grader tries both names for that reason.

And do not move the customer back to the old address. It works, it is tempting, and it leaves them on something you have told them you are retiring.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the new address fails and the old one works

What the evidence proved

Command	What it proved	What it did not prove
/app/upload.sh against reports	Exit 60, no alternative certificate subject name matches target hostname	Nothing about the dates, which is what the code alone suggests
The same upload against gateway	Accepted immediately, same client, same credentials, same payload
openssl x509 -dates on what is served	Valid until 2035, so the customer was right
openssl x509 -ext subjectAltName	The certificate covers gateway and nothing else
The same read across v1, v2 and v3	v1 covered both names. v2 covers one

The two uploads are the whole diagnosis and they have to be run together. One address fails and one succeeds against the same server, so nothing about the service can account for it. Whatever differs has to depend on which name was asked for, and in a verified connection there is exactly one thing that does.

The other piece is the error text. This is the second time this customer has seen curl: (60) with the same four-line paragraph under it, and it is a different fault both times. April's line said certificate has expired. This one names a hostname. Reading the code and stopping is how the customer's own theory ("it is that again") gets adopted without being tested.

Root cause

The gateway presents v2.pem, whose subject alternative names list gateway and nothing else. The customer was migrated onto reports, which reaches the same process through a second name on the same service.

A certificate is a claim that a server owns particular names. Adding a name to a service does not add it to the certificate, so the new address was unverifiable from the moment it existed, and only became visible when somebody was told to use it.

The reason it is v2 and not something older is the part worth carrying forward. v1, the certificate that expired in April, covered both names. The reissue that replaced it covers one. That rotation fixed the incident in front of it and narrowed the certificate at the same time, and nothing failed for a week because nobody was calling the second name yet.

April's fix caused this. Not through carelessness about the thing being fixed, but by changing something adjacent that nobody was watching.

Scoped fix

In labs/networking/_stack/compose.override.yaml:

services:
  gateway:
    environment:
      GATEWAY_CERT: /certs/v3.pem

Then:

tse apply
tse check

v3.pem was issued by the same internal CA, runs to the end of 2035, and covers both names. Nothing changes for callers on either address.

Not the fix: reissuing for reports alone. It clears the customer's failure and breaks every caller still on gateway, which is most of them. The check tries both addresses because moving an outage is the most common way this one gets closed as resolved.

Also not the fix: moving the customer back to the old address. It works, it takes ten seconds, and it parks them on something you have told them you are retiring. That is a ticket reopened in a month with less goodwill.

Customer update

You were right that it was not the same problem as April, and thank you for checking before you raised it. The certificate we serve is valid until 2035, exactly as we told you.

What went wrong is different. A certificate lists the specific addresses a server is allowed to be called by, and the one we were serving listed only the old address. When you moved to the new one, your job did the correct thing and refused to continue, because we could not prove that address belonged to us. Your export, your credentials, and the service itself were all fine, which is why the old address kept working.

This was ours. The certificate we issued in April to fix your previous incident was created without the second address on it, and that gap only became visible when we asked you to move. We have replaced it with one covering both, so the new address now works and the old one is unaffected.

Please go ahead and move back to the new address when convenient. You do not need to change anything else, and we will confirm tonight's export landed.

Engineering escalation, if you needed one

Impact: one nightly export lost for Ardent Logistics after we asked them to migrate. The migration instruction was ours and the address we sent them to was unusable from the moment it was published. Evidence: no alternative certificate subject name matches target hostname 'reports' on the new address; the same upload accepted on the old one; the served certificate lists DNS:gateway only. Confirmed: one service behind both names, valid dates, healthy gateway. Ruled out: expiry, credentials, the customer's client, the payload. Suspected cause: the April reissue dropped reports from the alternative names. The gap existed for a week and was invisible until the migration. Request: two things. Nothing compared the names on the new certificate against the names on the one it replaced, and nothing checks that every address we publish is covered by what we serve on it. The first would have caught this at issue time, the second at migration time, and we currently have neither.

The rotation is a one-line fix. The durable part is that a reissue narrowed a certificate and no check noticed, which will happen again on a different name unless something compares the two.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The same upload failed on one address and succeeded on the other, with one gateway behind both. What did that establish?

• That the new address was routed to a different server that had not been configured yet
• That whatever differs has to depend on the name that was asked for, since nothing else about the two attempts differed
• That the customer's client was caching a connection to the old address
• That the new address was not yet resolvable

Question 2 of 3Both this failure and April's printed `curl: (60)` and the same four-line paragraph. What follows from that?

• That the exit code identifies the class of problem, and the line above the paragraph is what identifies the problem
• That it is the same underlying problem recurring, since the same failure produces the same output
• That the client should be configured to report more detail
• That exit codes are unreliable and should be ignored in favor of the message

Question 3 of 3The certificate that expired in April covered both names. Its replacement covered one. Why does that matter beyond this incident?

• Because it means the April incident was diagnosed incorrectly
• Because a fix narrowed something adjacent that nothing was watching, and the failure surfaced a week later looking like an unrelated problem
• Because certificates should never be rotated while a migration is planned
• Because the customer should have been told to stay on the old address

3 questions, none answered yet.

Why this one exists

A certificate proves a server owns particular names, not that it is the right server. Adding a name to a service is a change to the service, and unless the certificate is reissued to cover it, the new name is unverifiable even though it reaches exactly the same process. The most common way this arrives is a reissue that quietly drops a name nobody was tracking.

In an interview

This is the follow-up question after somebody answers the expiry one. Two failures that print the same code and the same paragraph, with one line between them, and being able to say what that line means is the difference between having read an error and having recognized one. It also tests whether you notice that a service answering to two names is not the same as a server that can prove it owns both.

Commands introduced

• openssl x509 -ext subjectAltName
• openssl s_client -servername

Evidence layers

• the error the client returned, and how it differs from last time
• both addresses tried against the same server
• the names the server's certificate actually covers