The nightly upload stopped and nothing changed

Can I prove what the server is presenting, rather than that it is up?

The ticket

CUSTOMER TICKET: nightly export has not uploaded for two nights

Account: Ardent Logistics (enterprise) Impact: two nightly exports not delivered Started: overnight, with no release on either side

Our nightly export has not gone through for two nights running. Nothing was deployed on our side, nothing was deployed on yours as far as we can tell, and your status page has been green the whole time.

The job says it could not establish a secure connection and gives up. We have not changed the address, the credentials, or the schedule. This has run untouched every night since March.

One of our engineers opened the address in a browser and it loaded fine. It did ask whether he wanted to continue first, but he clicked through and the page came up, so the service is clearly running.

We are two nights behind on reporting now. What changed on your side?

Your job

1. Prove what the gateway is presenting to callers, rather than proving it is running. Those are different claims and only one of them is in question.
2. Answer the customer's question honestly. Nothing changed, and it broke anyway. Be able to say why that is possible.
3. Restore the upload without weakening what the client checks.

Working notes

The customer's integration runs in the client service. You can run it on demand:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client /app/upload.sh

The gateway's configuration is in labs/networking/_stack/compose.override.yaml. After editing, run tse apply, then tse check.

Everything runs inside these containers rather than from your own shell, so what you see is what the customer's integration sees.

Making the client stop checking would clear the error. Do not. The grader inspects what the gateway serves, so it would not pass, and it is the wrong answer anyway.

Track

Networking, DNS, TLS

Time

about 25 minutes

Difficulty

Straightforward

Tier

Core

Start it

In a Codespace or a local clone:

tse start networking/01-nightly-upload-stopped-and-nothing-changed

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Start by believing the customer.

They say nothing changed. Your instinct is to disbelieve that, because almost every incident starts with somebody saying it and being wrong. Here they are right, and the sooner you accept it the faster this goes.

So ask the question that follows:

What breaks with no change on either side?

Not much does. Something that ran out. Something that filled up. Something that had a deadline nobody was watching. That is a short list, and it is a much better list than the one you get from hunting for a deployment that does not exist.

The other thing worth taking seriously is the browser. Their engineer opened the address and it loaded, and that is true too. But read what they actually described: it asked whether he wanted to continue, and he clicked through. A person can agree to proceed. An automated client has nobody to ask, so it does the only thing it can and stops.

That difference is a clue about where the failure is happening. It is not in the service. It is in what the caller decided about the service.

Run the upload and read the error properly before going further.

Hint 2 of 3

Rule out the boring things first, then ask the port what it is serving.

Does the name resolve, and does the port accept?

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client getent hosts gateway

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client nc -z -w 3 gateway 8443

Both work. So this is not a name that stopped resolving and not a port that stopped listening, which between them cover most of what people mean when they say a network problem.

Now look at the gateway's own log:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               logs gateway

One line, from startup. Nothing about the failed uploads at all.

That is evidence, not the absence of it. Two uploads failed and the server has no record of either. A request that had arrived would have been logged, so the connection is being abandoned before any request is sent. The caller is deciding to hang up. That is why the customer's status page is green and their engineer's browser worked: nothing is wrong with the service.

So ask the caller's question directly. Connect to the port and read what comes back before any request is sent:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client sh -c \
  'echo | openssl s_client -connect gateway:8443 -servername gateway 2>/dev/null \
     | openssl x509 -noout -subject -dates -ext subjectAltName'

Read the dates.

Hint 3 of 3

The evidence, side by side:

curl: (60) SSL certificate OpenSSL verify result: certificate has expired (10)

subject=C=US, O=Prove It Lab, CN=gateway
notBefore=Jan  1 00:00:00 2025 GMT
notAfter=Apr  1 00:00:00 2025 GMT

The gateway is presenting a certificate that stopped being valid on the first of April 2025. It presents it perfectly happily: a server has no opinion about whether its own certificate is still good, it just serves what it was given. The client is the one that checks, and the client is the one that stopped.

That is the answer to the customer's question. Nothing changed. The certificate had a date on it and the date passed. A deadline nobody was watching is one of the few things that can break a system with no deployment on either side, and this is the most common example of it in production.

The gateway holds every version the internal CA has issued:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec gateway ls -1 /certs

Check the dates on each one before choosing, rather than assuming the highest number is the right answer:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec gateway sh -c \
  'for c in /certs/v*.pem; do echo "$c"; openssl x509 -in "$c" -noout -dates; done'

The fix is in labs/networking/_stack/compose.override.yaml:

services:
  gateway:
    environment:
      GATEWAY_CERT: /certs/v2.pem

Then tse apply and tse check.

Do not fix this by making the client skip the check. It would work tonight, it would work every night, and the day somebody put a genuinely wrong server in front of that address the customer would upload their data to it without a complaint. The check exists to fail. Ours failed correctly.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the nightly upload stopped and nothing changed

What the evidence proved

Command	What it proved	What it did not prove
/app/upload.sh	The client stops with exit 60 before sending anything	Nothing about why it refused
getent hosts gateway	The name resolves	Nothing about what answers on it
nc -z gateway 8443	The port accepts connections	Nothing about what it serves
docker compose logs gateway	The server has no record of either failed upload
openssl s_client piped to x509 -dates	It presents a certificate that expired on 1 April 2025

The empty gateway log is the piece worth dwelling on, because it is the one that reads as a dead end and is not. Two uploads failed and the server logged nothing about either. That is not a logging gap. A request that reached the application would have been logged, so no request was ever sent. The client opened the connection, looked at what came back, and hung up before asking for anything.

Which is also why every signal the customer had was green, and why they were right about all of them. The service is running. The port is open. The status page is honest. Nothing is wrong with the server, and the server is not the thing that failed.

Root cause

The gateway was configured to present v1.pem, which the internal CA issued with a validity window of 1 January 2025 to 1 April 2025. That window closed.

A server does not check its own certificate. It presents whatever file it was pointed at, on every connection, forever. The check happens at the other end, which is why the failure appeared at the customer with no change at either site, and why it appeared overnight rather than at a deploy.

The browser test their engineer ran is the same fact from the other side. The browser also refused, then offered him the choice of continuing anyway, and he took it. An unattended client has no one to ask.

Scoped fix

In labs/networking/_stack/compose.override.yaml:

services:
  gateway:
    environment:
      GATEWAY_CERT: /certs/v2.pem

Then:

tse apply
tse check

v2.pem was issued by the same internal CA in April 2025 to replace v1, and runs to the end of 2035. The client already trusts that CA, so nothing changes on the customer's side at all.

Not the fix: making the client skip the check. It clears the error immediately and permanently, and it removes the only thing standing between the customer and uploading their data to whatever happens to answer on that address in future. The check did its job. It is the thing that noticed.

Also not the fix: picking the highest version number without reading it. v3.pem is also valid here and would also have worked, which is exactly the habit that causes the next exercise.

Customer update

Your uploads were failing because the certificate our gateway presents had reached its expiry date. Your integration checks that certificate before it sends anything, which is correct and is why it stopped rather than uploading to something it could not verify.

You were right that nothing changed. Certificates are issued with a fixed end date, and this one passed its date overnight, so the same configuration that worked in the evening did not work in the morning. That is also why our status page stayed green: the service itself was healthy throughout, and the failure happened at your end of the connection before any request reached us.

Your engineer's browser test tells the same story. The browser refused too, then offered him the option to continue anyway and he accepted it. Your nightly job has nobody to ask, so it stops, which is the behavior you want.

We have rotated the gateway onto a current certificate issued by the same authority, so nothing needs to change on your side. Tonight's export will run normally. We can also send the two exports you are missing.

Engineering escalation, if you needed one

Impact: two nightly exports not delivered for Ardent Logistics. The customer holds no data and had no failing signal on either side to look at. Evidence: the client exits 60 with certificate has expired; the gateway serves a certificate with notAfter=Apr 1 00:00:00 2025 GMT; the gateway logged nothing for either attempt. Confirmed: the name resolves, the port accepts, the service is healthy. Ruled out: a deployment on either side, credentials, the schedule. Suspected cause: the gateway was left pointed at v1.pem when it was replaced, so the rotation was issued but never picked up. Request: nothing monitors the expiry of what the gateway serves, and nothing alerts on a handshake that fails before a request. Both were silent for two nights. Can we get an expiry check on what is actually being presented, not on what we believe is configured.

The rotation is a one-line fix and it is not the durable part. A certificate that was issued and never deployed will happen again, and the reason nobody noticed for two nights is that every dashboard was measuring the server.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3Two uploads failed and the gateway's log had one line in it, from startup. What did that establish?

• That no request ever reached the application, so the connection was abandoned before one was sent
• That the gateway's logging was misconfigured and needs fixing before the incident can be diagnosed
• That the uploads never left the customer's network
• That the gateway was down during both attempts and recovered before anyone looked

Question 2 of 3Their engineer opened the address in a browser and the page loaded, after clicking through a prompt. What did that prove?

• That the problem is in the customer's job rather than in anything on your side
• That the service was running, which was never in question, and nothing about whether an unattended caller would accept it
• That the failure was intermittent, since one client succeeded and another did not
• That the browser and the job were reaching different servers

Question 3 of 3Turning off the client's verification would have fixed tonight's upload and every upload after it. Why is that not the fix?

• It would be slower to roll out than rotating the certificate
• It would break the customer's other integrations
• It removes the check that detected the problem, so the next time something wrong answers on that address the data is uploaded to it without complaint
• It would not actually have worked

3 questions, none answered yet.

Why this one exists

A service can be entirely healthy and still be unusable, because whether a caller accepts what a server presents is decided by the caller. A failure at that stage never reaches the application, so the server logs nothing and its own health checks stay green. Silence in the server log is evidence about where the failure happened, not evidence that there was not one.

In an interview

Everybody can say a certificate expired. The question that separates answers is what you did before you knew that, because at the start this looks like a network problem, a firewall change, or an outage. Being able to say that the name resolved, the port accepted, the service logged nothing, and the failure was therefore on the client side of the connection is the part worth hearing.

Commands introduced

• getent hosts
• nc -z
• openssl s_client
• openssl x509

Evidence layers

• the error the client returned
• whether the name resolves and the port accepts
• what the server presents to a caller
• the dates on it