Every health check is green and the customer still gets nothing

Can I prove the customer's requests are reaching the application at all?

The ticket

CUSTOMER TICKET: the whole application is unreachable, but your status page says it is fine

Account: Halden Freight (enterprise) Impact: every user, nothing loads at all Started: this morning, after a change went out last night

Nobody can get in. Every page just fails to load, not an error message, it simply returns nothing. We have checked from three different offices and two phones on mobile data, so it is not us.

What I do not understand is that your own status page has been green the whole time, and the engineer we spoke to last night said the deploy came up clean and everything was reporting healthy. Something is clearly wrong somewhere, because we have had no service since about 6am.

Your job

1. Work out which layer is failing. This ticket does not tell you, and neither does the folder it is filed under.
2. Prove where the customer's request actually stops.
3. Restore service with the smallest correct change, then verify the workflow the customer described rather than the one that was already passing.

Working notes

The customer reaches this service at http://127.0.0.1:8100, and the workflow they mean is GET /customers.

Nothing here tells you which system is involved. Finding that out is part of the exercise, and it is a real skill: a ticket rarely arrives with the technology attached. When you have found the configuration that is wrong and changed it, run tse apply to recreate the services, then tse check.

Track

Mixed incidents

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Do these first: Same symptom as the last ticket, different cause, Same symptom as the last ticket, and the labels are correct, Batch job succeeds at the start and fails partway through, A report that was instant is now painful

Start it

In a Codespace or a local clone:

tse start mixed/01-every-check-is-green-and-nothing-works

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Start by taking the customer seriously on both halves of what they said, because both halves are true.

They cannot reach the service. The engineer who looked last night saw everything reporting healthy. People usually resolve that contradiction by deciding one side must be mistaken, and here neither is.

So the useful question is not "who is wrong". It is:

What exactly does a health check prove, and who was asking?

A health check is a question the system asks itself, from inside itself. The customer is asking a different question from much further away. Those two questions can have different answers at the same instant, and when they do, the gap between them is the fault.

Before you look at any configuration, work out how far the customer's request actually gets. Not whether the application is alive, which you already know it is. Where the request stops.

One more thing worth noticing: this ticket did not tell you what technology is involved, and the folder it is filed under will not either. Finding what is running is your first real step, and it is one command.

Hint 2 of 3

docker compose ls will tell you what is running and which files define it. That is where the configuration lives.

Now separate two claims that people usually merge into one:

Claim	What proves it	Who is asking
The application process is alive	The container's state	The platform
The application is serving requests	Its own health check	The container, from inside
A request from outside arrives	A request from outside	The customer

The first two are green. You can confirm that in one command. The third is the customer's ticket, and nobody has tested it yet.

So test it, and pay attention to how it fails. There is a real difference between these, and each one points somewhere different:

• Nothing is listening at that address, so the connection is refused outright.
• Something accepted the connection and then gave you nothing back.
• Something answered, and the answer was an error.

Get the exact failure, not a summary of it. Then compare two numbers: the address the customer is sent to, and the address the application says it is listening on when it starts up. The application announces that in its very first log line.

Hint 3 of 3

The connection is accepted and then closed with nothing sent. curl reports that as an empty reply on some hosts and as a connection reset by peer on others, depending on how your machine forwards published ports. Either wording means the same thing here, and both are a world away from a refusal.

That already rules a lot out: something is listening at that address, so this is not a service that failed to start, and it never got far enough to be an application error.

Look at what the published address actually points at:

docker ps --filter 'label=com.docker.compose.project=proveit-docker' \
          --format '{{.Names}}  {{.Status}}  {{.Ports}}'

Then look at what the application itself said when it started:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               logs app | head -1

One of those two numbers is where traffic is being sent. The other is where the application is actually listening. They do not match, which is why the connection is accepted by the forwarding layer and then dies with nothing behind it.

The health check never noticed because it runs inside the container and connects straight to the application, never crossing the mapping that is wrong.

The change is in labs/docker/_stack/compose.override.yaml, and it is one digit.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: every check is green and nothing works

What the evidence proved

Command	What it proved	What it did not prove
docker compose ls	What is running, and which files configure it	Nothing about the fault. This was orientation
docker ps	The container is up and its health check reports healthy	Nothing about whether anyone outside can reach it
curl http://127.0.0.1:8100/customers	The connection is accepted and closed with no reply	Nothing about which side closed it
	Reported as an empty reply or as a connection reset depending on the host, and neither is a refusal
docker compose logs app	The application started cleanly and is serving requests on 8080	Nothing about which address those requests arrive on
docker ps --format '{{.Ports}}'	Traffic to 8100 is forwarded to 8081 inside the container

The last two lines are the whole diagnosis, and neither means anything alone. The application is listening on one port. The published mapping delivers to a different one. Every request is handed to a port with nothing behind it.

Worth stating plainly, because it is the habit this exercise is built to break: the health check passing was never evidence that the customer could reach anything. It runs inside the container and connects directly to the application, so it never crosses the mapping that is broken. It was telling the truth about a narrower question than the one being asked.

Root cause

Last night's change edited the published address and set the container-side target to 8081. The application listens on 8080 and always has.

The forwarding layer accepts connections on 127.0.0.1:8100 because the mapping exists, then tries to deliver them to 8081 inside the container, where nothing is listening. The connection is closed with nothing sent, which is why the customer described it as pages returning nothing rather than as an error.

Everything inside the container stayed healthy throughout, because nothing inside the container was wrong.

Scoped fix

In labs/docker/_stack/compose.override.yaml, point the mapping at the port the application actually listens on:

services:
  app:
    ports: !override
      - "127.0.0.1:8100:8080"

Then:

tse apply
tse check

!override is there because Compose adds published ports from a second file to the ones in the first rather than replacing them. Without it you get two mappings competing for the same published address, which fails in a new and more confusing way.

Customer update

The outage was caused by a configuration change in last night's release. The address our platform publishes for your service was pointed at the wrong internal port, so connections were accepted and then dropped before reaching the application. That is why pages returned nothing rather than an error, and why our own health checks stayed green: those run inside the service and were not affected by the change. We have corrected the address and confirmed your user list is loading. No data was affected at any point. We are also reviewing why our external checks did not catch this, since your experience should not have been the first signal.

Engineering escalation, if you needed one

Impact: total loss of service for all users from roughly 06:00, every request accepted and closed with no response. Evidence: container healthy throughout; application logs show a clean start and port=8080; published mapping is 127.0.0.1:8100->8081/tcp; requests to 8100 return an empty reply. Confirmed: application process health, application startup, database availability. Ruled out: application crash, dependency failure, credential or data problem. Suspected cause: the release changed the container-side target of the published port from 8080 to 8081. Request: our health checks cannot see this class of fault, because they run inside the container. Can we add a check that connects from outside the published address, so the next one is caught by us rather than by a customer.

That last request is the part that stops this recurring. The fix takes one digit. The monitoring gap is what let it run from 06:00 until a customer complained.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The container health check reported healthy for the entire outage, and it was telling the truth. What had it actually established?

• That the application was serving requests to anything that could already reach it inside its own container
• That the whole request path from the customer to the application was working
• That the application contained no defects
• That the published address had been configured correctly

Question 2 of 3Requests came back as an empty reply rather than as a refused connection. Why is telling those two apart worth the effort?

• A refusal means nothing accepted the connection, so an accepted one that returns nothing proves something is listening and forwarding
• They mean the same thing, and only the wording differs between tools
• An empty reply always means the application crashed while responding
• A refusal indicates a firewall and an empty reply indicates a certificate problem

Question 3 of 3The fix was a single digit. What is the honest thing to raise with engineering afterwards?

• That the release changed a value nobody reviewed
• That the health check should run more often
• That every monitoring signal was green while the service was completely unavailable, so the monitoring cannot see this class of fault
• That the customer should have reported it sooner

3 questions, none answered yet.

Why this one exists

A health check answers a narrower question than the customer asked. It runs inside the thing it is checking, so it proves the application is serving itself, not that anyone outside can reach it. Reachability is its own layer with its own evidence.

In an interview

Almost every candidate has a first move, and this rewards noticing when the first move came back clean. Saying "the process is up, its own health check passes, and the request never reaches it, so the problem is between the customer and the process" is a genuinely different sentence from "it looks fine to me", and it is the one that moves a ticket forward.

Commands introduced

• docker compose ls
• docker port

Evidence layers

• published address
• container state
• application logs
• what the application says it bound to