Service unavailable after a release

Can I prove the application started at all?

The ticket

CUSTOMER TICKET: dashboard unavailable since this morning

Account: Northwind Freight (enterprise) Impact: all users, complete outage Started: shortly after the 09:14 release

Nobody on our team can load the customer dashboard. It spins for a while and then times out. It was working fine yesterday afternoon and we have not changed anything on our side. We have a board review at 14:00 and this is the data we present from. Please treat as urgent.

Your job

1. Prove what is actually failing before you name a cause.
2. Restore service with the smallest correct change.
3. Verify the customer's own workflow, not just that something responds.

Working notes

The service is expected on http://127.0.0.1:8100. The customer workflow is GET /customers. Runtime configuration lives in labs/docker/_stack/compose.override.yaml. After editing it, run tse apply to recreate the services, then tse check.

Track

Docker

Time

about 25 minutes

Difficulty

Gentle

Tier

Core

Start it

In a Codespace or a local clone:

tse start docker/01-service-unavailable-after-deploy

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

You have a customer symptom, not a fact. "The dashboard times out" is consistent with at least four different failures, and you have not yet ruled out the cheapest one.

Before you think about the application code, the database, or the network, ask the smallest possible question:

Did the application process ever start?

There is a meaningful difference between a service that is running and answering incorrectly, one that is running and hung, and one that never came up at all. Each leaves different evidence. Start with the evidence that takes one command and no assumptions.

Hint 2 of 3

docker ps only shows what is running right now. A process that exits immediately on startup will never appear there, which makes the service look absent rather than broken. That distinction matters, because "it is not running" and "it refuses to start" lead to completely different next steps.

Two moves:

1. Widen the view so stopped and restarting containers are included.
2. Read what the process printed on its way out.

A process that fails a precondition usually says so in its last line of output before exiting. You are looking for the application's own words, not Docker's.

Hint 3 of 3

Run these in order and read each result before moving on:

docker ps -a

The app container is cycling rather than staying up. Now read its output:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               logs --tail 20 app

The final line names the exact precondition the application refused to start without. Compare that name against what the release actually applied:

cat labs/docker/_stack/compose.override.yaml

The value is present in the configuration, which is why nothing looks obviously missing. Look at what it is set to, not whether it is set.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: service unavailable after a release

What the evidence proved

Command	What it proved	What it did not prove
docker ps	The app container is not running now	Nothing about whether it ever ran
docker ps -a	The container exists and is cycling, so the image resolved and the container was created	Nothing about why it exits
docker compose logs app	The process exits deliberately with required environment variable APP_SECRET is not set	Nothing about who changed it
cat compose.override.yaml	APP_SECRET is present but set to an empty string	Nothing about intent

The important distinction: the variable was not missing. It was declared and empty. That is why a quick scan of the release diff looks fine, and it is why the application treats it as unset.

Root cause

The release applied an override that declared APP_SECRET with an empty value. The application validates required configuration at startup and exits rather than running in an unknown security state. The restart: on-failure policy then retried it, producing a container that appears intermittently rather than a clean, obvious failure.

Scoped fix

In labs/docker/_stack/compose.override.yaml:

services:
  app:
    environment:
      APP_SECRET: "lab-secret"

Then:

tse apply
tse check

Nothing else needs to change. Resist widening the fix: the database, the port mapping, and the image were all proven healthy by the evidence above.

Customer update

The dashboard outage was caused by a configuration value that was cleared during this morning's release. The application is designed to stop rather than start with incomplete security configuration, which is why it never came back up. We have restored the value and confirmed the customer dashboard is loading your data correctly. No data was lost and nothing on your side needs to change. I will follow up by 13:00 to confirm it is still stable ahead of your 14:00 review.

Note what this does not say: it does not blame a person, it does not speculate about how the value was cleared, and it does not promise it cannot recur.

Engineering escalation, if you needed one

Impact: complete outage of the customer dashboard from 09:14. Evidence: app container in a restart loop; startup log line ERROR: required environment variable APP_SECRET is not set; the applied compose override declares APP_SECRET: "". Confirmed: image resolved, container created, database healthy and reachable. Ruled out: image tag, port mapping, database availability. Suspected cause: the release pipeline emitted an empty value for a required secret rather than failing the deploy. Request: confirm whether the pipeline should fail closed when a required secret resolves empty.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The process list shows nothing for the app. What has that established?

• Only that it is absent at this instant, which is equally consistent with never having started and with dying in a loop.
• The image failed to download from the registry.
• The application crashed.
• The host has run out of resources.

Question 2 of 3Startup fails saying a required setting is not set, yet the override file plainly declares it. How are both true at once?

• The application reads the value before the override is applied.
• The file has a syntax error that stops it from loading.
• It is declared with an empty value, and empty is not something the application will accept.
• The name is misspelled inside the application.

Question 3 of 3Why did this reach the customer as an intermittent outage rather than as a service that was plainly down?

• Traffic was being balanced between a healthy instance and a broken one.
• The restart policy kept bringing it back, so a fresh container genuinely existed for a moment before exiting again.
• The health check was reporting the wrong status.
• A cache was serving some of the requests successfully.

3 questions, none answered yet.

Why this one exists

A running container is not a healthy application, and an absent container is not a missing image. Separate "never started", "started and exited", and "running but unhealthy" before you form a hypothesis.

In an interview

The most common opening move a candidate gets wrong. "The site is down" invites a guess about the application, when the cheapest available evidence proves whether a process ever started. Interviewers watch for whether you widen from running containers to all containers, and whether you read the exit output before naming a cause.

Commands introduced

• docker ps
• docker ps -a
• docker compose ps
• docker compose logs
• docker inspect

Evidence layers

• container state
• container logs
• runtime configuration