Application loads but customer data does not

Can I prove the application can reach its database?

The ticket

CUSTOMER TICKET: customer list is empty and will not load

Account: Beacon Analytics (growth) Impact: all users, core workflow unusable Started: after last night's maintenance window

The app itself seems to come up fine now, but the customer list never loads. We just get an error box that says the service is unavailable. Other pages that do not show customer data seem OK. We had someone in doing maintenance last night, if that helps.

Your job

1. Prove which layer is actually failing before you name a cause.
2. Restore service with the smallest correct change.
3. Verify the customer's own workflow, not just that something responds.

Working notes

The service is expected on http://127.0.0.1:8100. The customer workflow is GET /customers. Runtime configuration lives in labs/docker/_stack/compose.override.yaml. After editing it, run tse apply to recreate the services, then tse check.

Track

Docker

Time

about 30 minutes

Difficulty

Straightforward

Tier

Core

Do these first: Service unavailable after a release

Start it

In a Codespace or a local clone:

tse start docker/02-customer-data-not-loading

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Notice what the customer told you that the last ticket did not: the application comes up, and only one workflow is broken.

That is a genuinely different situation from a service that never started, and it rules something out for free. If the process is running and answering requests, the failure is not "the app is down". Something the app depends on is either unreachable, refusing it, or absent.

The smallest question now is not "is it running", because you already know it is. It is:

Can the application reach the thing it needs?

Read what the application itself says when the failing request comes in. It is running, so it is talking.

Hint 2 of 3

The application logs a database_connection_failed event with a detail message. Read the detail, not just the event name.

"Cannot reach the database" splits into at least four separate causes, and each has different evidence:

Cause	What would prove it
The database is down	The database container's own state and health
The name does not resolve	Resolving the name from inside the app container
Credentials are rejected	The database's own error text, which says so explicitly
Wrong database or schema	A successful connection but missing tables

Check the database's state first, because it is one command and it eliminates a whole branch. Then work out where the app is actually trying to connect. A hostname is only meaningful relative to where it is evaluated.

Hint 3 of 3

The database is healthy, so the problem is on the path between the two containers. Prove what the app is aiming at:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               exec app printenv DB_HOST

Now prove what that name means from inside the app container, which is the only place it matters:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               exec app getent hosts "$(docker compose \
                 -f labs/docker/_stack/compose.yaml \
                 -f labs/docker/_stack/compose.override.yaml \
                 exec -T app printenv DB_HOST | tr -d '\r')"

Compare that against the name Compose actually publishes for the database service. The value in the config is a perfectly valid hostname. It just does not point where the maintenance work assumed it did.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: customer data not loading

What the evidence proved

Command	What it proved	What it did not prove
docker compose ps	Both containers are up and the database is healthy	Nothing about whether they can talk
curl /customers	The app answers, and answers 503	Nothing about which dependency failed
docker compose logs app	The failure is database_connection_failed, with a connection-refused detail	Nothing about why the connection was refused
exec app printenv DB_HOST	The app is aiming at localhost	Nothing on its own, until you know where it is evaluated
exec app getent hosts localhost	Inside the container, localhost is the container itself

The database was never down, and the credentials were never wrong. Both of those are worth ruling out explicitly, because both are the first guess most people reach for.

Root cause

The maintenance change set DB_HOST to localhost. Inside a container, localhost is that container's own network namespace, so the application spent every request trying to connect to a database inside itself. Nothing is listening on port 5432 there, so the connection is refused immediately.

This is why the symptom looked like a database outage while the database was provably healthy: the app never reached it.

On the shared Compose network, the database is reachable by its service name, postgres, which Compose resolves through its internal DNS.

Scoped fix

In labs/docker/_stack/compose.override.yaml:

services:
  app:
    environment:
      DB_HOST: postgres

Then:

tse apply
tse check

Customer update

The customer list failure was caused by a connection setting changed during last night's maintenance. The application was pointed at the wrong address for the database, so those requests could not complete, which is why only the pages that read customer data were affected. We have corrected the setting and confirmed the customer list is loading. No data was lost or changed. If you notice any other pages still failing, tell me which ones and I will check them against the same setting.

Engineering escalation, if you needed one

Impact: GET /customers returning 503 for all users since the maintenance window. Evidence: app healthy and serving; database_connection_failed with connection refused; DB_HOST=localhost inside the app container; postgres container healthy and accepting connections on the Compose network. Confirmed: database availability, database credentials, app process health. Ruled out: database outage, credential rotation, schema change. Suspected cause: maintenance replaced the Compose service name with localhost, which resolves to the app container itself. Request: confirm whether the maintenance template is shared with other services that would have taken the same edit.

That last line matters. One misconfigured service is an incident. A shared template is an outage waiting for the next deploy.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The database reports itself healthy while the application fails every request. What has that health status actually proved?

• That the application is pointed at the right place.
• That the credentials in use are valid.
• That the fault has to be in the application's own code.
• That the database is up and answering for itself, and nothing at all about whether anyone else can reach it.

Question 2 of 3Inside a running container, what does the name `localhost` refer to?

• That container's own network namespace, so it resolves to the container itself and nowhere else.
• The host machine running the container runtime.
• Whichever service was started first in the same file.
• It is undefined and varies with the base image.

Question 3 of 3Having ruled out the dependency being down, which single piece of evidence moved this furthest forward?

• The error the customer saw.
• Both containers appearing in the process list.
• Reading the address the application was aiming at from inside the running process.
• The log line naming a failure to connect.

3 questions, none answered yet.

Why this one exists

Inside a container, localhost means that container, not the host and not another service. Name resolution is a separate provable layer from credentials and from availability.

In an interview

Cross-layer reasoning, which is where most candidates lose the thread. The application is genuinely healthy as a process and genuinely broken as a service. Being able to say "the app is running but cannot reach the database" rather than "the app is broken" is the difference between an escalation engineering can act on and one they have to redo.

Commands introduced

• docker compose exec
• docker inspect
• getent hosts
• printenv

Evidence layers

• container state
• application logs
• service name resolution
• runtime configuration