Same symptom as the last ticket, different cause

Can I prove why the database refused the connection?

The ticket

CUSTOMER TICKET: customer list is empty and will not load

Account: Copperline Robotics (growth) Impact: all users, core workflow unusable Started: this morning

Same thing your team fixed for Beacon last week, I think. The app loads but the customer list just shows service unavailable. Our security team did a scheduled rotation overnight, but that is supposed to be routine and it has never caused this before.

Your job

1. Prove the cause for this incident. Do not assume it matches the last one.
2. Restore service with the smallest correct change.
3. Verify the customer's own workflow, not just that something responds.

Working notes

The service is expected on http://127.0.0.1:8100. The customer workflow is GET /customers. Runtime configuration lives in labs/docker/_stack/compose.override.yaml. After editing it, run tse apply to recreate the services, then tse check.

Track

Docker

Time

about 30 minutes

Difficulty

Straightforward

Tier

Core

Do these first: Application loads but customer data does not

Start it

In a Codespace or a local clone:

tse start docker/03-data-access-lost-after-credential-rotation

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer told you this is the same as the last one. The customer is guessing. Guessing is their job to do badly and yours to do carefully.

Before you apply last week's fix, ask what would actually prove it. If the cause were the same, you would expect the same evidence: a connection that is refused before the database ever sees it.

So check that first. If the evidence does not match, you have learned something valuable in about ten seconds, and you have avoided reporting a fix that fixed nothing.

Read the failure detail the application logs. Read the whole line.

Hint 2 of 3

The log detail is not "connection refused" this time. The database answered. That single word changes everything.

A refused connection means nothing was listening at that address. A rejection means the database received the connection, evaluated it, and turned it down. The network path is therefore proven working, which rules out the entire class of cause from the previous ticket.

What is left is what the database evaluates before it lets a client in:

• Which user is connecting
• What that user presented
• Which database and schema they asked for

The customer mentioned an overnight rotation. That is a lead, not a conclusion. Prove which side of the rotation is out of step, because both the application and the database hold a copy of the same fact.

Hint 3 of 3

Read the exact rejection text:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               logs --tail 20 app

It names password authentication for the user. Now prove what each side believes. What the application is presenting:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               exec app printenv DB_USER DB_PASSWORD

And what the database was actually built with:

docker compose -f labs/docker/_stack/compose.yaml \
               -f labs/docker/_stack/compose.override.yaml \
               exec postgres printenv POSTGRES_USER POSTGRES_PASSWORD

Only one side took the rotation. Note which one, because that determines whether the correct fix is to update the application or to roll the database credential forward. In this lab the database is the source of truth.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: data access lost after credential rotation

What the evidence proved

Command	What it proved	What it did not prove
docker compose logs app	The database rejected the login, rather than refusing the connection	Nothing about which side is wrong
The rejection wording	The network path works and the database is up	Nothing about who rotated what
exec app printenv DB_USER DB_PASSWORD	What the application presents	Whether it is correct
exec postgres printenv POSTGRES_USER POSTGRES_PASSWORD	What the database was created with

The distinction that matters: connection refused and authentication failed are not the same evidence. The first means nothing was listening. The second means something listened, evaluated you, and said no. Only the second proves the network path is fine.

Root cause

The overnight rotation updated the application's DB_PASSWORD but not the credential the database was initialized with. PostgreSQL's user password is set from POSTGRES_PASSWORD when the data directory is first created, and it is not re-read on later restarts, so the database kept the original value while the application moved on to the new one.

The two halves of one credential drifted apart. The symptom looked identical to the previous incident because both end in the same customer-visible 503, which is exactly why the symptom is not evidence.

Scoped fix

Bring the application back in step with the credential the database actually holds:

services:
  app:
    environment:
      DB_PASSWORD: demo-password

Then:

tse apply
tse check

In production the correct direction is usually the opposite: rotate the database credential forward to match the new secret rather than reverting the application to the old one. What matters here is that you can state which side is out of step and why, before you change either.

Customer update

The customer list failure was caused by the overnight credential rotation being applied to the application but not to the database, so the database was correctly rejecting the application's login. This is different from the issue another account saw last week, which is why the earlier fix would not have resolved it. We have brought the two back in step and confirmed your customer list is loading. No data was lost, and no customer data was exposed by the failed logins.

Engineering escalation, if you needed one

Impact: GET /customers returning 503 for all users since the overnight rotation. Evidence: database_connection_failed with a password authentication failure for user support; database container healthy and accepting connections; application and database hold different values for the same credential. Confirmed: network path, service name resolution, database availability. Ruled out: connectivity, the DB_HOST misconfiguration seen previously. Suspected cause: the rotation updated the consumer but not the provider, and PostgreSQL does not re-apply POSTGRES_PASSWORD to an existing data directory. Request: confirm whether the rotation job is expected to update both sides, and whether other services rotated in the same window are affected.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The log records that the login was rejected, rather than that the connection was refused. What does that distinction buy you?

• It proves the value the application holds is the wrong one.
• The network path is sound, because something received the attempt, evaluated it and answered.
• The database has been set up incorrectly.
• Nothing much, since both end as the same customer-facing failure.

Question 2 of 3Only one side was updated during the rotation. Why did restarting the other side not pick the new value up?

• The restart happened too quickly for the change to take effect.
• The new value never made it into the file.
• Credentials are cached in memory and survive a restart.
• It is applied once, when the data directory is first created, and an existing directory is never re-read for it.

Question 3 of 3Two services disagree about a secret. What is the most direct way to observe the disagreement?

• Ask each running process what it currently holds, instead of reading the files that were meant to set it.
• Compare the deployment history of the two services.
• Rotate it again on both sides and see whether the problem clears.
• Turn on verbose logging and reproduce the failure.

3 questions, none answered yet.

Why this one exists

Identical customer symptoms do not imply identical causes. "Cannot reach the database" and "the database refused me" are different proofs with different evidence, and the distinction is visible in the error detail.

In an interview

This exercise exists to break pattern matching. It presents the customer symptom you just solved, with a different root cause. Candidates who learned a fix rather than a method will apply the previous answer and report success without checking. The habit being trained is reading the specific error text before reaching for the cause that worked last time.

Commands introduced

• docker compose logs
• docker compose exec
• psql

Evidence layers

• application logs
• database authentication
• runtime configuration