The dashboard is green and one customer is timing out

Can I prove what this customer is seeing, rather than what everyone is seeing?

The ticket

CUSTOMER TICKET: every report we run is slow and you keep telling us the service is fine

Account: Northwind Traders (enterprise) Impact: every report request slow, all of their users Started: noticed over the past week, possibly longer

Every report we run takes well over a second before anything comes back. It has been like this for at least a week and it is now holding up our month end close.

We have raised this twice. Both times we were told the service is healthy and that response times are comfortably within target, and both times we were sent a link to your dashboard showing exactly that.

We are not disputing your dashboard. We are telling you that on our side, every single request is slow. Not some of them, not the big ones. Every one.

Somebody there must be able to tell the difference between "the service is fine" and "the service is fine for you".

Your job

1. Prove what this one account is experiencing, separately from what the service reports across everybody.
2. Explain how both readings can be honest at the same time. Your colleagues who closed the previous two tickets were not being lazy.
3. Make their requests as fast as everyone else's, without changing what those requests return.

Working notes

The service is on 127.0.0.1:8102 and reports what it has served since it started:

curl -s http://127.0.0.1:8102/metrics | jq

A sample workload runs once when the stack comes up, so there is real traffic to read. The traffic container showing Exited (0) means that finished.

The customer's account is northwind. Their configuration lives in labs/observability/_stack/tenants.json. After editing, run tse apply, then tse check.

Giving them the right speed with the wrong data is not a fix. The grader checks both.

Track

Observability

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Start it

In a Codespace or a local clone:

tse start observability/01-the-dashboard-is-green-and-they-are-timing-out

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Two people are answering two different questions and neither has noticed.

Your colleagues asked "is the service healthy" and got back a truthful yes. The customer asked "is the service healthy for us" and nobody has answered it, because the reading everyone keeps quoting cannot answer it. Averages and overall rates describe a population. This customer is not a population. They are one member of it, and a small one.

So the reframing is:

What is this account's own reading, separately from everyone else's?

Notice how strong their claim is, because it is the most useful thing in the ticket. Not "sometimes slow", not "slow on big reports". Every single request. A problem that affects every request from one account and no requests from anybody else is not a capacity problem and not an intermittent one. Something is different about the path their requests take.

Two readings, side by side, is the whole first move. Get the overall one, then get theirs, and put the numbers next to each other before you form a theory.

If the service will not give you a per account reading, that gap is itself the finding and worth writing down. It happens to give you one.

Hint 2 of 3

Get both readings. The endpoint takes a filter:

curl -s http://127.0.0.1:8102/metrics | jq
curl -s 'http://127.0.0.1:8102/metrics?tenant=northwind' | jq
curl -s 'http://127.0.0.1:8102/metrics?tenant=contoso' | jq

The third one is not optional. Comparing this account to the overall figure tells you they are worse than average, which you already knew. Comparing them to a specific other account tells you whether "slow" is normal here, and it is not: somebody else's requests are all inside the objective.

So one account is entirely slow and another is entirely fine, on the same service, at the same time. Whatever is different has to be something the service does per account.

Now run one of their reports yourself and read the whole response rather than just timing it:

curl -s 'http://127.0.0.1:8102/v1/reports?tenant=northwind' | jq
curl -s 'http://127.0.0.1:8102/v1/reports?tenant=contoso' | jq

The two responses have the same shape and one field differs. That field says where the service got the account's configuration from, and the two accounts are not being answered from the same place.

Then look at what the second service is being asked to do, and for whom:

docker compose -f labs/observability/_stack/compose.yaml \
               -f labs/observability/_stack/compose.override.yaml \
               logs downstream | grep directory | tail

Hint 3 of 3

The evidence, side by side:

overall     {"requests": 100, "slow_requests": 5, "within_objective_pct": 95.0}
northwind   {"requests":   5, "slow_requests": 5, "within_objective_pct":  0.0}
contoso     {"requests":   5, "slow_requests": 0, "within_objective_pct": 100.0}

Ninety-five percent of requests met the objective. Zero percent of theirs did. Both numbers come from the same service at the same instant and both are correct. The customer is a twentieth of the traffic, so being entirely broken costs the overall figure five points, which is not enough to make anything go red.

And the reason, from the report itself:

northwind   "config_source": "directory"
contoso     "config_source": "local"

The service keeps a local copy of who its customers are and what they are entitled to. When an account is not in that copy it falls back to the authoritative directory, which is correct, returns the right answer, and takes a second and a half. Every time. There is no cache.

northwind is missing from labs/observability/_stack/tenants.json. They were onboarded and never added to it.

Add them, with the values the directory actually returns for them rather than values copied from another account:

curl -s 'http://127.0.0.1:8102/v1/reports?tenant=northwind' | jq '{plan, row_limit}'

Then put that entry into tenants.json, run tse apply, and tse check.

Do not give them a placeholder to make the slowness go away. An enterprise account on somebody else's row limit is a worse ticket than a slow one, and it is the kind that gets found weeks later.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the dashboard is green and one customer is timing out

What the evidence proved

Command	What it proved	What it did not prove
/metrics	100 requests, 5 slow, 95% within the objective	Nothing about any individual account
/metrics?tenant=northwind	5 requests, 5 slow, 0% within the objective
/metrics?tenant=contoso	5 requests, none slow, 100%	That slow is normal here, which is what rules out a capacity problem
/v1/reports?tenant=northwind	Correct data, and config_source: directory
/v1/reports?tenant=contoso	Same data shape, and config_source: local
`logs downstream	grep directory`	Directory lookups happen for one account and no other

The two readings are the diagnosis, and neither means anything without the other. Ninety-five percent of requests met the objective. Zero percent of this customer's did. Same service, same instant, both correct.

That is worth stating plainly because it is what made this expensive. Nobody was being lazy. The reading everybody quoted was accurate and it was answering a question the customer had not asked. An account that is one twentieth of the traffic can be entirely unusable while costing the overall figure five points, which is not enough to turn anything red.

The third reading is the one that turns an observation into a diagnosis. Knowing this account is worse than average tells you nothing you did not already believe. Knowing another account is at a hundred percent tells you slow is not normal here, which rules out capacity, load, and a bad afternoon, and leaves only things the service does per account.

Root cause

northwind was missing from tenants.json, the service's local copy of who its customers are.

When an account is not in that copy, the service falls back to the authoritative directory. The fallback is correct and returns exactly the right plan and limits. It also takes a second and a half, it runs on every single request, and nothing caches it. So the account was never broken in the sense of returning anything wrong. It was answered from the slow path, every time, since the day they were onboarded.

Nothing about this is a bug in the fallback. A service without one would simply fail for any account added since the local copy was last written. The bug is that an onboarding step did not finish, and that nothing noticed for a week because no reading was scoped to the account it affected.

Scoped fix

Add them to labs/observability/_stack/tenants.json, with the values the directory holds for them:

"northwind": {
  "plan": "enterprise",
  "row_limit": 50000
}

Then:

tse apply
tse check

Their requests drop from about 1.5 seconds to about 15 milliseconds, and config_source changes from directory to local.

Not the fix: adding them with a placeholder or with a row copied from another account. It resolves locally, the latency disappears completely, the customer stops complaining, and an enterprise account is quietly running on a starter row limit. That surfaces weeks later as a truncated report and is a far worse ticket than a slow one. The check reads the plan and the limit for exactly this reason.

Customer update

You were right, and I am sorry it took three tickets to get here.

Our dashboard reports across all traffic, and your account is a small share of it, so an account that is completely slow moves that number by about five points. It stayed green while every one of your requests was over a second. Both things were true, and we were only looking at the one that could not answer your question.

Here is what we can see for your account specifically: of your last five requests, all five were over our one second target, against ninety-five percent of all traffic meeting it. That is the reading we should have pulled the first time you raised it.

The cause was on our side. When your account was set up, it was not added to the service's local record of customer configuration, so every request was looking your details up from our directory rather than reading them locally. You were always getting the correct plan and limits, just by the slow route, every single time.

That is now corrected and your reports return in a few hundredths of a second. Nothing changed about what they contain or what you are entitled to. If your month end close needs anything rerun, tell me which reports and I will confirm the timings myself.

Engineering escalation, if you needed one

Impact: one enterprise account had every request served from the directory fallback for at least a week. Three tickets, two closed as working as intended, month end close delayed. Evidence: within_objective_pct 95.0 overall and 0.0 for this account, from the same endpoint at the same time; config_source: directory on their reports and local on everybody else's; directory lookups in the renderer's log for one account only. Confirmed: correct data returned throughout, one and a half seconds per request, reproducible on every request. Ruled out: capacity and load, since another account was at 100% in the same window. Their client, their reports, and the size of their data. Suspected cause: onboarding did not add them to tenants.json, and the fallback made that invisible by being correct. Request: two things, and the second matters more. The fallback has no cache, so a missing entry costs a second and a half on every request forever rather than once. And nothing alerts on an account whose objective rate is zero while the overall rate is healthy, which is the shape of every ticket like this one.

The missing entry is fixed for this customer. What will happen again is the next account onboarded with a step skipped, and the reason nobody caught it for a week is that every dashboard was measuring the service instead of its customers.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The service reported 95% of requests within the objective and 0% for this account, at the same instant. What follows?

• One of the two readings is wrong and the aggregate is the one to trust
• Both are accurate, and an account small enough to be five percent of traffic can be entirely unusable without moving the overall figure enough to notice
• The objective is set too loosely and should be tightened
• The service degrades under load, which the aggregate smooths out

Question 2 of 3You pulled a reading for a second, unaffected account as well. Why was that worth doing?

• To confirm the complaining account was worse than average
• To rule out load and capacity, since an account at 100% in the same window means slow is not the normal state of this service
• Because comparing two customers is the only way to compute a percentile
• To check whether the second account had also been affected without reporting it

Question 3 of 3Adding the account with a placeholder configuration would have removed the slowness entirely. Why is that not the fix?

• It would have left the directory lookup running in the background
• It trades a visible slowness for a silent wrong entitlement, and an enterprise account on somebody else's row limit surfaces weeks later as a truncated report
• The service would reject a configuration it had not issued itself
• It would have been rejected in review

3 questions, none answered yet.

Why this one exists

An average over every customer cannot answer a question about one customer, and a dashboard that reports only the aggregate will keep saying everything is fine while a whole account is unusable. Both statements can be true at once. The skill is knowing which reading answers the question in front of you, and noticing when the one you have does not.

In an interview

Almost everyone can say "look at the tail rather than the average". Far fewer reach for the segment, and the segment is what actually resolves tickets, because a customer never asks how the service is doing. Being able to say "ninety-five percent of requests met the objective and zero percent of yours did, and here is the reading for both" is the answer that ends the argument instead of continuing it.

Commands introduced

• curl /metrics
• jq -c
• docker compose logs --tail

Evidence layers

• what the service reports across all traffic
• what it reports for this customer alone
• where their configuration is being resolved from
• what the second service is doing on their behalf