They keep sending you the reference and you have no record of it

Can I prove where the identifier the customer holds stops?

The ticket

CUSTOMER TICKET: we have sent you three references and you say none of them exist

Account: Northwind Traders (enterprise) Impact: month end report fails, and three support attempts have gone nowhere Started: first raised nine days ago

Our month end report fails. When it does, your product shows an error with a reference on it and tells us to quote that reference to support.

We have now done that three times. Every time, the reply is that there is no record of it and could we please confirm we copied it correctly. We copied it correctly. Here is the most recent one, and a screenshot of where it came from: req-nw7k2p9x4m31.

I want to be clear about how this looks from here. Your product tells us to quote a reference. We quote it. You tell us it does not exist. Either your product is showing us something meaningless, or somebody is not looking properly.

The report still fails. At this point I care more about the failure than the reference, but we cannot get anyone to look at the failure without one.

Your job

1. Take the reference seriously. The customer copied it correctly and the product did show it to them.
2. Prove where it stops. "We have no record of it" is a statement about your search, not about their report.
3. Make it so the reference they are given can actually be followed all the way through.

Working notes

The service is on 127.0.0.1:8102. You can reproduce their failure, and you can supply the reference yourself rather than waiting for one:

curl -s -H 'X-Request-Id: req-nw7k2p9x4m31' \
  'http://127.0.0.1:8102/v1/reports?tenant=northwind&rows=50000' | jq

Two services handle a report, api and downstream. Both write to their own logs:

docker compose -f labs/observability/_stack/compose.yaml \
               -f labs/observability/_stack/compose.override.yaml \
               logs api

docker compose -f labs/observability/_stack/compose.yaml \
               -f labs/observability/_stack/compose.override.yaml \
               logs downstream

Configuration is in labs/observability/_stack/compose.override.yaml. After editing, run tse apply, then tse check.

Fixing the report itself is a separate ticket. This one is about being able to find it.

Track

Observability

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Do these first: The dashboard is green and one customer is timing out

Start it

In a Codespace or a local clone:

tse start observability/02-you-say-you-have-no-record-of-it

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer is right about the thing that matters, so start by conceding it.

They were shown a reference by your product and told to quote it. They quoted it. Three people searched, found nothing, and told them the reference does not exist. But your product does not invent references to show people, so it did exist somewhere, at least for long enough to be printed on a screen.

That reframes the question. Not "does this reference exist", which three people have already answered wrongly. Instead:

Where does it stop?

An empty search result is a claim about the search. It says the string was not in the place you looked. It does not say the request never happened, and here you have independent proof the request did happen, because the customer's report failed and they have a screenshot of the aftermath.

So find the boundary. Their report goes through more than one service on the way to failing, and each of those services writes its own log. If the reference is in one and not another, the gap between them is your answer, and the gap is a much more useful thing to report than an absence.

You do not have to wait for the customer to send another one. You can reproduce the failure yourself and choose the reference you send with it, which makes it trivial to search for afterwards. Do that first.

Hint 2 of 3

Reproduce it with a reference you chose, then look for that reference in each service in turn.

curl -s -H 'X-Request-Id: trace-me-9001' \
  'http://127.0.0.1:8102/v1/reports?tenant=northwind&rows=50000' | jq

The response carries the reference back, so the customer's screenshot was accurate. Now:

docker compose -f labs/observability/_stack/compose.yaml \
               -f labs/observability/_stack/compose.override.yaml \
               logs api | grep trace-me-9001

docker compose -f labs/observability/_stack/compose.yaml \
               -f labs/observability/_stack/compose.override.yaml \
               logs downstream | grep trace-me-9001

One of those returns a line. The other returns nothing.

That is the boundary, and it is worth being precise about what it means. The second service is not missing the request. It handled the request, and it is where the report actually failed. Go and look at the same moment without filtering by the reference:

docker compose -f labs/observability/_stack/compose.yaml \
               -f labs/observability/_stack/compose.override.yaml \
               logs downstream | grep render_failed | tail -3

The failure is right there, in full detail, with a reference on it. Just not the one the customer was given, and not one that appears anywhere else in the world.

So the question is no longer where the reference went. It is why the second service invented its own. Look at what the first service is configured to pass on when it calls the second.

Hint 3 of 3

The evidence, side by side. In api:

level=error service=api request_id=trace-me-9001 event=report_failed tenant=northwind rows=50000

In downstream, at the same instant:

level=error service=downstream request_id=gen-880e254a7190 event=render_failed tenant=northwind rows=50000 budget=25000

Same request. Same failure. Two references, and only one of them was ever shown to anybody. Searching for the customer's returns exactly one line, from the service that did not fail.

The cause is in compose.override.yaml:

FORWARD_HEADERS: "authorization,x-tenant"

The first service passes on an explicit list of headers when it calls the second. An allowlist is the right design and this one is simply missing an entry, so the reference stops at the boundary. The second service, receiving no reference, correctly generates one, which is also right and is what makes this so hard to see: nothing errors, no log is empty, and every line has a perfectly good reference on it. They just do not join up.

The fix:

services:
  api:
    environment:
      FORWARD_HEADERS: "authorization,x-tenant,x-request-id"

Then tse apply, and reproduce it again. The same reference now appears in both logs and the failure is one search away.

Do not fix the failing report while you are here. It is a real bug and it deserves its own ticket, but making it succeed would remove the evidence rather than make it findable, and the next customer to hit a different failure would be in exactly the same position. The grader checks the failure is still being recorded for that reason.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: they keep sending you the reference and you have no record of it

What the evidence proved

Command	What it proved	What it did not prove
Reproducing with a chosen reference	The failure is real and reproducible on demand
The response body	The reference comes back to the caller, so the screenshot was accurate	Nothing about what was recorded
logs api | grep <ref>	One line, from the service that did not fail
logs downstream | grep <ref>	Nothing at all	That the request never reached it, which is the wrong conclusion
logs downstream | grep render_failed	The failure, in full, under a reference nobody has ever seen

The pair of searches is the diagnosis. One service knows the customer's reference and the other has never heard of it, and the one that has never heard of it is where the report actually failed.

Worth being precise about the mistake three colleagues made, because it is an easy one and it is not laziness. They searched, found nothing, and reported that the reference did not exist. An empty result is a claim about the search. It says the string was not in the place you looked. Here there was independent evidence the request happened, in the form of a customer whose report failed and who had a screenshot, and that evidence should have outweighed the absence.

The other thing that makes this hard is that nothing looks broken. No log is empty, no error is unhandled, and every single line has a perfectly well formed reference on it. The logs are healthy and unjoinable, which is a much quieter failure than a log that stops being written.

Root cause

The api passes an explicit allowlist of headers when it calls the renderer:

FORWARD_HEADERS: "authorization,x-tenant"

x-request-id is not in it, so the reference stops at the service boundary. The renderer receives a request with no reference on it, generates its own, and logs the failure against that. Both services behave correctly in isolation. An allowlist is the right design, generating a reference when none arrives is also right, and between them they produce a request that cannot be followed.

The customer's reference was real, was shown to them accurately, and only ever existed in the log of the one service that did not fail.

Scoped fix

In labs/observability/_stack/compose.override.yaml:

services:
  api:
    environment:
      FORWARD_HEADERS: "authorization,x-tenant,x-request-id"

Then:

tse apply
tse check

Reproduce the failure again and the same reference now appears in both logs.

Not the fix: correlating by timestamp instead. It works on a quiet service and stops working exactly when you need it, because two customers failing in the same second are indistinguishable, and it does not scale past one person doing it by hand. It also does not give the customer anything.

Also not the fix: making the report succeed. It is a real bug and deserves its own ticket, but this ticket is that a reference cannot be followed. Making the failure go away removes the evidence rather than making it findable, and the next customer to hit a different failure is in exactly the same position. The grader checks the failure is still being recorded.

Customer update

You copied the reference correctly, our product was right to show it to you, and we were wrong to tell you it did not exist. I am sorry it took three attempts and nine days to establish that.

Here is what was happening. A report passes through two of our services. The reference you were shown was created by the first one and was not being passed to the second, and the second is where your report was actually failing. So when we searched for your reference we were searching in the only place it could never appear, and the failure was sitting in the other log the whole time under an internal reference nobody had given you.

That is fixed. References now follow a request all the way through, so the one you are shown will find the failure directly.

On the report itself: it is failing because it exceeds an internal size limit for rendering, which I now have the detail on and have raised separately. I will come back to you on that specifically rather than leaving it inside this thread, and I will give you the reference for it.

Engineering escalation, if you needed one

Impact: an enterprise customer's month end report has failed for nine days. Three support attempts closed as unreproducible because the reference they were given cannot be found in the service that fails. Evidence: a request carrying X-Request-Id appears once in api and zero times in downstream; the corresponding render_failed in downstream carries a generated reference instead. Confirmed: reproducible on demand, both services healthy, nothing erroring. Ruled out: the customer copying it wrongly, log retention, the reference being invented by the product. Suspected cause: FORWARD_HEADERS on the api does not include x-request-id, so the reference stops at the boundary and the renderer generates its own. Request: the one line is fixed. The durable ask is that nothing detects this. Every log was full, every line well formed, and the only symptom was three support engineers failing to find something. A check that a reference entering at the edge appears in every service that handled the request would have caught it on the day it shipped, and would catch the next service added without it.

The header is a one line fix and it is not the interesting part. What will recur is the next service added to this path, because nothing about adding one forces anybody to think about propagation, and the failure it produces looks like support being careless rather than like a defect.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3Three colleagues searched for the customer's reference, found nothing, and told them it did not exist. What was wrong with that?

• Nothing, given the tools they had. The reference genuinely was not there
• An empty result says the string was not where you looked, and there was independent evidence the request happened that should have outweighed it
• They should have asked the customer to reproduce it while they watched
• The logs had rotated and the evidence was genuinely gone

Question 2 of 3Both services behaved correctly in isolation. What produced the failure between them?

• The renderer was misconfigured and should not generate a reference of its own
• An allowlist of forwarded headers was missing one entry, so a correct design and a correct default combined into a request nobody can follow
• The two services disagreed about the header's name
• The customer's client was not sending a reference at all

Question 3 of 3You could have found the failure by matching timestamps across the two logs instead. Why is that not the fix?

• The clocks on the two services are not synchronized
• It works on a quiet service, stops working the moment two customers fail in the same second, and gives the customer nothing they can quote
• Timestamps are not written to these logs
• It would require access neither service grants

3 questions, none answered yet.

Why this one exists

A correlation identifier is only worth anything if it survives every hop. One that is generated at the edge and not passed on gives every service its own private view of a request that nobody can join up, and the logs stay full, healthy and useless. An empty search result is a claim about your search, not about whether the event happened.

In an interview

Correlating a reference a customer hands you across the services that handled their request is the single most common thing a support engineer does, and it is the first thing that stops working at scale. Being able to say "it stops between these two services, here is the request in one log and here is the same moment in the other under a different reference" is a concrete finding. "We could not find it" is not.

Commands introduced

• curl -H
• grep across two services
• docker compose logs <service>

Evidence layers

• what the customer was given
• where that identifier appears
• where the failure is actually recorded
• what the two services put in their logs for the same request