Their sync reports success on every run and nothing ever arrives

Can I prove the request was understood, rather than merely accepted?

The ticket

CUSTOMER TICKET: our sync says it worked every night and nothing is in your system

Account: Halden Freight (enterprise) Impact: no events delivered since the integration went live Started: since we switched the integration on, eight days ago

Our nightly job has run every night since we turned it on and it has reported success every single time. We log the response from your side and it comes back as accepted, every run, no failures at all.

But nothing has ever shown up in your dashboard. Not one event in eight days. Our team has been through our side twice and cannot find anything wrong, because from where we are sitting it is working perfectly.

We are starting to wonder whether the events are being dropped after you accept them.

Your job

1. Work out which layer is failing. This ticket does not tell you, and neither does the folder it is filed under.
2. Prove whether the request is understood, not just whether it is accepted.
3. Make the customer's own call succeed in the sense they actually meant, then verify it.

Working notes

The customer's integration call is reproduced exactly as their system makes it. Once you have found what is running, you will find their call alongside it, and that file is what you edit. Their credentials and the current contract are documented with the service.

When you have changed it, run tse check.

Track

Mixed incidents

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Do these first: Same symptom as the last ticket, different cause, Same symptom as the last ticket, and the labels are correct, Batch job succeeds at the start and fails partway through, A report that was instant is now painful

Start it

In a Codespace or a local clone:

tse start mixed/02-their-sync-reports-success-and-nothing-arrives

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer is not mistaken and neither are you. Their job really does get a success response on every run, and nothing really has arrived. Both are true at once, and the gap between them is the fault.

That is worth sitting with, because almost every instinct you have built so far is for finding an error, and there is no error here to find. Nothing failed. Something succeeded at doing the wrong thing.

So the question is not "why did it fail". It is:

What did the server understand the request to be?

A success status is a statement about the request being accepted. It is not a statement about what the request contained, or about the server agreeing that it contained anything useful.

Run their call yourself and read the whole response rather than the first line of it. Not just what came back, but what came back empty. Absence is the hardest thing to notice in output you are skimming, and that is what makes this class of incident run for eight days.

Finding what is running is your first step, and it is one command.

Hint 2 of 3

docker compose ls will tell you what is running and where its files are. The customer's call is stored beside the service, along with the credentials and the current contract.

Run their call and look at the body, field by field, against what they meant to send:

What they sent	What the response says came back	What that means
An identifier for their workspace	Look at the workspace field
A name for the event	Look at the event field

Two of those fields come back as null. The request was accepted, the credentials were fine, the route was right, and the values the customer cared about were not picked up at all.

A server that reads only the fields it recognizes does not complain about the ones it does not. An unrecognized name is not an error to it. It is simply absent, and absence produces no message.

So compare two things carefully: the names of the fields in the customer's payload, and the names the service says it expects. The values are correct. Ask what they are labeled.

Hint 3 of 3

Run the customer's call and read the body:

bash labs/api/_stack/request.sh

You get 202 with "status": "accepted", and then:

"workspace": null,
"event": null,

The server accepted the request and understood none of the content. Now look at what it says it wants, which it tells you in its own error text when a payload is genuinely unreadable:

curl -s -X POST http://127.0.0.1:8101/v2/webhooks/events \
     -H 'X-API-Key: wk_live_active_3c95' \
     -H 'Content-Type: application/json' \
     -d 'not json'

That names both fields it is looking for. Compare them against the names in the customer's payload in labs/api/_stack/request.sh.

The values the customer is sending are correct. ws_4471 is the right workspace and order.created is the right event. Only the labels on them are wrong, and because the server takes what it recognizes and ignores the rest, wrong labels produce a success rather than a complaint.

Fix the two field names and run tse check.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: their sync reports success and nothing arrives

What the evidence proved

Command	What it proved	What it did not prove
docker compose ls	What is running, and where its files are	Nothing about the fault. This was orientation
bash request.sh, first line	The call returns 202 Accepted, exactly as the customer reported	Nothing about what was accepted
bash request.sh, the body	workspace and event both came back null	Nothing yet about why
A deliberately unreadable payload	The service names the two fields it reads: workspace and event
The customer's payload	Those values are sent as workspace_id and event_type

The customer's credentials were valid, their route was current, their JSON was well formed, and their values were right. Every layer anyone would normally suspect was working. That is what made this survive eight days of their team looking at it.

Worth naming, because it is the habit this exercise exists to build: a success status was treated as proof the request had done something. It is not. It is proof the request was accepted. What the server understood is a separate question, and the answer to it was sitting in the response body the whole time.

Root cause

The customer's integration labels its fields workspace_id and event_type. The service reads workspace and event.

The service takes the fields it recognizes and ignores the rest, which is ordinary and mostly desirable behavior. An unrecognized field name is not an error to it, so nothing was reported. The two fields it wanted were simply absent, so it recorded an event with no workspace and no type, and answered 202 Accepted because the request itself was perfectly well formed.

Nothing was dropped after acceptance, which was the customer's theory. The events arrived carrying nothing to identify them.

Scoped fix

In labs/api/_stack/request.sh, label the values with the names the service reads:

-d '{"workspace":"ws_4471","event":"order.created","id":"evt_9013"}'

Then:

tse check

The values do not change. Only the two field names do.

Customer update

Your job has been running correctly and the responses you logged were accurate. The problem is in the field names rather than anything failing. Your payload sends the workspace as workspace_id and the event as event_type, and our webhook endpoint reads those two values from workspace and event. Because the rest of the request is valid, our service accepted it and returned success, but the two values it needed to file the event were not present under the names it looks for. Nothing was dropped on our side after acceptance. Renaming those two fields will make the events appear. I have confirmed a test event end to end with the corrected names against your workspace. I am sorry this took eight days to surface. A success response that carries an empty result is a bad experience, and I have raised it with our API team.

Engineering escalation, if you needed one

Impact: one enterprise integration delivered zero usable events for eight days while receiving 202 Accepted on every request. Evidence: POST /v2/webhooks/events returns 202 with "workspace": null and "event": null when the payload uses unrecognized field names. Confirmed: authentication, route version, payload validity, network delivery. Ruled out: anything dropping events after acceptance. Suspected cause: the endpoint accepts payloads whose required fields are absent, and reports success. Request: can the endpoint reject a webhook that carries neither workspace nor event, rather than accepting it. Silent acceptance means the customer cannot detect this and neither can we, and this one only surfaced because somebody eventually looked at an empty dashboard.

That request is the real outcome of this ticket. The field rename fixes one customer. Rejecting an empty payload fixes everyone who makes the same mistake next.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The customer logged an accepted response on all eight nights, and those logs were accurate. What did that response entitle them to conclude?

• That their events had been recorded and would appear
• That the request was well formed and authorized enough to be taken, and nothing beyond that
• That the service had validated every field they sent
• That nothing further could be wrong on their side

Question 2 of 3The service answered with two nulls instead of refusing the payload. Why does that behavior make this so expensive to find?

• Because null values are always hidden from response bodies
• Because an unrecognized field name produces no error, so the only signal is something absent rather than something reported
• Because the failure only happened intermittently
• Because the customer had no way to test their integration

Question 3 of 3Renaming two fields fixes this customer today. What is worth raising with the API team afterwards?

• That the documentation should be clearer about field names
• That the endpoint should reject a payload carrying neither field it requires, rather than accepting it
• That the customer should be given a dedicated test environment
• That the service should log more detail on every webhook

3 questions, none answered yet.

Why this one exists

A success status says the request was accepted, not that it did what was intended. When a server takes only the fields it recognizes, an unrecognized name is not an error, it is an absence, and absence is the hardest thing to see in output you were not reading closely.

In an interview

Silent success is the failure mode candidates handle worst, because every habit they have is built around finding an error and there is not one. The move that separates people here is reading the response body after seeing a success status, and being able to say what the server understood rather than what it returned.

Commands introduced

• curl -D -

Evidence layers

• response status
• response body
• payload contract