Batch job succeeds at the start and fails partway through

Can I prove the failure depends on request rate rather than on the data?

The ticket

CUSTOMER TICKET: nightly usage sync keeps failing halfway through

Account: Tidewater Health (enterprise) Impact: usage reporting incomplete, billing review blocked Started: since they increased their record volume last week

Our nightly usage sync pulls all our usage records from your API. It used to finish fine. Since we grew, it now gets partway through and then starts erroring, and we end up with an incomplete file every night. The first few pages always come back fine, so it cannot be authentication. We think you have corrupt records somewhere in the middle of our data.

Your job

1. Reproduce the failure and look at where in the run it starts.
2. Prove whether the failures track the data or something else.
3. Get the full sync completing, then write the update.

Working notes

The API is at http://127.0.0.1:8101. The customer's own request lives in labs/api/_stack/request.sh. Run it to reproduce, edit it until it succeeds, then run tse check. Credentials you have access to are listed in labs/api/_stack/credentials.md.

Track

APIs

Time

about 35 minutes

Difficulty

Involved

Tier

Core

Do these first: Integration rejected with an authentication error

Start it

In a Codespace or a local clone:

tse start api/04-nightly-sync-fails-partway

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer's theory is corrupt records in the middle of their data. Test that theory directly, because it is falsifiable and the test is free.

If the failures were caused by specific records, they would follow those records. Run the sync twice and compare. Do the same pages fail both times? Do the failures move if you change the order or the page size?

If the failing position stays the same while the data behind it changes, the data is not what is failing.

Look at where the run turns, not just that it turned.

Hint 2 of 3

The failures start at a consistent position, not on consistent records. That rules out the corrupt-data theory and points at something that accumulates across the run.

The status is 429, which is not an error in the usual sense. Nothing is broken. The API is telling the client it is going too fast, and it is doing so in a structured way.

This is also why the customer's own reasoning misled them. "The first few pages always work, so it cannot be authentication" was correct, but they concluded the problem must be in the data, when the other thing that changes as a run progresses is how many requests you have made.

Read the response headers on the first failing page. The API states the limit, what remains, and exactly how long to wait.

Hint 3 of 3

Look at a rejected response in full:

for i in $(seq 1 8); do
  curl -s -o /dev/null -D - -H 'X-API-Key: wk_live_active_3c95' \
    "http://127.0.0.1:8101/v2/usage?page=$i" | head -1
done

Then inspect the headers on one that returns 429:

Retry-After: 11
RateLimit-Limit: 5
RateLimit-Remaining: 0

Retry-After is the fix. The server has told you precisely how long to wait, so the client should sleep for that long and retry the same page rather than failing it or retrying immediately.

Edit labs/api/_stack/request.sh so that a 429 causes it to read Retry-After, wait, and retry the page it was on. Capture headers with curl -D to a file so you can read the value. Then run tse check.

Retrying immediately, or with a guessed delay, is the common wrong answer. It turns one throttled client into a client that is throttled continuously.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: nightly sync fails partway through

What the evidence proved

Evidence	What it proved	What it did not prove
First pages succeed, later ones fail	The credential and the route are fine	Nothing about the cause
The same position fails across runs	The failure is positional, not data-dependent
HTTP 429	The API is deliberately throttling, not erroring
RateLimit-Limit: 5, RateLimit-Remaining: 0	The exact budget and that it is spent
Retry-After: 11	Precisely how long to wait
Waiting and retrying returns 200	The records are fine. Nothing is corrupt

Root cause

The sync issues requests as fast as it can. The API allows 5 requests per 10-second window per key, so the first pages succeed, the budget is exhausted, and every subsequent request is rejected with 429 until the window rolls over. The client treats 429 as a permanent failure and drops the page, producing an incomplete file.

This is not a fault on either side. The API is throttling correctly and the client is not honoring the contract it is being told about.

Why the customer's theory was wrong, and why it was reasonable

They concluded corrupt records because some requests worked. That is a sound instinct, and the test that kills it is cheap: if bad records were the cause, the failures would follow those records. They do not. They follow the position in the run, which stays constant even as the data behind it changes.

The general rule worth keeping:

When early requests succeed and later ones fail, suspect rate before data.

The other giveaway is in the customer's own message: the problem started when their volume grew. Volume changes how many requests a run makes. It does not usually corrupt anything.

Scoped fix

The client must absorb throttling rather than fail on it. The API states exactly how long to wait, so honor that value instead of guessing:

if [[ $status == "429" ]]; then
    wait_for=$(grep -i '^Retry-After:' /tmp/tse-page.headers | tr -d '\r' | awk '{print $2}')
    sleep "${wait_for:-2}"
    continue   # retry the same page, do not skip it
fi

Two mistakes to avoid, both common:

• Retrying immediately. This turns one throttled client into a permanently throttled one and can look like an attack.
• Guessing a fixed delay. Retry-After is authoritative. A guess is either too short, which fails again, or too long, which slows every run.

Then tse check.

Customer update

I reproduced the sync and your records are not corrupt. Every page that failed returns correctly when it is retried, including the ones in the middle of the run. The failures are our API rate limiting your client: the sync requests pages faster than the account's limit of 5 requests per 10 seconds, so once the budget is used the remaining requests are rejected until the window resets. This started when your volume grew because more records means more requests in the same burst.

The fix is on the client side, and our API gives it the information it needs. When a request is rejected we return a Retry-After header stating how many seconds to wait. If the sync waits for that period and retries the same page, the run will complete. I would avoid retrying immediately, since that keeps the client throttled continuously.

No usage data was lost. The records were never sent, so a completed run will pick them all up.

Engineering escalation, if you needed one

Impact: enterprise account's nightly usage export incomplete since their volume increase, blocking billing review. Evidence: 429 with RateLimit-Limit: 5, RateLimit-Remaining: 0, and Retry-After present; failures positional rather than record-linked; retried pages return 200 with valid data. Confirmed: authentication, routing, data integrity. Ruled out: corrupt records, credential expiry, partial outage. Suspected cause: client does not implement backoff on 429. Request: confirm whether this account's rate limit is appropriate for their current volume, and whether our published integration guide documents Retry-After handling clearly enough to have prevented this.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The same point in the run fails on every attempt, while the records sitting at that point change from night to night. What does that pattern rule out?

• That the run is failing at all.
• That the client is the side at fault.
• That the failure is deliberate.
• That the contents of any particular record are causing it.

Question 2 of 3The refusal arrived with an allowance, a remaining count of zero, and a number of seconds. What does that combination let you say?

• That the account has outgrown its plan.
• That the refusal is deliberate, exactly how much is permitted, and precisely how long to wait before carrying on.
• That the service is straining under load from other customers.
• That the request was malformed.

Question 3 of 3Neither side has a defect. So what has to change?

• The client has to wait as long as it is being told to, rather than treating a refusal as final.
• The allowance should be raised until the whole job fits inside it.
• The job should be moved to a quieter time of day.
• The failing pages should be skipped and reconciled afterward.

3 questions, none answered yet.

Why this one exists

When early requests succeed and later ones fail, suspect rate rather than data. A well-behaved API states exactly how long to wait, and honoring that is the fix rather than retrying harder.

In an interview

Partial failure is the most misdiagnosed pattern in support. The instinct is to look for bad records, because some requests worked. Noticing that the failures start at a consistent position rather than on consistent data is the move that separates candidates.

Commands introduced

• Retry-After header
• RateLimit headers
• backoff loops

Evidence layers

• HTTP status code
• rate limit headers
• failure distribution across a batch