The service restarts every few minutes and nobody changed the code

Can I prove whether the application crashed or was killed?

The ticket

CUSTOMER TICKET: order service keeps restarting

Account: Copperline Robotics (growth) Impact: intermittent failures, roughly every few minutes Started: after a tuning change on Tuesday

The order service keeps dying and coming back. Users get errors for maybe thirty seconds, then it recovers, then it happens again. We changed a cache setting on Tuesday to improve performance but that is a config value, not code, so we do not think that is related. Nobody has deployed new code in two weeks.

Your job

1. Prove whether the application is failing or something is stopping it.
2. Those are different causes with different fixes, so do not guess.
3. Restore stable service and verify the customer's own workflow.

Working notes

The workload runs in the tse-training namespace on the kind-proveit context. Unlike the Docker track there is no file to edit and no tse apply: change the cluster directly, the way you would in production, then run tse check. tse reset puts the ticket's state back.

Track

Kubernetes

Time

about 35 minutes

Difficulty

Involved

Tier

Core

Do these first: The service is down and there are no logs at all

Start it

In a Codespace or a local clone:

tse start kubernetes/03-orders-service-keeps-dying

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer has told you two things that sound like exclusions and are not. "It is only a config value, not code" and "nobody has deployed in two weeks" both assume that only code changes can cause a crash. Plenty of things stop a process that have nothing to do with its source.

The distinction worth proving first is bigger than which setting changed:

Did the application fail, or was it stopped?

Those look identical from the outside. Both produce restarts, both produce user errors, both recover. They have completely different causes and completely different fixes, and the evidence that separates them is recorded on the pod itself.

Also note the timing the customer gave you. Recovering and failing on a cycle suggests something is being hit repeatedly, not something that broke once.

Hint 2 of 3

A container that has restarted has two stories. The one you get by default is the new instance, which is usually running happily and has nothing useful to say. The interesting one belongs to the instance that already died.

Two places hold it:

• The previous instance's output, which you have to ask for explicitly.
• The last terminated state on the pod, which records why it ended and with what exit code.

That second one is the decisive evidence. An application that failed on its own exits with a code its own code chose, and normally logs a reason first. A process stopped from outside exits with a code the platform assigned, and its logs simply stop mid-sentence with no error at all.

Read the reason field. If it names something other than an error, believe it, then go and look at what the workload is allowed to consume.

Hint 3 of 3

kubectl -n tse-training get pods

The restart count is climbing. Get the previous instance's ending:

kubectl -n tse-training describe pod -l app.kubernetes.io/name=orders-api | grep -A6 'Last State'
kubectl -n tse-training logs -l app.kubernetes.io/name=orders-api --previous --tail=20

Reason: OOMKilled with Exit Code: 137 means the kernel stopped the process for exceeding its memory allowance. The logs stop without an error because the process was never given the chance to write one.

Now compare what it is allowed against what it now uses:

kubectl -n tse-training get deployment orders-api \
  -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'
kubectl -n tse-training get deployment orders-api \
  -o jsonpath='{.spec.template.spec.containers[0].env}{"\n"}'

The cache setting the customer changed on Tuesday raised what the process allocates, and the allowance was never raised to match. Either the allowance goes up or the cache comes down, and that is a conversation to have with them.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the service keeps restarting

What the evidence proved

Command	What it proved	What it did not prove
kubectl get pods	Restart count is climbing steadily	Whether it crashed or was stopped
describe pod, Last State	Reason: OOMKilled, Exit Code: 137	Nothing about the application's own logic
logs --previous	Output stops mid-run with no error
-o jsonpath on resources	The memory limit is 128Mi
-o jsonpath on env	ALLOCATE_MB is set to 200

The decisive distinction: exit code 137 and a reason of OOMKilled mean the kernel stopped the process for exceeding its allowance. The application did not fail. It was killed, which is why the logs simply stop rather than ending in an error, and why there is nothing in the code to fix.

Root cause

Tuesday's cache tuning raised the amount of memory the service allocates at startup to roughly 200MB, while its container memory limit remained at 128Mi. The process exceeds the limit shortly after starting, the kernel terminates it, the platform restarts it, and the cycle repeats. That is exactly the "fails for thirty seconds, recovers, fails again" pattern the customer described.

The customer's reasoning that a config value cannot cause this was understandable and wrong. A config value that controls memory allocation is every bit as capable of stopping a process as a code change.

Scoped fix

Either raise the allowance to fit the workload, or reduce what the workload asks for. Which is correct depends on whether the larger cache is wanted, so it is a decision to make with the customer rather than for them. Here the tuning was deliberate, so the limit moves:

kubectl -n tse-training edit deployment/orders-api
# limits.memory: 128Mi -> 512Mi
kubectl -n tse-training rollout status deployment/orders-api

Watch the restart count settle rather than assuming. A crash loop on a timer looks resolved for the first minute regardless of what you changed.

Customer update

The service was not crashing. It was being stopped by the platform for using more memory than it is allowed, which is why you saw a clean restart cycle rather than errors in the logs. The cache setting changed on Tuesday raised how much memory the service reserves at startup to around 200MB, and its memory allowance was still 128MB, so it exceeded the limit within seconds of starting every time.

That is why the timing lined up with a change you reasonably thought was unrelated: the setting is a configuration value, but what it configures is memory use. We have raised the allowance to accommodate the new cache size and the restarts have stopped.

Worth deciding on your side whether the larger cache is worth the extra memory, since the alternative fix is to reduce the cache back down. Either is fine, and I would rather you choose than have us pick for you.

Engineering escalation, if you needed one

Impact: intermittent failures on order lookups every few minutes since Tuesday. Evidence: Last State: Terminated, Reason: OOMKilled, Exit Code: 137; previous-instance logs end mid-run without an error; ALLOCATE_MB=200 against limits.memory: 128Mi. Confirmed: image, configuration, scheduling, Service routing. Ruled out: application defect, recent deploys, dependency failure. Suspected cause: a cache tuning change raised allocation without a corresponding limit change. Request: confirm the intended cache size so the limit can be set once deliberately rather than raised reactively.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The record of the previous run says it was terminated for exceeding its allowance. What does that settle?

• That the application is leaking memory.
• That the node ran out of memory.
• That the application threw an error it did not handle.
• That the process was stopped from outside rather than failing, so there is nothing in the code to fix.

Question 2 of 3The customer argued that a configuration value could not possibly stop a process. How would you answer that?

• A value that decides how much memory to reserve can stop a process just as thoroughly as any code change.
• They are right, so the cause must lie elsewhere.
• Configuration changes are always riskier than code changes.
• It depends on whether the change went through review.

Question 3 of 3The customer described it failing for about thirty seconds, recovering, then failing again. What produces that rhythm?

• Traffic arriving in bursts that overwhelm the service.
• It is stopped shortly after starting, brought back automatically, and then does the same thing again.
• A health check flapping between two states.
• Two instances competing for the same resource.

3 questions, none answered yet.

Why this one exists

A restarting container has two histories, and the useful one belongs to the instance that already died. Read the previous logs and the last terminated state before the current ones.

In an interview

Distinguishing an application crash from a platform kill is a senior-sounding distinction that is genuinely easy once you know where to look. Exit code 137 and a Reason of OOMKilled mean the kernel stopped the process, which is a completely different conversation from a stack trace.

Commands introduced

• kubectl describe pod
• kubectl logs --previous
• kubectl get pod -o jsonpath

Evidence layers

• last terminated state
• exit code
• resource limits