A release went out and the new version never arrived

Can I prove whether the workload was ever able to start?

The ticket

CUSTOMER TICKET: this morning's release never went live

Account: Northwind Freight (enterprise) Impact: order lookups unavailable to all users Started: after the 08:40 release

We were told the release went out at 08:40 but the new version is definitely not live, and now the order screen is not loading at all. Our deploy pipeline reported success, so we assumed it worked. Nothing changed on our side.

Your job

1. Prove how far the release actually got before you name a cause.
2. Restore service with the smallest correct change.
3. Verify the customer's own workflow, not just that something responds.

Working notes

The workload runs in the tse-training namespace on the kind-proveit context. Unlike the Docker track there is no file to edit and no tse apply: change the cluster directly, the way you would in production, then run tse check. tse reset puts the ticket's state back.

Track

Kubernetes

Time

about 30 minutes

Difficulty

Straightforward

Tier

Core

Start it

In a Codespace or a local clone:

tse start kubernetes/01-new-version-never-came-up

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The pipeline reporting success is not the same fact as the release running. A pipeline usually reports that it submitted the change, not that the change became live, and those two things fail independently.

So the smallest useful question is not "why is the new version missing". It is:

How far did this actually get?

There is a chain here: the desired state was accepted, something was scheduled, something was created, and something was started. Each link leaves its own evidence, and the break will be at exactly one of them.

Start by looking at what is running now, and notice that the state you find is already telling you which link failed.

Hint 2 of 3

The workload was accepted and something was scheduled, so this is not a capacity or permissions problem. What you have is a container that exists on paper and has never run.

Resist going to the logs. A container that never started has produced no output, so an empty log is guaranteed here and tells you nothing you did not already know.

The reason a container cannot start is recorded by the platform, not by the application. Two places hold it:

• The detailed description of the object itself, which includes the container's current state and a reason string.
• The recent event stream, which is often blunter and quotes the underlying error directly.

Read the reason. It names the resource it could not obtain.

Hint 3 of 3

kubectl -n tse-training get pods

The pods report ImagePullBackOff. Now get the reason rather than the state:

kubectl -n tse-training describe pod -l app.kubernetes.io/name=orders-api | tail -20
kubectl -n tse-training get events --sort-by=.lastTimestamp | tail -10

The events quote the pull failure and the exact reference that failed. Compare that reference against what was intended:

kubectl -n tse-training get deployment orders-api \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

The repository is right and the tag is not. Correct it in place:

kubectl -n tse-training set image deployment/orders-api app=python:3.12-alpine
kubectl -n tse-training rollout status deployment/orders-api

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the release never went live

What the evidence proved

Command	What it proved	What it did not prove
kubectl get pods	Pods exist and are stuck in ImagePullBackOff	Nothing about why the pull failed
kubectl get events	The registry rejected the reference by name	Nothing about who set it
describe pod	The container has never started, so there are no logs to read
get deployment -o jsonpath	The image tag is 3.12-alpne rather than 3.12-alpine

Two things worth stating explicitly, because both are useful in the write-up: the desired state was accepted and pods were scheduled, so this was never a permissions or capacity problem. And the pipeline was honest: it reported that it submitted the change, which it did.

Root cause

The release set the container image to a tag that does not exist, a single-character typo in 3.12-alpine. Kubernetes accepted the Deployment because the spec is valid, created the pods, and then could not pull the image, so the containers never started. The previous version's pods were replaced as part of the rollout, which is why the outage is total rather than partial.

Scoped fix

kubectl -n tse-training set image deployment/orders-api app=python:3.12-alpine
kubectl -n tse-training rollout status deployment/orders-api

Nothing else needs touching. The Service, the configuration, and the application were all proven fine by the evidence above.

Customer update

The release was accepted but never started. The deployment referenced a container image tag that does not exist, so the platform created the new instances and then could not retrieve the image for them. That is why your pipeline reported success: it submitted the change correctly, and the failure happened afterwards when the image was requested. We have corrected the reference and the service is serving order lookups again.

Worth raising with whoever owns your pipeline: a deploy that cannot pull its image will report success and take the service down. Adding a rollout status check after the deploy step would have caught this in the pipeline instead of in production.

Engineering escalation, if you needed one

Impact: total outage of order lookups from 08:40. Evidence: pods in ImagePullBackOff; events show the registry rejecting python:3.12-alpne; deployment spec confirms the tag. Confirmed: scheduling, permissions, configuration, and the Service. Ruled out: application fault, resource pressure, networking. Suspected cause: a typo in the image tag in the release. Request: confirm whether the deploy pipeline waits on rollout status, since it currently reports success for a release that never became live.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The release pipeline reported success and the customer is entirely offline. Can both accounts be true?

• No, the pipeline must have reported incorrectly.
• Yes. The desired state was valid and was accepted, and accepting a description is a different event from running it.
• Yes, because a rollout always takes time to become available.
• No, since a successful release means running workloads by definition.

Question 2 of 3There is no application output at all to read. What does that absence establish?

• The logging setup is broken.
• The application is failing silently.
• The container never started, so the code was never reached and no application-level cause is possible.
• The output was rotated away before anyone looked.

Question 3 of 3Why is the customer completely down rather than merely degraded?

• Every node in the cluster was affected.
• Traffic was never spread across more than one instance.
• The platform removed the workload once the rollout failed.
• The working instances were replaced before the new ones had proved they could start.

3 questions, none answered yet.

Why this one exists

Pod state and container state are different things. A pod can be scheduled and still have a container that has never run, and the reason lives in the events rather than in the application.

In an interview

The first Kubernetes question most candidates get, and the one where reaching for logs immediately marks you out. A container that never started has no logs to read, and knowing to go to describe and events instead is the whole answer.

Commands introduced

• kubectl get pods
• kubectl describe pod
• kubectl get events
• kubectl rollout status

Evidence layers

• pod phase and container state
• cluster events
• image reference