Same symptom as the last ticket, and the labels are correct

Can I prove why a running pod is being excluded from its Service?

The ticket

CUSTOMER TICKET: order lookups return nothing again

Account: Juniper Labs (starter) Impact: order lookups failing for all users Started: after a monitoring change this morning

This looks like the issue Tidewater reported last week. Order lookups return nothing and everything reports healthy. Our platform engineer already checked the labels this time because he saw your write-up, and he says they match, so he is out of ideas. He did adjust some health check settings this morning but those only affect monitoring.

Your job

1. Confirm what the customer ruled out, then keep going. The labels really are correct.
2. Prove why the request still does not reach the application.
3. Restore service and verify the customer's own workflow.

Working notes

The workload runs in the tse-training namespace on the kind-proveit context. Unlike the Docker track there is no file to edit and no tse apply: change the cluster directly, the way you would in production, then run tse check. tse reset puts the ticket's state back.

Track

Kubernetes

Time

about 35 minutes

Difficulty

Involved

Tier

Core

Do these first: Every pod is healthy and the service still answers nothing

Start it

In a Codespace or a local clone:

tse start kubernetes/05-healthy-pods-still-no-traffic

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Take the customer at their word. The labels really do match, and repeating that check is how you waste the first ten minutes of this ticket.

What is worth noticing is that the previous incident and this one share a symptom, and you already know that symptom means requests are not reaching the application. So the front door is still selecting nothing, or selecting something it refuses to use.

That second possibility is the new one. A workload can match perfectly and still be deliberately excluded, because matching is not the only condition for receiving traffic.

Look carefully at how the workloads are reported. There are two separate pieces of state there that people read as one, and only one of them controls whether traffic arrives.

Hint 2 of 3

Running and Ready are different columns, and the difference is the entire ticket. Running means the process is alive. Ready means the platform is willing to send it real traffic.

A workload that is alive but not ready is removed from the front door on purpose. This is a feature: it is how a starting instance avoids receiving requests before it can serve them. It also means the exclusion is completely silent, because nothing has failed. The container is fine and the logs are clean.

Readiness is decided by a check the platform runs against the application. If that check is pointed somewhere the application does not answer, it will never pass, and the workload will never be admitted no matter how healthy it is.

The customer mentioned adjusting health check settings this morning. Go and read what those checks are actually asking for, and compare it against what the application actually serves.

Hint 3 of 3

kubectl -n tse-training get pods

Note the READY column shows 0/1 while STATUS shows Running. The workloads are alive and are not being admitted.

kubectl -n tse-training describe pod -l app.kubernetes.io/name=orders-api | grep -A5 'Readiness'
kubectl -n tse-training describe pod -l app.kubernetes.io/name=orders-api | tail -12

The readiness check is aimed at a port the application does not listen on, so it fails every time and the workload is never admitted to the Service. The events show the probe failing with a connection refused.

Compare it against what the container actually exposes:

kubectl -n tse-training get deployment orders-api \
  -o jsonpath='{.spec.template.spec.containers[0].ports}{"\n"}'

Point the readiness check back at the port the application serves:

kubectl -n tse-training edit deployment/orders-api
kubectl -n tse-training get endpointslices -l kubernetes.io/service-name=orders-api

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: correct labels, still no traffic

What the evidence proved

Command	What it proved	What it did not prove
get service and get pods --show-labels	The labels genuinely match. The customer was right	Nothing about routing
get endpointslices	Still no ready endpoints	Why, given the labels match
kubectl get pods	STATUS is Running and READY is 0/1
describe pod	The readiness probe targets port 8081 and fails
-o jsonpath on ports	The container serves 8080 only

The distinction that resolves this: matching the selector is necessary for routing but not sufficient. A pod also has to be ready, and readiness is a separate condition that the Service enforces silently.

Root cause

This morning's health check adjustment pointed the readiness probe at port 8081. The application listens only on 8080, so every readiness check is refused and the probe never passes. A pod that is not ready is deliberately excluded from its Service's endpoints, so the Service has nothing to route to.

Nothing failed. The container is alive, the application is serving correctly on its real port, the logs are clean, and the labels match. The platform is doing precisely what it was configured to do, which is why this is so quiet.

Why the previous fix did not apply

The customer's engineer checked the labels because of the earlier write-up, found them correct, and stopped. That was good instinct and an incomplete model. Both tickets end in an empty endpoint list, and there are two separate routes to that state:

Cause	Pod STATUS	Pod READY	Labels
Selector mismatch	Running	1/1	do not match
Readiness failing	Running	0/1	match

The READY column is what separates them, and it is one column away from the one everybody reads.

Scoped fix

kubectl -n tse-training edit deployment/orders-api
# readinessProbe.httpGet.port: 8081 -> 8080
kubectl -n tse-training rollout status deployment/orders-api
kubectl -n tse-training get endpointslices -l kubernetes.io/service-name=orders-api

Customer update

Your engineer was right that the labels match, and that check was worth doing. This is a different cause with the same symptom.

The health check change this morning pointed the readiness check at port 8081, and the order service listens on 8080. Readiness checks do more than report status: an instance that is not ready is deliberately kept out of the load balancing pool, so the check failing meant every instance was excluded even though all of them were running and serving correctly. That is why nothing errored and nothing appeared in the logs.

We have pointed the readiness check back at the port the service listens on and traffic is flowing again. Worth knowing for the future: health check settings are not monitoring-only, they actively control whether an instance receives traffic.

Engineering escalation, if you needed one

Impact: order lookups returning empty for all users since the morning health check change. Evidence: pods Running with READY 0/1; readiness probe targets 8081; container exposes 8080 only; probe failures with connection refused in events; EndpointSlice empty; Service selector and pod labels match. Confirmed: labels, application health, image, configuration. Ruled out: the selector mismatch seen previously, application fault. Suspected cause: a probe port edit applied without checking the container's exposed port. Request: confirm whether the same change was applied to other workloads, and whether probe ports can be validated against container ports at admission.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The customer's engineer checked the labels, found them correct, and stopped there. What was incomplete about that?

• The labels were not in fact correct.
• Being selected is necessary for traffic but not sufficient, and the second condition is enforced without announcing itself.
• Labels should have been checked with a different command.
• They should have read the earlier writeup more carefully.

Question 2 of 3The workload reports as running while showing zero of one ready. How should that be read?

• As a passing state that will resolve itself.
• As an inconsistency in the reporting.
• The process is alive, and the platform has separately concluded it is not fit to be sent anything.
• The container has started but its image is still downloading.

Question 3 of 3Nothing crashed, nothing errored, the output is clean and the labels match. What does that combination suggest about where to look?

• At the application's own code, since everything around it is healthy.
• At the network path between the client and the workload.
• At recent changes to the data the service depends on.
• At the settings the platform enforces on the workload's behalf, because a system following a wrong instruction correctly produces no errors at all.

3 questions, none answered yet.

Why this one exists

Readiness is a routing decision, not a health opinion. A failing readiness probe removes a pod from its Service while leaving the container running and the logs clean.

In an interview

This is the follow-up an interviewer asks after you solve the selector version. Running and Ready are different columns for a reason, and a pod that is Running but not Ready is deliberately kept out of its Service. Candidates who fixed the previous ticket by pattern often check labels, find them correct, and stall.

Commands introduced

• kubectl get pods
• kubectl describe pod
• kubectl get endpointslices

Evidence layers

• ready column versus status column
• probe configuration
• endpoint readiness conditions