Every pod is healthy and the service still answers nothing

Can I prove whether the Service is actually routing to any pods?

The ticket

CUSTOMER TICKET: order lookups return nothing, but everything looks healthy

Account: Tidewater Health (enterprise) Impact: order lookups failing for all users Started: after yesterday's platform work

Our order lookups have stopped returning anything since some platform work yesterday. Our team checked and everything reports healthy on your side, so they are stuck. The application does not error, it just gets nothing back.

Your job

1. Everything healthy plus nothing working is a specific pattern. Work out what it points at.
2. Prove where the request stops before you name a cause.
3. Restore service and verify the customer's own workflow.

Working notes

The workload runs in the tse-training namespace on the kind-proveit context. Unlike the Docker track there is no file to edit and no tse apply: change the cluster directly, the way you would in production, then run tse check. tse reset puts the ticket's state back.

Track

Kubernetes

Time

about 35 minutes

Difficulty

Involved

Tier

Core

Do these first: A release went out and the new version never arrived

Start it

In a Codespace or a local clone:

tse start kubernetes/04-service-reachable-but-returns-nothing

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

"Everything reports healthy and nothing works" is not a contradiction. It is a pattern, and it points somewhere specific.

If the workload is healthy, and the application is fine, and the client gets nothing rather than an error, then the request is not reaching the application at all. Something in between is accepting it and delivering it nowhere.

So stop checking health. You already have that answer, and repeating it is the loop most people get stuck in. The question now is:

Does the thing in front of the application actually know about any of it?

The connection between a front door and the workload behind it is not a hardcoded link. It is a query, evaluated continuously, and a query can return nothing while looking completely correct.

Hint 2 of 3

The front door here matches workloads by their labels rather than by name. That matching happens constantly, and its result is written down in an object you can read directly.

That object is where the emptiness becomes visible. The front door itself will look perfectly healthy whether it matches two workloads or none, because from its own point of view nothing is wrong: it was asked to route to whatever matches, and it is doing exactly that.

Read the current result of the query. If it is empty, then you have proven the break is in the matching and not in the workload, the application, or the network.

Then compare the two halves the query joins: what is being asked for, and what the workload actually carries.

Hint 3 of 3

kubectl -n tse-training get endpointslices -l kubernetes.io/service-name=orders-api

No addresses. The Service is selecting nothing, which proves the failure is in the matching rather than in the workload or the application.

Now read both halves. What the Service asks for:

kubectl -n tse-training get service orders-api -o jsonpath='{.spec.selector}{"\n"}'

And what the pods actually carry:

kubectl -n tse-training get pods --show-labels

The values differ. Yesterday's platform work changed one side and not the other. Correct the Service so its selector matches the labels the pods already have, since the pods are running correctly and should not be disturbed:

kubectl -n tse-training edit service orders-api
kubectl -n tse-training get endpointslices -l kubernetes.io/service-name=orders-api

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: healthy everywhere, routing nowhere

What the evidence proved

Command	What it proved	What it did not prove
kubectl get pods	Pods are Running and Ready	Nothing about routing
get endpointslices	The Service has no endpoints at all	Why the selection is empty
get service -o jsonpath	The selector asks for app.kubernetes.io/name: orders
get pods --show-labels	The pods carry app.kubernetes.io/name: orders-api

Empty endpoints is the fact that turns this ticket. It proves the break is in the selection rather than in the workload, the application, or the network, and it is invisible from everything the customer had already checked.

Root cause

A Service does not link to a Deployment. It runs a label query continuously and routes to whatever currently matches. Yesterday's platform work changed the Service's selector to orders while the pods kept the label orders-api, so the query matches nothing and the Service routes to nothing.

Every health check the customer ran was accurate. The pods really were healthy, the Service really did exist, and DNS really did resolve. A Service with no endpoints still resolves and still accepts connections. It simply has nowhere to send them, which is why the client gets nothing back rather than an error.

Scoped fix

The pods are running correctly and should not be disturbed, so the Service moves:

kubectl -n tse-training edit service orders-api
# spec.selector: app.kubernetes.io/name: orders -> orders-api
kubectl -n tse-training get endpointslices -l kubernetes.io/service-name=orders-api

Confirm endpoints appear before declaring it fixed. Endpoints appearing is the proof, not the Service looking healthy again, which it did throughout.

Customer update

Your team was right that everything reports healthy, and that was genuinely true rather than a false reading. The problem was in between: the internal address for the order service selects the instances it should route to by label, and yesterday's platform work changed the label it looks for without changing the label the instances carry. The result is an address that resolves and accepts connections but has nothing behind it, which is why requests returned nothing instead of failing outright.

We have corrected the selection and confirmed the order service is now receiving traffic and returning results. Nothing was wrong with the application or its instances at any point.

If the same platform work touched other services, they are worth checking, as the same edit would produce the same silent failure.

Engineering escalation, if you needed one

Impact: order lookups returning empty for all users since yesterday's platform work. Evidence: EndpointSlice for orders-api has no addresses; Service selector is app.kubernetes.io/name=orders; pods carry app.kubernetes.io/name=orders-api; pods Running and Ready throughout. Confirmed: pod health, application health, DNS resolution. Ruled out: application fault, image, resource pressure, readiness. Suspected cause: a selector edit during platform work not matched by a label change on the workload. Request: identify which other Services were edited in the same change, since this failure mode reports healthy everywhere.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The customer verified the workload was healthy, the service existed, and the name resolved. All three results were accurate. Why did none of it help?

• They were looking at the wrong environment.
• Their tooling was showing stale information.
• Each one covers a different link, and the broken link sits between them: which instances are being selected right now.
• They should have restarted the workload before checking anything.

Question 2 of 3The client receives nothing back rather than an error. Why?

• The request is being silently dropped by a network rule.
• The response is being lost on the way back.
• The workload is too slow to answer in time.
• The name resolves and the connection is accepted whether or not there is anywhere to forward it.

Question 3 of 3What is the relationship between a service and the workload sitting behind it?

• There is no fixed relationship. A label query runs continuously and traffic goes to whatever matches at that moment.
• The service holds a reference to the deployment that created it.
• They are bound together when the workload is first created.
• The workload registers itself when it starts up.

3 questions, none answered yet.

Why this one exists

A Service is a label selector, not a link. It can exist, resolve, and accept connections while selecting nothing at all, and the endpoints are the only place that emptiness is visible.

In an interview

The classic Kubernetes networking question. Everything you look at first reports healthy, which is why candidates loop on pods and logs. Knowing that a Service is just a label query, and that its EndpointSlice is where that query's result becomes visible, is the whole test.

Commands introduced

• kubectl get endpointslices
• kubectl get pods --show-labels
• kubectl describe service

Evidence layers

• pod readiness
• service selector and pod labels
• endpoints