The service is down and there are no logs at all

Can I prove why a container never started, when it has produced no output?

The ticket

CUSTOMER TICKET: order service is down and we cannot see why

Account: Beacon Analytics (growth) Impact: order lookups unavailable to all users Started: after a config change last night

The order service is completely down since a config change went in last night. Our on-call engineer went looking and says there is nothing in the logs at all, which is what is confusing us. If it had crashed we would expect an error somewhere. Can you tell us what happened?

Your job

1. Treat the absence of logs as evidence and work out what it rules out.
2. Prove the cause before you name it.
3. Restore service and verify the customer's own workflow.

Working notes

The workload runs in the tse-training namespace on the kind-proveit context. Unlike the Docker track there is no file to edit and no tse apply: change the cluster directly, the way you would in production, then run tse check. tse reset puts the ticket's state back.

Track

Kubernetes

Time

about 35 minutes

Difficulty

Involved

Tier

Core

Do these first: A release went out and the new version never arrived

Start it

In a Codespace or a local clone:

tse start kubernetes/02-service-down-with-no-logs

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

The customer thinks the missing logs are the mystery. They are actually the best clue in the ticket, and saying so will change how they see the incident.

Work through what an empty log rules out. If the application had started and then failed, it would almost certainly have said something first: a stack trace, a connection error, a validation message. Nothing at all means your code was probably never reached.

That splits the world cleanly. Everything that happens before the process runs is still in play, and everything the application itself does is not.

So the question is no longer "what went wrong in the service". It is:

What has to succeed before the process is allowed to start?

Hint 2 of 3

The platform has to assemble quite a lot before it can hand control to your code: pull an image, resolve every value the container asks for, mount every volume, and apply security settings. Any one of those failing stops the container before the first line runs.

The previous ticket was the first of those. This one is not: the image is fine and the pods are not reporting a pull problem.

Look at the container's state and reason again, and note that the state itself names the category. Then go to the configuration the workload references. A value can be requested from a source that exists while the specific key it asks for does not, and that is enough to block startup entirely.

Compare what the workload asks for against what the configuration actually contains. Both halves are one command each.

Hint 3 of 3

kubectl -n tse-training get pods
kubectl -n tse-training describe pod -l app.kubernetes.io/name=orders-api | tail -25

The container state is CreateContainerConfigError, and the events name the missing key. Now compare the two halves. What the workload asks for:

kubectl -n tse-training get deployment orders-api \
  -o jsonpath='{.spec.template.spec.containers[0].env[*].valueFrom.configMapKeyRef.key}{"\n"}'

And what the configuration actually holds:

kubectl -n tse-training get configmap orders-api-config -o jsonpath='{.data}' | tr ',' '\n'

The workload asks for a key that the config change never created. The configuration source is correct, so the fix is on the workload side. Point it back at the key that exists, for example with:

kubectl -n tse-training edit deployment/orders-api

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: down with no logs

What the evidence proved

Command	What it proved	What it did not prove
kubectl logs	Refuses with container "app" ... is waiting to start: CreateContainerConfigError, so the code was never reached	Nothing about the application itself
describe pod	Container state is CreateContainerConfigError	Which value was missing
get events	The named key could not be found	Who removed or renamed it
get deployment -o jsonpath	The workload asks for APP_SECRET_V2
get configmap	The configuration holds APP_SECRET, not APP_SECRET_V2

The missing log was the most valuable evidence in the ticket, and the customer had already found it and discarded it. Asking for logs does not return nothing: it returns a refusal that names the container state outright, so the answer is handed over by the very command that appeared to be a dead end. No output at all means the process never ran, which eliminates the entire application as a suspect and leaves only what the platform does on its behalf: pull the image, resolve values, mount volumes, apply security settings.

Root cause

Last night's config change updated the Deployment to read APP_SECRET from a ConfigMap key named APP_SECRET_V2, but that key was never created. Kubernetes cannot assemble a container whose environment references a key that does not exist, so it stops before starting the process and reports CreateContainerConfigError.

The image was fine, the ConfigMap existed, and the application was never at fault. Only the specific key was wrong, which is why nothing looked obviously broken at a glance.

Scoped fix

Point the workload back at the key that exists:

kubectl -n tse-training edit deployment/orders-api
# change configMapKeyRef.key from APP_SECRET_V2 back to APP_SECRET
kubectl -n tse-training rollout status deployment/orders-api

The other correct fix is to add the APP_SECRET_V2 key to the ConfigMap, and which one is right depends on what the config change was trying to achieve. Ask before choosing. Here the ConfigMap is the source of truth, so the workload moves.

Customer update

Your engineer's observation about the missing logs was the key to this. There were no logs because the application never started: the change made last night pointed the service at a configuration key named APP_SECRET_V2, and that key does not exist in the configuration it reads from. The platform will not start a container whose configuration it cannot fully resolve, so it stopped before your code ran, which is exactly why there was nothing to log.

We have pointed the service back at the existing key and order lookups are working again. If the intention last night was to move to a new key name, the key needs creating first and then the service can be switched over. Happy to walk through that sequence with your team so the same gap does not reopen.

Engineering escalation, if you needed one

Impact: total outage of order lookups since the overnight config change. Evidence: container state CreateContainerConfigError; events name APP_SECRET_V2 as not found; the ConfigMap contains APP_SECRET only. Confirmed: image, scheduling, the ConfigMap object itself. Ruled out: application fault, image availability, resource pressure. Suspected cause: a rename applied to the consumer without being applied to the provider. Request: confirm whether the rename was intended, and whether other workloads took the same edit.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3Asking for output returned a refusal instead of text. Why is that refusal worth more than output would have been?

• It names the container's state outright, which hands over the next branch of the investigation with no further command.
• It proves the application crashed during startup.
• It shows that permission to read output is missing.
• It confirms the pod does not exist.

Question 2 of 3With the application eliminated as a suspect, what is left to investigate?

• The dependencies it calls and the services behind them.
• The work the platform does on the container's behalf before any code runs: fetching the image, resolving values, mounting volumes, applying security settings.
• The spare capacity available on the node.
• The network rules applied to the workload.

Question 3 of 3The configuration source existed and the workload still could not be assembled. What was actually wrong?

• The configuration source was empty.
• The workload lacked permission to read it.
• The workload asked for one specific name, and nothing with that name was there.
• The value stored there had the wrong type.

3 questions, none answered yet.

Why this one exists

Empty logs are evidence, not a lack of it. They prove the container never reached your code, which rules out the application entirely and points at the wiring Kubernetes does on its behalf.

In an interview

An absence of logs is treated as a dead end by most candidates, and it is actually a strong signal. It narrows the failure to everything that happens before the process runs: image, config references, secrets, mounts, and permissions. Saying that out loud is a very good answer.

Commands introduced

• kubectl describe pod
• kubectl get events --sort-by
• kubectl get configmap -o yaml

Evidence layers

• container state and reason
• cluster events
• configuration references