Same two addresses, and the certificate is fine

Can I prove how far the connection got before it failed?

The ticket

CUSTOMER TICKET: same address, same failure, and we checked it ourselves this time

Account: Ardent Logistics (enterprise) Impact: nightly export failing, two nights Started: Tuesday night, no change on our side

This is the third time this year and we would like it to be the last, so we have done the work before raising it.

Your engineer gave us a command in April that prints the dates and the names your server presents. We ran it. It is good until 2035 and it lists both of our addresses, so it is not April's problem and it is not May's problem either. We checked.

The old address still works. The one you moved us to fails on every attempt, and it fails instantly, which is new. In May it sat there for a moment and then complained at length. Now it comes back straight away with a much shorter message and a different number at the front of it.

We have not changed anything. Same job, same credentials, same payload, same two addresses we have been using since May.

We are not going back to the old address again. Please tell us what is actually wrong this time.

Your job

1. The customer says it is not the same problem as either previous ticket. They have been right before. Confirm it rather than assuming it.
2. The failure is faster and the number is different. Both of those are evidence. Work out which stage of the connection is failing before you look at anything else.
3. Fix it without moving the customer, and without touching what the two previous tickets fixed.

Working notes

The customer's integration runs in the client service, and GATEWAY_URL controls which address it calls:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client /app/upload.sh

Everything the platform team configures for this customer is in labs/networking/_stack/compose.override.yaml. After editing, run tse apply, then tse check.

Moving the customer back to the old address is not a fix. The grader checks both, and the customer has told you twice now that they will not accept it.

Track

Networking, DNS, TLS

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Do these first: The new address fails and the old one works

Start it

In a Codespace or a local clone:

tse start networking/03-same-two-addresses-and-the-certificate-is-fine

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Two of the customer's three sentences are measurements, and you have been treating them as complaints.

They said it fails instantly. They said the number at the front is different. Both of those are facts about the failure that arrived before you gathered a single piece of evidence, and both of them narrow the problem enormously.

Think about what a connection is actually made of. The caller has to turn a name into an address. It has to reach that address and be accepted. Then the two sides have to agree that each is who it claims to be. Then, finally, the request is sent and the application answers.

Those are four separate stages and they fail in four separate ways. The two previous tickets both failed at the third one, and a failure at the third stage takes a moment, because the two sides had to reach each other and exchange things before either could object. The customer is telling you this one does not take a moment.

So before anything else, run the upload and read what comes back. Not the paragraph, which is the same paragraph you have read twice. The number.

Then ask yourself the only question worth asking at this point: if the caller never got as far as being shown anything, what have you just ruled out? And what is left?

Hint 2 of 3

Three failures from the same service, in order:

curl: (60) SSL certificate OpenSSL verify result: certificate has expired (10)
curl: (60) SSL: no alternative certificate subject name matches target hostname 'reports'
curl: (7) Failed to connect to reports:8443 after 1 ms: Could not connect to server

The first two are the same number. This one is not, and that is the whole difference. Exit 60 means a connection was established and the caller then refused what it was shown. Exit 7 means there was nothing to refuse. Nothing ever accepted the connection, so nothing was ever presented, so the customer is right that it cannot be about what the server presents. They have ruled out the correct thing for the correct reason.

The name resolved, though. Look at the message again: it names a port and it says connect, so the caller got far enough to have an address to dial. A name that does not resolve fails earlier than this and says so differently.

So the name turned into an address, and that address did not accept a connection. There is one gateway, it is healthy, and the other name reaches it perfectly. Read what each name actually resolves to, and read them the same way so the comparison means something:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client sh -c 'getent hosts reports; getent hosts gateway'

Two names for one service. Look at what came back for each of them, and then look at the first one again.

Hint 3 of 3

The evidence:

127.0.0.1         reports  reports
192.0.2.2         gateway  gateway

gateway points at the machine the service runs on. reports points at 127.0.0.1, which is not a place. It is a word every machine reads as "me", so the customer's job dialed itself, found nothing listening on 8443, and stopped in about a millisecond. That is the whole incident.

Confirm it came from the resolver rather than from something on the client:

docker compose -f labs/networking/_stack/compose.yaml \
               -f labs/networking/_stack/compose.override.yaml \
               exec client nslookup -type=A reports.

The dot on the end is not a typo. It makes the name absolute, so the resolver asks for exactly what you typed instead of trying it against each search domain in turn first. Worth having in your fingers: it is the difference between asking a question and asking several.

Nothing here is broken. The resolver is healthy, it answered immediately, and it answered with exactly what it was configured to say. A wrong answer given confidently by a working service is the reason this took two nights: every component reports itself fine, because every component is fine.

The records are in compose.override.yaml, as name=target. Point it back at the service instead of at the caller:

services:
  resolver:
    environment:
      LAB_RECORDS: "reports=gateway"

Then tse apply and tse check.

Naming the service rather than an address is deliberate and worth copying. The gateway gets a new address every time the stack is recreated, so a record holding the address it had this morning is a record that will be wrong again without anybody touching it.

Do not fix this by pointing the customer at the other name. They have told you twice that they will not accept it, and the grader checks both.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: same two addresses, and the certificate is fine

What the evidence proved

Command	What it proved	What it did not prove
/app/upload.sh against reports	Exit 7, Failed to connect, in about a millisecond	Nothing about trust. Nothing was presented to distrust
The same upload against gateway	Accepted immediately, same client, same credentials, same payload
getent hosts reports	The name resolves, and it resolves to 127.0.0.1
getent hosts gateway	The same read against the name that works, returning the gateway
nslookup reports	The answer came from the resolver, which is healthy and answered at once	Nothing was wrong with the caller's own configuration

The exit code is the diagnosis and it arrived before any evidence was gathered. Exit 60 in both previous tickets meant a connection was established and the caller then refused what it was shown. Exit 7 means nothing accepted the connection, so nothing was ever presented, so no question about trust can arise. The customer reasoned their way to that from the timing alone and they were right.

It also proves the name resolved. A failure to connect names a port and an address, which the caller can only have because the name became one. That places the fault between resolution and connection, and there is exactly one thing in that gap: what the name resolved to.

The two getent reads are the whole finding, and they have to be run together. One name reaches the service and one does not, against the same client, on the same machine, at the same moment.

Root cause

The internal resolver held reports pointing at 127.0.0.1. The record resolved, instantly and successfully, to the caller's own loopback address. The customer's job connected to itself, found nothing listening on 8443, and stopped.

127.0.0.1 is not an address that goes anywhere. It means "this machine" on every machine that reads it, so a record carrying it is correct only for whoever is standing on the box at the time. That is the fingerprint of this fault: somebody testing the endpoint on the gateway itself pointed the record at their own loopback, it worked from where they were standing, and it was wrong from everywhere else in the world including where the customer is.

Nothing was down. The resolver was healthy and fast, the gateway was healthy, the certificate was valid until 2035 and covered both names, and the customer's integration was correct. Every component reported itself fine because every component was fine. The only thing wrong was the content of an answer, and nothing checks the content of an answer.

That is why this took two nights and why the previous two tickets did not.

Scoped fix

In labs/networking/_stack/compose.override.yaml:

services:
  resolver:
    environment:
      LAB_RECORDS: "reports=gateway"

Then:

tse apply
tse check

The record names the service rather than an address, and that is the part to carry forward. The gateway is given a new address every time the stack is recreated, so a record holding today's address is a record that goes wrong on its own, later, with nobody having touched it. Pointing a name at a name is what keeps it correct through a rebuild.

Not the fix: moving the customer to the other address. It works, it takes ten seconds, and they have now told you twice they will not accept it. The check tries both addresses.

Also not the fix: deleting the record. The failure stops printing that particular error and the customer still cannot upload, which is why the check reads what the name resolves to rather than only whether the old message went away.

Customer update

You were right on both counts, and the reasoning you did before raising this saved us the first hour. It was not the same problem as either previous ticket, and the speed of the failure was the evidence that proved it.

What went wrong is that the address book entry for the address you use was pointing at the wrong place. Your job looked up the address correctly, got an answer immediately, and the answer sent it back to your own machine instead of to us. That is why it failed in a fraction of a second and why it never got far enough to complain about anything else. Your job, your credentials, your export and our service were all working the entire time.

This was ours, and it was ours in a way that none of our monitoring could see, because every part of the system was healthy and one of them was simply answering with the wrong value. We have corrected the entry and pointed it at the service by name, so it stays correct the next time the service moves.

Nothing needs to change on your side. We will confirm tonight's export landed.

Engineering escalation, if you needed one

Impact: two nightly exports lost for Ardent Logistics. Third incident on the same endpoint this year and the first that no monitoring could have caught. Evidence: curl: (7) Failed to connect in about a millisecond on reports; the same upload accepted on gateway; getent hosts reports returns 127.0.0.1 while getent hosts gateway returns the gateway. Confirmed: resolver healthy and answering, gateway healthy, certificate valid to 2035 and covering both names, customer integration unchanged. Ruled out: expiry, the names on the certificate, credentials, the payload, and the customer's client. The exit code rules out everything above the connection before any of it is tested. Suspected cause: the record was pointed at loopback during testing on the gateway itself and never pointed back. Request: two things. Nothing verifies that a record we publish resolves to the service it names from anywhere other than the machine it was created on, and nothing refuses a loopback address in a record that is meant to be reachable by somebody else. Either one would have caught this at the moment it was made rather than two nights later.

The record is a one-line fix. The durable part is that every health check in the path was green throughout, because none of them ask whether an answer is correct, only whether one arrived.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The first two tickets from this customer failed with exit 60 and this one failed with exit 7. What does that difference establish before any other evidence is gathered?

• That the failure is more serious, since the connection did not even open
• That nothing was ever presented to the caller, so every question about what the server serves is already answered
• That the service is down, since a connection was refused
• That the customer's client was upgraded and now reports failures differently

Question 2 of 3The name `reports` resolved successfully and the upload still failed at the connection. Why is resolving successfully not the same as resolving correctly?

• Because resolution can succeed and hand back an answer that reaches nothing useful, and nothing in the lookup checks that
• Because the answer had not finished propagating and would have become correct on its own
• Because the resolver was unhealthy and returning corrupted answers
• Because the name should have been fully qualified and the short form is ambiguous

Question 3 of 3The fix set the record to `reports=gateway` rather than to the gateway's address. Why does that matter after the ticket is closed?

• Because naming the service is faster to type and easier for the next person to read
• Because an address is only correct until the service is next recreated, so a record holding one goes wrong later with nobody having touched it
• Because addresses in configuration are a security problem
• Because the resolver cannot store literal addresses

3 questions, none answered yet.

Why this one exists

A connection is a sequence of stages and each one fails its own way: resolving the name, reaching the address, agreeing on trust, then the application answering. A name that resolves is not a name that resolves correctly, and a wrong answer from a healthy resolver is indistinguishable from a working network until somebody reads the answer instead of assuming it.

In an interview

The third failure on the same pair of addresses, and the one that separates people who have learned a fix from people who have learned a method. The two before it were both refusals by the caller after a connection was established. This one never establishes anything, and the exit code says so before any evidence is gathered. Being able to name which stage a failure belongs to, and to say what that rules out, is most of what makes somebody fast at this.

Commands introduced

• getent hosts
• nslookup

Evidence layers

• the exit code, and which stage of the connection it belongs to
• what the name the customer calls actually resolves to
• what the name that works resolves to, read the same way