The file is right there and the service cannot read it

Can I prove which account the process is actually running as?

The ticket

CUSTOMER TICKET: daily report stopped submitting and the config is untouched

Account: Ardent Logistics (enterprise) Impact: no daily report submitted since the security work Started: the morning after your hardening pass

Our daily report has not gone out since your team did the security work last week. The job says it cannot read its own configuration.

We checked, and the file is exactly where it has always been, with the same contents it has always had. Nobody has touched it. We can open it ourselves without any trouble.

The only thing that changed is the security work, and your engineer told us that change was a no-op, that it just wrote down explicitly what was already the case. So either that is not true, or something else is going on.

Your job

1. Prove which account the process is running as, rather than which one you expect it to be running as.
2. Work out what the change actually changed. The engineer was not lying.
3. Restore the report with the smallest correct change, and keep the hardening.

Working notes

The job runs in the worker service. You can run it on demand:

docker compose -f labs/linux/_stack/compose.yaml \
               -f labs/linux/_stack/compose.override.yaml \
               exec worker python3 /app/reporter.py

The security change is in labs/linux/_stack/compose.override.yaml. After editing, run tse apply, then tse check.

Running the service as a more powerful account would make the error go away. Do not. The grader checks for it, and so would a reviewer.

Track

Linux and CLI

Time

about 30 minutes

Difficulty

Involved

Tier

Core

Do these first: The disk has plenty of space and the writes still fail

Start it

In a Codespace or a local clone:

tse start linux/02-the-file-is-right-there-and-unreadable

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Two claims are being run together, and separating them is the whole exercise.

The customer says the file is there, unchanged, and that they can open it. All of that is true and you can confirm it yourself. What none of it establishes is the thing that actually matters:

Can this process, running as whatever it is running as, read that file?

"The file exists" and "this process can read it" are different claims with different evidence. Being able to open something yourself proves nothing about another account, and the customer opening it proves nothing about the service.

So stop looking at the file for a moment and look at the reader. Run the job and read what it tells you about itself. It reports more than the error.

The other thing to hold on to: your engineer said the change was a no-op, and they believed it. Take that seriously rather than dismissing it. A change that genuinely looks like it sets things to what they already were, and is not, is much more interesting than somebody being careless.

Hint 2 of 3

Get both halves and put them side by side.

What the file wants:

docker compose -f labs/linux/_stack/compose.yaml \
               -f labs/linux/_stack/compose.override.yaml \
               exec worker ls -l /etc/reporting/credentials.conf

What the process has:

docker compose -f labs/linux/_stack/compose.yaml \
               -f labs/linux/_stack/compose.override.yaml \
               exec worker id

The file is owned by one account and readable by one group. The process is not that account. So the only question left is whether it is in that group.

An identity on this system is three things, not one:

Part	What it is
The user	who you are
The primary group	the group you count as by default
Supplementary groups	every other group you also belong to

The third is the one people forget, and it is the one that grants access here. The image puts the service account into the group that owns the credentials precisely so it can read them.

Now read the change from the hardening pass again, and ask what it says about each of those three parts. It names two of them. Ask yourself what happens to the third when you name the first two explicitly.

Hint 3 of 3

The evidence, side by side:

-rw-r----- 1 root reporting 78 /etc/reporting/credentials.conf
uid=1000(app) gid=1000(app) groups=1000(app)

The file is readable by root and by anyone in reporting. The process is app, and its group list contains only app. It is not in reporting, so it gets nothing.

Now the part that makes this worth an exercise. The hardening change was:

user: "1000:1000"

That is the same user id and the same group id the image already ran as. Your engineer was telling the truth: as far as those two numbers go, it changed nothing.

What it also did, invisibly, is replace the entire group membership. Naming a user and a group explicitly means exactly those, so the supplementary groups the image had set up were dropped. groups=1000(app),2000(reporting) became groups=1000(app).

The fix keeps the hardening and puts back what it removed:

services:
  worker:
    user: "1000:1000"
    group_add:
      - "2000"

Then tse apply and tse check.

Do not run it as root. It would work, the report would submit, and you would have quietly reverted a security change to fix a one-line group problem.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the file is right there and unreadable

What the evidence proved

Command	What it proved	What it did not prove
Running the report	[Errno 13] Permission denied on a path that exists	Nothing about who was denied
ls -l on the file	Owned root:reporting, mode 0640, unchanged	Nothing about the reader
id inside the container	uid=1000(app) gid=1000(app) groups=1000(app)
The hardening change	user: "1000:1000", the same two numbers as before

The customer's three claims were all true. The file is there, it is unchanged, and they can open it. None of them was ever evidence about this process, which is the distinction the exercise exists to build.

The engineer's claim was also true, and more interestingly so. user: "1000:1000" really does name the user and group the service already had. It looks like writing down what was already the case, and for those two numbers it is.

Root cause

Naming a user and a group explicitly replaces the process's entire group membership with exactly those, which drops every supplementary group the image had configured.

The image puts app into reporting on purpose, because reporting owns the credentials file at mode 0640. After the hardening pass the process ran as the same user, with the same primary group, and with that membership gone:

before  uid=1000(app) gid=1000(app) groups=1000(app),2000(reporting)
after   uid=1000(app) gid=1000(app) groups=1000(app)

Two of the three parts of the identity were preserved. The third was silently discarded, and it was the one granting access.

Scoped fix

In labs/linux/_stack/compose.override.yaml, keep the hardening and restore what it removed:

services:
  worker:
    user: "1000:1000"
    group_add:
      - "2000"

Then:

tse apply
tse check

Not the fix: running the service as root. It makes the error go away, and it undoes a security change in order to solve a group membership problem. The check asserts the process is still uid 1000 for exactly that reason.

Also not the fix: loosening the file to 0644. That grants every account on the machine read access to a credential in order to grant one account the access it was already supposed to have.

Customer update

Your daily report stopped submitting because the service lost its membership of the group that owns its configuration file. You were right that the file is untouched, and our engineer was right that the change looked like a no-op: it pinned the service to the same account it was already using. What it also did, which is not obvious, is reset the list of additional groups that account belongs to, and one of those was what allowed it to read the file.

We have restored that group membership and kept the security change in place. Your report submitted successfully on the next run. Nothing about the file or its contents was altered at any point, and no other account gained access to it as part of the fix.

I will check whether the same hardening was applied to any other service that depends on group membership, and come back to you by Wednesday either way.

Engineering escalation, if you needed one

Impact: daily report submission failed for Ardent Logistics from the morning after the hardening pass, no reports delivered since. Evidence: [Errno 13] Permission denied on /etc/reporting/credentials.conf, which is root:reporting mode 0640; id in the container reports groups=1000(app) where the image configures 1000(app),2000(reporting). Confirmed: the file is present and unchanged, the service runs, the account and primary group are what the hardening intended. Ruled out: a change to the file, a change to its ownership, a missing file, a bad path. Suspected cause: user: "1000:1000" replaces the full group set, dropping supplementary groups configured in the image. Request: the hardening pass was applied across several services in the same change. Can we identify every service whose image adds a supplementary group, because each of those is either already broken or one restart away from it.

That last line is why this is worth escalating rather than just fixing. One service failing is a ticket. A hardening pass applied uniformly to services that relied on group membership is a queue of identical tickets arriving over the next few weeks, each looking unrelated.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The customer said the file was unchanged and that they could open it themselves. What did that establish?

• That the problem had to be in the application rather than in permissions
• That the file exists and their account can read it, neither of which is a claim about the process that failed
• That the ownership and mode must have been correct
• That someone had modified the file after the hardening pass

Question 2 of 3The engineer said the change was a no-op, and the numbers in it really were the ones already in use. What did it change?

• The order in which the service starts relative to its dependencies
• Nothing, and the real cause is elsewhere
• The supplementary group memberships, which naming a user and group explicitly replaces entirely
• The file's mode, by applying a stricter default

Question 3 of 3Running the service as root would have made the report submit immediately. What is wrong with shipping that?

• It would have solved a group membership problem by discarding the security change that exposed it
• It would have been slower to apply than the group fix
• It would not actually have worked
• Loosening the file to be world readable would have been the better shortcut

3 questions, none answered yet.

Why this one exists

"The file exists" and "this process can read it" are two separate claims, and only the second one matters. Identity is not just a user: supplementary group membership is part of it, and it can be dropped by a change that appears to set the identity to exactly what it already was.

In an interview

Permission problems are where people reach for root fastest, and reaching for root is the answer that ends the conversation. Being able to say which account the process runs as, which account the file belongs to, and which single group is missing between them is a different level of answer, and it is the one that keeps the hardening in place.

Commands introduced

• id
• ls -l
• stat

Evidence layers

• the error the process returned
• the file's owner, group and mode
• the identity the process actually has