The disk has plenty of space and the writes still fail

Can I prove which resource actually ran out?

The ticket

CUSTOMER TICKET: nightly export failing, but you say the disk is fine

Account: Ardent Logistics (enterprise) Impact: no exports delivered for three nights Started: since the tuning work last week

Our export has failed three nights running. The message in your job output says there is no space left, so we asked our own team to check and they say the volume is almost empty, under one percent used. We have looked twice.

Someone suggested we buy more storage, which we are happy to do, but we would rather not spend the money on something that is apparently already empty. Please tell us what is actually full.

Your job

1. Prove which resource ran out. The error names the failure, not the cause.
2. Do not resize anything until you can say what would fill up again.
3. Restore the export with the smallest correct change, then confirm it can run again tomorrow rather than only once.

Working notes

The job runs in the worker service. You can run it on demand:

docker compose -f labs/linux/_stack/compose.yaml \
               -f labs/linux/_stack/compose.override.yaml \
               exec worker python3 /app/exporter.py

Its configuration is in labs/linux/_stack/compose.override.yaml. After editing, run tse apply, then tse check.

Track

Linux and CLI

Time

about 30 minutes

Difficulty

Straightforward

Tier

Core

Start it

In a Codespace or a local clone:

tse start linux/01-disk-has-space-and-writes-still-fail

That provisions the broken system and prints the ticket above. Investigate with ordinary tools, then run tse check.

Look at the evidence

Real output, captured by running these commands against the broken system and checked against it on every build. It shows you what the evidence looks like. It cannot fix anything, and it will not tell you what is wrong.

Type a command you would reach for, or help.

CommandRun

Enter runs it. Shift and Enter start a new line. The up and down arrows walk back through what you have typed.

Show which commands are recorded (this is close to a hint)

Investigation scratchpad

Saved in this browser as you type. Nothing is uploaded. 0 of 7 filled in.

1. Customer symptom

In their words, not yours. Include scope and urgency.

2. Prediction

Before running anything: target layer, expected output, two likely causes.

3. Evidence gathered

The command or query, and why it is safe to run here.

4. Confirmed / disproven / still unknown

Three separate lists. This is the step people skip.

5. What it proves

One proof sentence, one safe next step, one alternate hypothesis.

6. Customer update and escalation note

Plain language. Impact first. No blame, no speculation.

7. Recall

One gap, one command to repeat tomorrow, one confidence score.

Copy as markdownClear

Hints

Each hint gives away a little more. Try to spend a few minutes on your own evidence first, because the recall is what makes it stick.

Hint 1 of 3

Take the customer at their word, because they are right. The volume really is almost empty, and the write really did fail saying there was no room.

People usually resolve that by deciding somebody misread something. Nobody misread anything. Both readings are correct at the same moment, and the reason is that they are about different things.

The question worth asking is not "who is wrong". It is:

Which resource ran out, and does the error actually say?

Read the failure again and notice how little it commits to. It tells you the write failed and it tells you the kernel's reason code. It does not tell you what the device ran out of, and that omission is the whole exercise.

Before you look at any setting, run the job yourself and get the exact error rather than the customer's summary of it. Then find out how many things a filesystem can run out of. It is more than one, and only one of them is the number everybody checks.

Hint 2 of 3

A filesystem allocates two things separately, and running out of either one produces the same error:

Resource	What it holds	What reports it
Blocks	the contents of files	df -h
Inodes	the files themselves, one each	df -i

The customer checked the first. Nobody has checked the second.

That is why the error is so unhelpful: the kernel returns ENOSPC for both, and "No space left on device" is its wording for either. The message names the failure, not the cause.

Run both against the spool directory and compare:

docker compose -f labs/linux/_stack/compose.yaml \
               -f labs/linux/_stack/compose.override.yaml \
               exec worker sh -c 'df -h /var/spool/exports; df -i /var/spool/exports'

One of those is going to say almost nothing is used. The other is going to say it is completely full.

Then ask the question that follows: what is creating that many files. Count them, and compare the count against what the job was asked to produce. The error message itself carries both numbers if you read past the first line.

Hint 3 of 3

Space is at one percent. Inodes are at one hundred:

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           64.0M    0  64.0M   0% /var/spool/exports

Filesystem     Inodes  Used Avail Use% Mounted on
tmpfs             512   512      0 100% /var/spool/exports

Every file consumes one inode no matter how small it is, and these are tiny. The job ran out of files it was allowed to create long before it came close to running out of room to put them in.

The failure itself tells you the scale:

{"files_written": 511, "files_expected": 5000}

Five thousand files for five thousand rows. Compare that against the setting in labs/linux/_stack/compose.override.yaml:

EXPORT_ROWS_PER_FILE: "1"

The tuning work last week set the number of rows in each file to one. The export is not too big. It is split into five thousand pieces.

Put it back to a sane chunk size and the same 5000 rows land in a handful of files. Then tse apply and tse check.

Resist resizing anything. More inodes would let tonight's run finish and would leave the real defect in place, which is a job that creates one file per row forever.

Solution

Write your customer update before you read this. Comparing your wording against the model answer is worth more than reading it cold.

Reveal the solution

Solution: the disk has space and the writes still fail

What the evidence proved

Command	What it proved	What it did not prove
Running the export	It fails with [Errno 28] No space left on device	Nothing about which resource ran out
df -h /var/spool/exports	Blocks are at 0%, the customer was right	Nothing about files, only about their contents
df -i /var/spool/exports	Inodes are at 100%, 512 of 512 used
The failure payload	files_written: 511, files_expected: 5000
EXPORT_ROWS_PER_FILE	The job was told to put one row in each file

The two df invocations are the whole diagnosis, and neither means anything without the other. Space is fine. Files are exhausted. Both are true, and only the second matches the symptom.

Worth stating plainly, because it is what makes this expensive in practice: the error message is not being unhelpful by accident. The kernel returns ENOSPC when a filesystem cannot allocate a block or an inode, so "No space left on device" is the correct wording for two entirely different problems. The message names the failure. It never names the cause.

Root cause

Last week's tuning work set EXPORT_ROWS_PER_FILE to 1.

The export writes one file per chunk, so five thousand rows became five thousand files. The spool directory has 512 inodes. The job got 511 files in and then could not create another one, with the volume still empty because each file holds twenty bytes.

The export is not too large. It is split into five thousand pieces.

Scoped fix

In labs/linux/_stack/compose.override.yaml:

services:
  worker:
    environment:
      EXPORT_ROWS_PER_FILE: "1000"

Then:

tse apply
tse check

Five files, six inodes used out of 512, and room to run again tomorrow.

Not the fix: raising the inode budget, or buying the storage the customer offered to buy. Either would let tonight's run finish and leave a job that creates one file per row in place, so the next larger export fails the same way. The check asserts the directory is not left full for exactly this reason.

Customer update

Your export was failing because it ran out of file slots rather than out of space, which is why your team was right that the volume was nearly empty. A filesystem limits how many files it can hold separately from how much data it can hold, and the error message our job reported covers both cases without distinguishing them.

The cause was a setting changed during last week's tuning work, which told the export to write one file per row. Your five thousand rows became five thousand files, and the directory allows five hundred and twelve. We have put the chunk size back, so the same data now writes as five files. Tonight's export will run normally.

Please do not buy the additional storage. It would not have helped, and the setting was the whole problem. If you would like, I can send you the two commands that tell these apart, so your team can check both next time.

Engineering escalation, if you needed one

Impact: three consecutive nightly exports failed for Ardent Logistics, no data delivered, customer was about to purchase storage that would not have helped. Evidence: [Errno 28] No space left on device after 511 of 5000 files; df -h at 0% and df -i at 100% on the same directory. Confirmed: the volume has space, the job runs, the failure is reproducible. Ruled out: disk capacity, permissions, a data volume change. Suspected cause: EXPORT_ROWS_PER_FILE was set to 1 during tuning, making the job write one file per row. Request: the exporter reports ENOSPC without saying which resource was exhausted, which cost three nights and nearly cost the customer a storage purchase. Can it check free inodes before a run and say so in the failure.

That request is the durable part. The setting is fixed for this customer. The next customer to hit this gets the same unhelpful message unless the job learns to say which of the two things it ran out of.

Check your understanding

Three questions on what the evidence here proved, and what it pointedly did not. Wrong answers explain themselves, and so do right ones.

tse quiz

Check your understanding

Three questions on what the evidence proved and what it did not. Every answer explains itself, including the right one.

Question 1 of 3The volume was at one percent used and the write failed for lack of room. Why is the error message not simply wrong?

• Because the reading was stale and the volume filled up momentarily
• Because a filesystem allocates space and file slots separately, and the kernel reports exhausting either one the same way
• Because the reported percentage excludes reserved space
• Because the write was to a different filesystem than the one they checked

Question 2 of 3The customer offered to buy more storage. Why would that have been the wrong sale to accept?

• Because their existing volume was still under warranty
• Because more space would not have created more file slots, and the job would keep failing at the same count
• Because the export could have been rescheduled to a quieter time
• Because support cannot approve customer purchases

Question 3 of 3Raising the file-slot budget would also have made tonight's export succeed. Why does the grader refuse it?

• Because changing that budget requires a maintenance window
• Because the job writing one file per row is the defect, and a larger budget only moves the failure to a bigger export
• Because the budget cannot be changed after the filesystem is created
• Because the customer had already refused to pay for it

3 questions, none answered yet.

Why this one exists

A filesystem runs out of two things independently: space for content and inodes for the files themselves. The kernel reports both as ENOSPC, so the error tells you what failed and not what ran out. `df -h` and `df -i` answer different questions and only one of them matches this symptom.

In an interview

"No space left on device" with an empty disk is a classic, and the reason it is asked is that it splits candidates cleanly. Reaching for `df -h`, seeing space, and concluding the error is wrong is the common answer. Knowing that a filesystem can exhaust two separate resources and that the message names neither is the one that gets a follow-up question.

Commands introduced

• df -h
• df -i
• ls | wc -l
• docker compose exec

Evidence layers

• the error the kernel returned
• free space
• free inodes
• what the job was configured to write