large-file-write-mkdir

Large File Write Fallback (with Directory Pre-creation)

Problem

When writing large files (typically several KB or more), two common tools may fail with unknown error due to internal payload size limits:

• write_file — has a maximum content size it can handle in a single call.
• shell_agent — may also hit payload limits when the task description includes large inline content.

A second class of failure occurs when the target directory does not exist: Python's open() will raise FileNotFoundError if any intermediate directory is missing. Always pre-create the parent directory to prevent this.

Solution

Use run_shell with a mkdir -p prefix and a Python heredoc pattern. The mkdir -p ensures the target directory exists; the heredoc streams the file content through stdin directly into Python's open(), bypassing the payload constraints of the other tools.

Template

mkdir -p "<PARENT DIRECTORY OF TARGET PATH>" && python3 - << 'EOF'
content = """<FILE CONTENTS HERE>"""
with open("<TARGET PATH>", "w") as f:
    f.write(content)
EOF

Pass this as the command parameter to run_shell.

Tip: If the parent directory is not known in advance, use the shell dirname helper: mkdir -p "$(dirname '<TARGET PATH>')" && python3 - << 'EOF'

Step-by-Step Instructions

1. Attempt the normal write using write_file first. If it succeeds, you are done.

2. If write_file fails (especially with unknown error or a timeout on large content), do NOT retry with shell_agent using inline content — it will likely fail for the same reason.

3. Use the run_shell heredoc fallback:

   • Pre-create the parent directory with mkdir -p <parent_dir>. This is a no-op when the directory already exists, so it is always safe to include. Skipping this step causes Python's open() to raise FileNotFoundError for any new or deeply-nested path.
   • Embed the full file content inside a Python triple-quoted string.
   • Specify the target path inside the open() call.
   • Pass the entire block (mkdir + heredoc) as the command to run_shell.

4. Escape carefully inside the heredoc:

   • Backslashes that should be literal in the file must be doubled (\\).
   • Triple-quotes inside the content must be escaped (\"\"\").
   • The heredoc delimiter EOF must not appear on a line by itself inside the content (rename it to PYEOF or FILEEOF if needed).

5. Verify the write by following up with a run_shell call such as:

   wc -l <TARGET PATH> && head -5 <TARGET PATH>
   

Full Example

Suppose you need to write a large TypeScript file to src/components/Dashboard.ts:

mkdir -p "src/components" && python3 - << 'PYEOF'
content = """import { foo } from './foo';

export interface DashboardData {
  title: string;
  items: string[];
}

export function createDashboard(data: DashboardData): string {
  return `<div>${data.title}</div>`;
}
"""
with open("src/components/Dashboard.ts", "w") as f:
    f.write(content)
PYEOF

Pass the above (without the surrounding code fence) as the command argument to run_shell.

When to Use This Pattern

Situation	Recommended tool
Small file (< ~2 KB)	write_file
Medium file, no errors yet	write_file (try first)
Large file or write_file failed	run_shell + mkdir -p + Python heredoc
shell_agent also fails on large inline content	run_shell + mkdir -p + Python heredoc
Target directory may not exist	Always add mkdir -p <parent_dir> first

Notes

• This pattern works for any text-based file (TypeScript, Python, JSON, YAML, Markdown, etc.).
• For binary files, adapt the approach to use base64 decoding inside the Python script.
• The heredoc delimiter (EOF, PYEOF, FILEEOF) can be any string not present as a standalone line in your content — choose accordingly.
• This technique is also useful when content contains characters that would need heavy shell escaping in a plain echo or printf approach.
• The mkdir -p command is idempotent: it succeeds whether or not the directory already exists, so it is always safe to include.
• To derive the parent directory from a target path in shell: mkdir -p "$(dirname '<TARGET PATH>')" works as a generic alternative when the parent path is not already known.