resilient-document-pipeline

Resilient Document Generation Pipeline

When to Use This Skill

Use this workflow when generating documents in challenging environments where:

• Tool failures are likely or have occurred: read_file, search_web, read_webpage, or execute_code_sandbox return errors
• Multiple formats are needed: Generate .docx, .pdf, .html from the same source
• Unicode/special characters are present: Documents contain em dashes, curly quotes, arrows, symbols, or non-ASCII text
• Primary conversion methods may fail: LaTeX/pandoc may not be available or misconfigured

This skill combines early failure detection, domain-knowledge content creation, progressive conversion fallbacks, and Unicode safety into a single resilient workflow.

Core Workflow Overview

┌─────────────────────────────────────────────────────────────────┐
│  1. DETECT tool failures (2+ indicators → pivot immediately)   │
└────────────────┬────────────────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────────────────┐
│  2. CREATE content with write_file using embedded domain       │
│     knowledge (professionally structured markdown)              │
└────────────────┬────────────────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────────────────┐
│  3. SANITIZE unicode (create LaTeX-safe version for PDF)       │
└────────────────┬────────────────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────────────────┐
│  4. CONVERT with progressive fallbacks:                        │
│     pandoc → reportlab → fpdf2 → wkhtmltopdf                   │
└────────────────┬────────────────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────────────────┐
│  5. VERIFY outputs exist and are valid                          │
└─────────────────────────────────────────────────────────────────┘

Step-by-Step Workflow

Step 1: Detect Tool Failure Pattern

Before attempting document generation, check for failure indicators:

TOOL_FAILURE_INDICATORS = [
    "read_file returns binary/image data instead of text",
    "search_web returns unknown error or empty results",
    "read_webpage returns unknown error on multiple URLs",
    "execute_code_sandbox fails unexpectedly",
    "pandoc command fails with 'unknown error'",
    "Multiple consecutive tool failures on data retrieval"
]

Decision point: If 2+ indicators are present within the first 2-3 iterations:

1. Stop attempting to fix the failing tools
2. Acknowledge the limitation briefly in your output
3. Commit to generating the document with available knowledge
4. Proceed to Step 2 immediately

Step 2: Create Content with write_file

Write your document as professionally structured Markdown using embedded domain knowledge:

write_file
path: /tmp/document_source.md
content: |
  # [Document Title]
  
  ## Executive Summary
  [Brief overview of key content - use domain knowledge]
  
  ## Background
  [Context and scope - leverage embedded expertise]
  
  ## Main Content
  [Organized sections with headers, lists, tables]
  
  ## Limitations & Notes
  [Transparent about data source limitations if relevant]
  
  ## Recommendations/Next Steps
  [Actionable guidance based on available information]

Best practices for content:

• Use general domain knowledge appropriately when external data is unavailable
• Clearly distinguish between verified facts and general guidance
• Include actionable frameworks rather than specific unverified data
• Add placeholder notes where specific data would enhance the document:

> **Note**: Specific [metric/data point] would typically be sourced from 
> [expected source]. The guidance below reflects established best practices.

Step 3: Sanitize Unicode Characters (Pre-PDF Conversion)

Critical: PDF generation via pandoc typically uses LaTeX, which has limited Unicode support. Create a sanitized version:

Option A: Manual write_file (Recommended for control)

write_file
path: /tmp/document_source_sanitized.md
content: |
  [Same content as Step 2, but with these replacements:]

Character replacement table:

Character	Issue	Safe Replacement
— (em dash)	May not render	-- or -
– (en dash)	May not render	-
" " (curly quotes)	Encoding errors	" " (straight quotes)
' ' (curly apostrophe)	Encoding errors	' (straight apostrophe)
… (ellipsis)	May not render	...
→ ← ↑ ↓ (arrows)	LaTeX incompatibility	-> <- ^ v
✓ ✗ (checkmarks)	May not render	[x] [ ]
★ ● (symbols)	May not render	* -
© ® ™	May require packages	(c) (r) (tm)
Non-ASCII letters (é, ñ, ü)	Font-dependent	Use xeLaTeX or keep for non-PDF formats

Option B: Shell script sanitization

run_shell
command: sed -e 's/—/--/g' -e 's/–/-/g' -e 's/"([^"]*)"/"\1"/g' -e "s/'([^']*)/'\1'/g" -e 's/…/.../g' -e 's/→/->/g' -e 's/←/<-/g' -e 's/©/(c)/g' -e 's/®/(r)/g' -e 's/™/(tm)/g' /tmp/document_source.md > /tmp/document_source_sanitized.md

Important: Keep the original unsanitized file for DOCX/HTML conversion (these formats handle Unicode better).

Step 4: Convert to Target Formats with Progressive Fallbacks

Use run_shell with progressive fallback strategy. Try each method; if it fails, move to the next.

For DOCX (from original markdown):

run_shell
command: pandoc /tmp/document_source.md -o output.docx

Fallback if pandoc fails:

execute_code_sandbox
code: |
  from docx import Document
  # Parse markdown and create DOCX with python-docx
  # (Implementation depends on complexity needs)

For PDF (from sanitized markdown):

Attempt 1 - pandoc with default engine:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf

Attempt 2 - pandoc with xeLaTeX (better Unicode):

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=xelatex

Attempt 3 - pandoc with wkhtmltopdf:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=wkhtmltopdf

Attempt 4 - Python fpdf2 (most reliable fallback):

execute_code_sandbox
code: |
  from fpdf import FPDF
  import re
  
  # Read sanitized markdown
  with open('/tmp/document_source_sanitized.md', 'r') as f:
      content = f.read()
  
  # Simple markdown-to-PDF conversion
  pdf = FPDF()
  pdf.add_page()
  pdf.set_font('Arial', '', 12)
  
  # Process content (strip markdown, handle basic formatting)
  lines = content.split('\n')
  for line in lines:
      # Remove markdown syntax
      clean = re.sub(r'[#*_`]', '', line)
      if clean.strip():
          pdf.cell(0, 10, clean, ln=True)
  
  pdf.output('output.pdf')
  print('PDF created successfully')

Attempt 5 - Python reportlab (alternative):

execute_code_sandbox
code: |
  from reportlab.lib.pagesizes import letter
  from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer
  from reportlab.lib.styles import getSampleStyleSheet
  import re
  
  doc = SimpleDocTemplate('output.pdf', pagesize=letter)
  styles = getSampleStyleSheet()
  story = []
  
  with open('/tmp/document_source_sanitized.md', 'r') as f:
      content = f.read()
  
  lines = content.split('\n')
  for line in lines:
      clean = re.sub(r'[#*_`]', '', line)
      if clean.strip():
          story.append(Paragraph(clean, styles['Normal']))
          story.append(Spacer(1, 6))
  
  doc.build(story)
  print('PDF created with reportlab')

For HTML (from original markdown):

run_shell
command: pandoc /tmp/document_source.md -o output.html

Fallback if pandoc fails:

execute_code_sandbox
code: |
  import markdown
  with open('/tmp/document_source.md', 'r') as f:
      md_content = f.read()
  html_content = markdown.markdown(md_content)
  with open('output.html', 'w') as f:
      f.write(f'<!DOCTYPE html><html><body>{html_content}</body></html>')

Step 5: Verify Outputs

Check that files were created successfully and have reasonable sizes:

run_shell
command: ls -lh output.* 2>/dev/null || echo "Some outputs missing"

run_shell
command: file output.pdf output.docx output.html 2>/dev/null

Validation criteria:

• [ ] File exists
• [ ] File size > 0 bytes
• [ ] File command recognizes the format correctly
• [ ] (Optional) Open/preview the file to confirm content renders

Complete Example

# Generate Project Status Report

## Step 1: Detect failures (if applicable)
Tool check: read_webpage failed 3x, search_web failed 2x → PIVOT to fallback workflow

## Step 2: Write Markdown source
write_file
path: /tmp/project_status.md
content: |
  # Project Status Report
  
  ## Executive Summary
  This report summarizes project progress using established methodologies.
  
  ## Current Status
  - Phase 1: Complete
  - Phase 2: In Progress (75%)
  - Phase 3: Not Started
  
  ## Risks & Mitigation
  Standard risk framework applied...
  
  ## Recommendations
  1. Continue current trajectory
  2. Schedule stakeholder review

> **Note**: Specific metrics from project management tool were unavailable; guidance reflects standard practices.

## Step 3: Sanitize for PDF
write_file
path: /tmp/project_status_sanitized.md
content: |
  [Same content with any special chars replaced per table above]

## Step 4: Convert (try each, proceed on failure)
run_shell
command: pandoc /tmp/project_status.md -o project_status.docx

run_shell
command: pandoc /tmp/project_status_sanitized.md -o project_status.pdf

run_shell
command: pandoc /tmp/project_status.md -o project_status.html

## Step 5: Verify
run_shell
command: ls -lh project_status.*

Decision Tree for Conversion Failures

pandoc PDF fails?
  ├─→ Try --pdf-engine=xelatex
  │   └─→ Fails?
  │       ├─→ Try --pdf-engine=wkhtmltopdf
  │       │   └─→ Fails?
  │       │       └─→ Use Python fpdf2 (most reliable)
  │       └─→ Success → Done
  └─→ Success → Done

Troubleshooting

Issue	Solution
PDF generation fails with encoding error	Use sanitized markdown; try --pdf-engine=xelatex
Pandoc not found	Install via apt-get install pandoc or use Python fallbacks
LaTeX not found	Use --pdf-engine=wkhtmltopdf or Python fpdf2
DOCX formatting issues	Add --reference-doc=template.docx for custom styles
Unicode in any format	Use -f markdown+utf8; ensure UTF-8 encoding with file -i source.md
All pandoc commands fail	Fall back to Python libraries (fpdf2, reportlab, python-docx, markdown)
Python libraries missing	Install via pip install fpdf2 reportlab python-docx markdown

Advantages Over Standard Approaches

Aspect	Standard Approach	Resilient Pipeline
Tool failure response	Retry same approach	Detect pattern, pivot immediately
Content source	External data only	Embedded domain knowledge when needed
Conversion robustness	Single method	Progressive fallbacks (5+ options)
Unicode handling	May cause failures	Proactive sanitization
Verification	Optional	Built-in validation step
Transparency	May hide limitations	Explicit about constraints

Best Practices

Do	Don't
Pivot after 2+ tool failures	Retry failing tools 5+ times
Sanitize before PDF conversion	Send raw Unicode to LaTeX
Try multiple conversion methods	Give up after first failure
Be transparent about limitations	Claim unverified facts as certain
Verify all outputs	Assume files were created
Use progressive fallbacks	Hard-code single approach

When to Return to Standard Methods

After successfully completing the resilient pipeline:

1. For similar future tasks: You can attempt standard shell_agent delegation first
2. Keep this skill ready: Use immediately if tool failures recur
3. For Unicode-heavy documents: Consider always using this workflow with sanitization
4. For critical deliverables: This workflow provides maximum reliability

Related Skills

• document-gen-fallback-enhanced: Original Unicode-safe fallback (parent)
• write-file-fallback-report: Domain-knowledge report generation (parent)
• Use this unified skill when you need all capabilities combined in one workflow *** End Files *** Add File: sanitize_for_pdf.sh #!/bin/bash

sanitize_for_pdf.sh - Replace problematic unicode chars for LaTeX/PDF

Usage: ./sanitize_for_pdf.sh <input.md> [output.md]

if [ -z "$1" ]; then echo "Usage: $0 <input.md> [output.md]" exit 1 fi

INPUT="$1" OUTPUT="${2:-${1%.md}_sanitized.md}"

Check input file exists

if [ ! -f "$INPUT" ]; then echo "Error: Input file not found: $INPUT" exit 1 fi

Apply sanitization replacements

sed -e 's/—/--/g'
-e 's/–/-/g'
-e 's/"([^"])"/"\1"/g'
-e "s/'([^'])/'\1'/g"
-e 's/…/.../g'
-e 's/→/->/g'
-e 's/←/<-/g'
-e 's/↑/^^/g'
-e 's/↓/vv/g'
-e 's/✓/[x]/g'
-e 's/✗/[ ]/g'
-e 's/★/*/g'
-e 's/●/-/g'
-e 's/©/(c)/g'
-e 's/®/(r)/g'
-e 's/™/(tm)/g'
"$INPUT" > "$OUTPUT"

echo "Sanitized: $INPUT -> $OUTPUT" echo "Verify with: diff $INPUT $OUTPUT" *** Add File: convert_with_fallbacks.sh #!/bin/bash

convert_with_fallbacks.sh - Progressive fallback document conversion

Usage: ./convert_with_fallbacks.sh <input.md> <output_format>

Formats: pdf, docx, html

if [ -z "$1" ] || [ -z "$2" ]; then echo "Usage: $0 <input.md> <pdf|docx|html>" exit 1 fi

INPUT="$1" FORMAT="$2" BASENAME="${INPUT%.*}" OUTPUT="${BASENAME}.${FORMAT}"

echo "Converting $INPUT to $FORMAT..."

if [ "$FORMAT" = "pdf" ]; then echo "Attempt 1: pandoc default" if pandoc "$INPUT" -o "$OUTPUT" 2>/dev/null; then echo "Success with pandoc default" exit 0 fi

echo "Attempt 2: pandoc with xeLaTeX" if pandoc "$INPUT" -o "$OUTPUT" --pdf-engine=xelatex 2>/dev/null; then echo "Success with xeLaTeX" exit 0 fi

echo "Attempt 3: pandoc with wkhtmltopdf" if pandoc "$INPUT" -o "$OUTPUT" --pdf-engine=wkhtmltopdf 2>/dev/null; then echo "Success with wkhtmltopdf" exit 0 fi

echo "Attempt 4: Python fpdf2" if python3 -c " from fpdf import FPDF import re pdf = FPDF() pdf.add_page() pdf.set_font('Arial', '', 12) with open('$INPUT', 'r') as f: for line in f: clean = re.sub(r'[#*_`]', '', line) if clean.strip(): pdf.cell(0, 10, clean.strip(), ln=True) pdf.output('$OUTPUT') " 2>/dev/null; then echo "Success with fpdf2" exit 0 fi

echo "All PDF conversion methods failed" exit 1

elif [ "$FORMAT" = "docx" ]; then echo "Attempt 1: pandoc" if pandoc "$INPUT" -o "$OUTPUT" 2>/dev/null; then echo "Success with pandoc" exit 0 fi

echo "Pandoc failed for DOCX - consider Python fallback (python-docx)" exit 1

elif [ "$FORMAT" = "html" ]; then echo "Attempt 1: pandoc" if pandoc "$INPUT" -o "$OUTPUT" 2>/dev/null; then echo "Success with pandoc" exit 0 fi

echo "Attempt 2: Python markdown" if python3 -c " import markdown with open('$INPUT', 'r') as f: content = f.read() html = markdown.markdown(content) with open('$OUTPUT', 'w') as f: f.write(f'<!DOCTYPE html><html><body>{html}</body></html>') " 2>/dev/null; then echo "Success with markdown library" exit 0 fi

echo "All HTML conversion methods failed" exit 1

else echo "Unknown format: $FORMAT (use pdf, docx, or html)" exit 1 fi