Phase 3: mdBook Build

Loading…

Phase 3: mdBook Build

Relevant source files

• scripts/build-docs.sh

Purpose and Scope

Phase 3 is the final transformation stage that converts enhanced markdown files into a searchable HTML documentation website using mdBook. This phase creates the book structure, generates the table of contents, injects templates, installs mermaid rendering support, and executes the mdBook build process.

This page covers the overall Phase 3 workflow and its core components. For detailed information about specific sub-processes, see:

• SUMMARY.md generation algorithm: 8.1
• Template injection mechanics: 8.2
• Configuration system: 3
• Template system details: 11

Phase 3 begins after Phase 2 completes diagram enhancement (see 7) and ends with the production of deployable HTML artifacts.

Sources: scripts/build-docs.sh:95-310

Phase 3 Process Flow

Phase 3 executes six distinct steps, each transforming the workspace toward the final HTML output. The process is orchestrated by build-docs.sh and coordinates multiple tools.

graph TB
    Input["Enhanced Markdown Files\n/workspace/wiki/"]
Step2["Step 2: Initialize mdBook Structure\nCreate /workspace/book/\nGenerate book.toml"]
Step3["Step 3: Generate SUMMARY.md\nDiscover file structure\nSort pages numerically"]
Step4["Step 4: Process Markdown Files\nInject header/footer templates\nCopy to book/src/"]
Step5["Step 5: Install Mermaid Assets\nmdbook-mermaid install"]
Step6["Step 6: Build Book\nmdbook build"]
Step7["Step 7: Copy Outputs\nbook/ → /output/book/\nbook.toml → /output/"]
Output["Final Outputs\n/output/book/ (HTML)\n/output/markdown/\n/output/book.toml"]
Input --> Step2
 
   Step2 --> Step3
 
   Step3 --> Step4
 
   Step4 --> Step5
 
   Step5 --> Step6
 
   Step6 --> Step7
 
   Step7 --> Output

High-Level Phase 3 Pipeline

Sources: scripts/build-docs.sh:95-310

mdBook Structure Initialization

The first step creates the mdBook workspace at /workspace/book/ and generates the configuration file book.toml. This establishes the foundation for all subsequent operations.

Directory Structure Creation

The script creates the following directory hierarchy:

/workspace/book/
├── book.toml          (generated configuration)
└── src/               (will contain markdown files)
    ├── SUMMARY.md     (generated in Step 3)
    ├── *.md           (copied in Step 4)
    └── section-*/     (copied in Step 4)

Sources: scripts/build-docs.sh:96-122

book.toml Configuration

The book.toml file is generated dynamically from environment variables. The configuration structure follows the mdBook specification:

The generated configuration:

The git-repository-url setting enables the GitHub icon in the rendered book’s header, providing navigation back to the source repository.

Sources: scripts/build-docs.sh:101-119

SUMMARY.md Generation Process

Step 3 generates src/SUMMARY.md, which defines the book’s table of contents and navigation structure. This is a critical file that mdBook requires to determine page ordering and hierarchy.

graph TB
    WikiDir["/workspace/wiki/\nAll markdown files"]
FindOverview["Find Overview File\ngrep -Ev '^[0-9]'\nFirst non-numbered file"]
FindMain["Find Main Pages\ngrep -E '^[0-9]'\nFiles matching ^[0-9]*.md"]
Sort["Sort Numerically\nsort -t- -k1 -n\nBy leading number"]
CheckSections{"Has Subsections?\nsection-N directory exists?"}
FindSubs["Find Subsections\nls section-N/*.md\nSort numerically"]
ExtractTitle["Extract Title\nhead -1 file.md\nsed 's/^# //'"]
BuildEntry["Build TOC Entry\n- [Title](filename)"]
BuildNested["Build Nested Entry\n- [Title](filename)\n - [Subtitle](section-N/file)"]
Summary["src/SUMMARY.md\nGenerated TOC"]
WikiDir --> FindOverview
 
   WikiDir --> FindMain
 
   FindOverview --> Summary
 
   FindMain --> Sort
 
   Sort --> CheckSections
    
 
   CheckSections -->|No| ExtractTitle
 
   CheckSections -->|Yes| FindSubs
    
 
   ExtractTitle --> BuildEntry
 
   FindSubs --> ExtractTitle
 
   ExtractTitle --> BuildNested
    
 
   BuildEntry --> Summary
 
   BuildNested --> Summary

File Discovery and Sorting

Numeric Sorting Algorithm

The sorting mechanism uses shell built-ins to extract numeric prefixes and sort appropriately:

1. List all .md files in /workspace/wiki/: ls "$WIKI_DIR"/*.md
2. Filter by numeric prefix: grep -E '^[0-9]'
3. Sort using field delimiter - and numeric comparison: sort -t- -k1 -n
4. For each main page, check for subsection directory: section-$section_num
5. If subsections exist, repeat sort for subsection files

This ensures pages appear in the correct order: 1-overview.md, 2-architecture.md, 2.1-subsection.md, etc.

Sources: scripts/build-docs.sh:124-188

For detailed documentation of the SUMMARY.md generation algorithm, see 8.1.

Markdown File Processing

Step 4 copies markdown files from /workspace/wiki/ to /workspace/book/src/ and injects HTML templates into each file. This step bridges the gap between raw markdown and mdBook-ready content.

graph TB
    HeaderTemplate["templates/header.html\nRaw template with variables"]
FooterTemplate["templates/footer.html\nRaw template with variables"]
ProcessTemplate["process-template.py\nVariable substitution"]
EnvVars["Environment Variables\nREPO, BOOK_TITLE, DEEPWIKI_URL\nGITHUB_BADGE_URL, etc."]
HeaderHTML["HEADER_HTML\nProcessed HTML string"]
FooterHTML["FOOTER_HTML\nProcessed HTML string"]
MarkdownFiles["Markdown Files\nsrc/*.md, src/*/*.md"]
Inject["Injection Loop\nfor mdfile in src/*.md"]
TempFile["$mdfile.tmp\nHeader + Content + Footer"]
Replace["mv $mdfile.tmp $mdfile\nReplace original"]
FinalFiles["Final Book Source\nsrc/*.md with templates"]
HeaderTemplate --> ProcessTemplate
 
   FooterTemplate --> ProcessTemplate
 
   EnvVars --> ProcessTemplate
    
 
   ProcessTemplate --> HeaderHTML
 
   ProcessTemplate --> FooterHTML
    
 
   HeaderHTML --> Inject
 
   FooterHTML --> Inject
 
   MarkdownFiles --> Inject
    
 
   Inject --> TempFile
 
   TempFile --> Replace
 
   Replace --> FinalFiles

Template Processing Workflow

Template Variable Substitution

The process-template.py script is invoked twice, once for the header and once for the footer:

The processed HTML strings are then injected into every markdown file through shell redirection.

Sources: scripts/build-docs.sh:190-261

For detailed documentation of template mechanics and customization, see 8.2 and 11.

Mermaid Asset Installation

Step 5 installs the mdbook-mermaid preprocessor assets into the book directory. This step configures the JavaScript libraries and stylesheets required to render Mermaid diagrams in the final HTML output.

Installation Command

The installation is performed by the mdbook-mermaid binary:

This command:

1. Creates theme assets in book/theme/
2. Installs mermaid-init.js for diagram initialization
3. Configures mermaid.js library version
4. Sets up diagram rendering hooks for mdBook’s preprocessor chain

The preprocessor was configured in book.toml during Step 2:

This configuration tells mdBook to run the mermaid preprocessor before generating HTML, which converts mermaid code blocks into rendered diagrams.

Sources: scripts/build-docs.sh:263-266

graph LR
    BookToml["book.toml\nConfiguration"]
SrcDir["src/\nSUMMARY.md\n*.md files\nsection-*/ dirs"]
MdBookBuild["mdbook build\nMain build command"]
Preprocessor["mermaid preprocessor\nConvert mermaid blocks"]
Renderer["HTML renderer\nGenerate pages"]
Assets["Copy static assets\ntheme/, images/"]
Search["Build search index\nsearchindex.js"]
BookOutput["book/\nCompiled HTML"]
BookToml --> MdBookBuild
 
   SrcDir --> MdBookBuild
    
 
   MdBookBuild --> Preprocessor
 
   Preprocessor --> Renderer
 
   Renderer --> Assets
 
   Renderer --> Search
    
 
   Assets --> BookOutput
 
   Search --> BookOutput

Book Build Execution

Step 6 executes the core mdBook build process. This step transforms the prepared markdown files and configuration into a complete HTML documentation website.

mdBook Build Pipeline

Build Command

The build is invoked with no arguments, using the current directory’s configuration:

mdBook automatically:

• Reads book.toml for configuration
• Processes src/SUMMARY.md to determine page structure
• Runs configured preprocessors (mermaid)
• Generates HTML with search index
• Applies the configured theme (rust)
• Creates navigation elements with git repository link

The resulting output is written to book/, relative to the current working directory (/workspace/book/).

Sources: scripts/build-docs.sh:268-271

graph TB
    BookBuild["book/\nBuilt HTML website"]
WikiDir["wiki/\nEnhanced markdown"]
RawDir["raw_markdown/\nPre-enhancement snapshots"]
BookConfig["book.toml\nBuild configuration"]
OutputBook["/output/book/\nDeployable HTML"]
OutputMD["/output/markdown/\nFinal markdown source"]
OutputRaw["/output/raw_markdown/\nDebug snapshots"]
OutputConfig["/output/book.toml\nReference config"]
BookBuild -->|cp -r| OutputBook
 
   WikiDir -->|cp -r| OutputMD
 
   RawDir -->|cp -r if exists| OutputRaw
 
   BookConfig -->|cp| OutputConfig

Output Collection

Step 7 consolidates all build artifacts into the /output/ directory, which is typically mounted as a volume for access from the host system.

Output Artifacts and Layout

Copy Operations

The script performs four copy operations:

The /output/book/ directory is immediately servable as a static website:

Sources: scripts/build-docs.sh:273-309

graph TB
    Phase1["Phase 1:\nMarkdown Extraction"]
Phase2["Phase 2:\nDiagram Enhancement"]
CheckMode{"MARKDOWN_ONLY\n== 'true'?"}
CopyMD["Copy wiki/ to\n/output/markdown/"]
CopyRaw["Copy raw_markdown/ to\n/output/raw_markdown/"]
Exit["Exit with success\nSkip Phase 3"]
Phase3["Phase 3:\nmdBook Build"]
Phase1 --> Phase2
 
   Phase2 --> CheckMode
    
 
   CheckMode -->|Yes| CopyMD
 
   CopyMD --> CopyRaw
 
   CopyRaw --> Exit
    
 
   CheckMode -->|No| Phase3

Markdown-Only Mode

The MARKDOWN_ONLY environment variable provides an escape hatch that bypasses Phase 3 entirely. When set to "true", the script exits after Phase 2 (diagram enhancement) without building the HTML book.

Markdown-Only Workflow

This mode is useful for:

• Debugging diagram placement without HTML build overhead
• Consuming markdown in alternative build systems
• Inspecting intermediate transformation results
• CI/CD pipelines that only need markdown output

Sources: scripts/build-docs.sh:67-93

For more details on markdown-only mode, see 12.1.

graph TB
    subgraph "Shell Script Orchestration"
        BuildScript["build-docs.sh\nMain orchestrator"]
Step2Func["Lines 95-122\nmkdir, cat > book.toml"]
Step3Func["Lines 124-188\nSUMMARY.md generation"]
Step4Func["Lines 190-261\ncp, template injection"]
Step5Func["Lines 263-266\nmdbook-mermaid install"]
Step6Func["Lines 268-271\nmdbook build"]
Step7Func["Lines 273-295\ncp outputs"]
end
    
    subgraph "External Tools"
        MdBook["mdbook\nRust binary\n/usr/local/bin/mdbook"]
MdBookMermaid["mdbook-mermaid\nPreprocessor binary\n/usr/local/bin/mdbook-mermaid"]
ProcessTemplate["process-template.py\nPython script\n/usr/local/bin/process-template.py"]
end
    
    subgraph "Key Files"
        BookToml["book.toml\nGenerated at line 102"]
SummaryMd["src/SUMMARY.md\nGenerated at line 130-186"]
HeaderHtml["templates/header.html\nInput template"]
FooterHtml["templates/footer.html\nInput template"]
end
    
    subgraph "Directory Structures"
        BookDir["/workspace/book/\nBuild workspace"]
SrcDir["/workspace/book/src/\nMarkdown sources"]
OutputDir["/output/\nVolume mount point"]
end
    
 
   BuildScript --> Step2Func
 
   Step2Func --> Step3Func
 
   Step3Func --> Step4Func
 
   Step4Func --> Step5Func
 
   Step5Func --> Step6Func
 
   Step6Func --> Step7Func
    
 
   Step2Func --> BookToml
 
   Step2Func --> BookDir
 
   Step3Func --> SummaryMd
 
   Step4Func --> ProcessTemplate
 
   Step4Func --> SrcDir
 
   Step5Func --> MdBookMermaid
 
   Step6Func --> MdBook
 
   Step7Func --> OutputDir
    
 
   HeaderHtml --> ProcessTemplate
 
   FooterHtml --> ProcessTemplate

Phase 3 Component Map

This diagram maps Phase 3 concepts to their concrete implementations in the codebase:

Sources: scripts/build-docs.sh:95-310

Phase 3 Error Handling

Phase 3 operates under set -e mode, causing immediate script termination on any command failure. Common failure scenarios:

All errors are reported to stderr and result in a non-zero exit code due to set -e.

Sources: scripts/build-docs.sh2 scripts/build-docs.sh:95-310

Summary

Phase 3 transforms enhanced markdown into a searchable HTML documentation website through six orchestrated steps:

1. Structure Initialization : Creates mdBook workspace and book.toml configuration
2. SUMMARY.md Generation : Builds navigation structure with numeric sorting
3. Markdown Processing : Injects templates and copies files to book source
4. Mermaid Installation : Configures diagram rendering support
5. Book Build : Executes mdBook to generate HTML
6. Output Collection : Consolidates artifacts to /output/ volume

Phase 3 is entirely orchestrated by build-docs.sh and coordinates shell utilities, Python scripts, and Rust binaries to produce the final documentation website. The process is deterministic and idempotent, generating consistent output from the same input markdown.

Sources: scripts/build-docs.sh:95-310

Dismiss

Refresh this wiki

Enter email to refresh