Markdown-Only Mode

Loading…

Markdown-Only Mode

Relevant source files

• README.md
• scripts/build-docs.sh

This document describes the MARKDOWN_ONLY mode, a special execution mode that terminates the build pipeline after markdown extraction, skipping all mdBook-related processing. This mode is primarily used for debugging the markdown extraction process and for workflows that require raw markdown files without HTML output.

For information about the complete three-phase pipeline, see Three-Phase Pipeline. For details on the mdBook build process that gets skipped in this mode, see Phase 3: mdBook Build.

Purpose and Scope

Markdown-only mode provides a lightweight execution path that produces only markdown files, bypassing the mdBook initialization, template injection, and HTML generation stages. This mode is controlled by the MARKDOWN_ONLY environment variable and affects the execution flow in build-docs.sh.

Key characteristics:

• Executes only Phase 1 (markdown extraction) of the pipeline
• Produces markdown/ and raw_markdown/ outputs
• Skips book/ HTML generation and book.toml configuration
• Reduces execution time by ~50-70% (no mdBook build overhead)
• Useful for debugging, alternative workflows, and markdown inspection

Sources: scripts/build-docs.sh:26-93 README.md37

Execution Path Comparison

The following diagram illustrates how the MARKDOWN_ONLY flag alters the execution flow in build-docs.sh:

Figure 1: Build Pipeline Execution Paths

graph TB
    Start["build-docs.sh starts"]
Config["Configuration phase\n[lines 8-59]"]
Step1["Step 1: deepwiki-scraper.py\nScrape wiki & extract markdown\n[lines 61-65]"]
Check{"MARKDOWN_ONLY == true\n[line 68]"}
subgraph "Markdown-Only Path [lines 69-92]"
        CopyMD["Step 2: Copy markdown/\nfrom WIKI_DIR to OUTPUT_DIR\n[lines 70-73]"]
CopyRaw["Step 3: Copy raw_markdown/\nfrom RAW_DIR to OUTPUT_DIR\n[lines 75-81]"]
Exit1["Exit with status 0\n[line 92]"]
end
    
    subgraph "Standard Path [lines 95-309]"
        Step2["Step 2: Initialize mdBook structure\nCreate book.toml, src/ directory\n[lines 96-122]"]
Step3["Step 3: Generate SUMMARY.md\nDiscover files, build TOC\n[lines 124-188]"]
Step4["Step 4: Process templates\nInject header/footer HTML\n[lines 190-261]"]
Step5["Step 5: Install mdbook-mermaid\n[lines 263-266]"]
Step6["Step 6: mdbook build\n[lines 268-271]"]
Step7["Step 7: Copy all outputs\n[lines 273-294]"]
Exit2["Exit with status 0"]
end
    
 
   Start --> Config
 
   Config --> Step1
 
   Step1 --> Check
 
   Check -->|true| CopyMD
 
   CopyMD --> CopyRaw
 
   CopyRaw --> Exit1
    
 
   Check -->|false default| Step2
 
   Step2 --> Step3
 
   Step3 --> Step4
 
   Step4 --> Step5
 
   Step5 --> Step6
 
   Step6 --> Step7
 
   Step7 --> Exit2

Sources: scripts/build-docs.sh:1-310

Configuration

The MARKDOWN_ONLY mode is enabled by setting the environment variable to the string "true". Any other value (including unset) results in standard mode execution.

Environment variable:

• Name: MARKDOWN_ONLY
• Type: String boolean
• Default: "false"
• Valid values: "true" (markdown-only) or any other value (standard mode)

Example usage:

Sources: scripts/build-docs.sh26 README.md37

Output Structure

The output directory structure differs significantly between standard and markdown-only modes:

Table 1: Output Comparison

In markdown-only mode, the output/markdown/ directory contains:

• All scraped wiki pages as .md files with numeric prefixes (e.g., 1-Overview.md)
• Section subdirectories (e.g., section-4/, section-5/)
• Enhanced content with Mermaid diagrams inserted via fuzzy matching
• No template injection (no header/footer HTML)

The output/raw_markdown/ directory contains:

• Pre-enhancement markdown files (before diagram injection)
• Useful for comparing before/after states
• Only present if RAW_DIR exists (created by deepwiki-scraper.py)

Sources: scripts/build-docs.sh:70-92 scripts/build-docs.sh:287-294

Technical Implementation

The markdown-only mode implementation is located in the orchestration script and uses a simple conditional to bypass most processing steps.

Figure 2: Implementation Logic in build-docs.sh

graph TD
    ConfigVar["MARKDOWN_ONLY variable\nline 26: default='false'"]
Step1Complete["deepwiki-scraper.py complete\nWIKI_DIR and RAW_DIR populated"]
Conditional["if [ MARKDOWN_ONLY = true ]\nline 68"]
subgraph "Early Exit Block [lines 69-92]"
        EchoStep2["echo 'Step 2: Copying markdown...'\nline 70"]
RemoveMD["rm -rf OUTPUT_DIR/markdown\nline 71"]
MkdirMD["mkdir -p OUTPUT_DIR/markdown\nline 72"]
CopyMD["cp -r WIKI_DIR/. OUTPUT_DIR/markdown/\nline 73"]
CheckRaw{"if [ -d RAW_DIR ]\nline 75"}
EchoStep3["echo 'Step 3: Copying raw...'\nline 77"]
RemoveRaw["rm -rf OUTPUT_DIR/raw_markdown\nline 78"]
MkdirRaw["mkdir -p OUTPUT_DIR/raw_markdown\nline 79"]
CopyRaw["cp -r RAW_DIR/. OUTPUT_DIR/raw_markdown/\nline 80"]
EchoComplete["echo 'Markdown extraction complete!'\nlines 84-91"]
ExitZero["exit 0\nline 92"]
end
    
    ContinueStandard["Continue to Step 2:\nInitialize mdBook structure\nline 96+"]
ConfigVar --> Step1Complete
 
   Step1Complete --> Conditional
 
   Conditional -->|= true| EchoStep2
 
   EchoStep2 --> RemoveMD
 
   RemoveMD --> MkdirMD
 
   MkdirMD --> CopyMD
 
   CopyMD --> CheckRaw
 
   CheckRaw -->|RAW_DIR exists| EchoStep3
 
   CheckRaw -->|RAW_DIR missing| EchoComplete
 
   EchoStep3 --> RemoveRaw
 
   RemoveRaw --> MkdirRaw
 
   MkdirRaw --> CopyRaw
 
   CopyRaw --> EchoComplete
 
   EchoComplete --> ExitZero
    
 
   Conditional -->|!= true| ContinueStandard

Key implementation details:

1. String comparison : The check uses shell string equality [ "$MARKDOWN_ONLY" = "true" ], not numeric comparison
2. Defensive copying : Uses rm -rf before mkdir -p to ensure clean output directories
3. Dot notation : cp -r "$WIKI_DIR"/. "$OUTPUT_DIR/markdown/" copies contents, not the directory itself
4. Conditional raw copy : Only copies raw_markdown/ if the directory exists (some runs may not generate it)
5. Early exit : Uses exit 0 to terminate successfully, preventing any subsequent processing

Sources: scripts/build-docs.sh:68-92

Use Cases

Markdown-only mode supports several workflows beyond standard documentation generation:

Table 2: Common Use Cases

Sources: scripts/build-docs.sh26 (comment), README.md37

Skipped Processing Steps

When MARKDOWN_ONLY=true, the following operations are completely bypassed:

Figure 3: Skipped Components in Markdown-Only Mode

graph TB
    subgraph "Skipped: mdBook Initialization [lines 95-122]"
        BookDir["mkdir -p BOOK_DIR/src"]
BookToml["Generate book.toml\nwith title, authors, config"]
end
    
    subgraph "Skipped: SUMMARY.md Generation [lines 124-188]"
        ScanFiles["ls WIKI_DIR/*.md\nDiscover file structure"]
SortNumeric["sort -t- -k1 -n\nNumeric sorting"]
ExtractTitles["head -1 file / sed 's/^# //'\nExtract titles"]
BuildTOC["Generate nested TOC\nwith section subdirectories"]
WriteSummary["Write src/SUMMARY.md"]
end
    
    subgraph "Skipped: Template Processing [lines 190-261]"
        LoadHeader["Load templates/header.html"]
LoadFooter["Load templates/footer.html"]
ProcessVars["process-template.py\nVariable substitution"]
InjectHTML["Inject header/footer\ninto all .md files"]
end
    
    subgraph "Skipped: mdBook Build [lines 263-271]"
        MermaidInstall["mdbook-mermaid install"]
MdBookBuild["mdbook build"]
GenerateHTML["Generate book/\nHTML output"]
end
    
    Note["None of these operations\nexecute when MARKDOWN_ONLY=true"]
BookDir -.-> Note
 
   BookToml -.-> Note
 
   ScanFiles -.-> Note
 
   SortNumeric -.-> Note
 
   ExtractTitles -.-> Note
 
   BuildTOC -.-> Note
 
   WriteSummary -.-> Note
 
   LoadHeader -.-> Note
 
   LoadFooter -.-> Note
 
   ProcessVars -.-> Note
 
   InjectHTML -.-> Note
 
   MermaidInstall -.-> Note
 
   MdBookBuild -.-> Note
 
   GenerateHTML -.-> Note

Impact of skipped steps:

• No template injection : Markdown files contain pure content without HTML header/footer wrappers
• Nobook.toml: Configuration metadata is not generated (title, authors, etc.)
• NoSUMMARY.md: Table of contents is not created (users must navigate by filename)
• No Mermaid rendering : Diagrams remain as code blocks (not rendered to SVG)
• No search index : Full-text search functionality is not available
• Faster execution : Typical time savings of 50-70% depending on content size

Sources: scripts/build-docs.sh:95-309

Console Output Differences

The console output differs between modes, providing clear feedback about the execution path:

Standard mode output:

Step 1: Scraping wiki from DeepWiki...
Step 2: Initializing mdBook structure...
Step 3: Generating SUMMARY.md from scraped content...
Step 4: Copying and processing markdown files to book...
Step 5: Installing mdbook-mermaid assets...
Step 6: Building mdBook...
Step 7: Copying outputs to /output...
✓ Documentation build complete!

Markdown-only mode output:

Step 1: Scraping wiki from DeepWiki...
Step 2: Copying markdown files to output (markdown-only mode)...
Step 3: Copying raw markdown snapshots...
✓ Markdown extraction complete!

The markdown-only output explicitly mentions “(markdown-only mode)” in Step 2 and terminates after Step 3, providing clear confirmation of the execution path.

Sources: scripts/build-docs.sh:69-91 scripts/build-docs.sh:296-309

Integration with CI/CD

Markdown-only mode can be used in GitHub Actions workflows for testing and validation without full HTML builds:

Example: Markdown quality check workflow

This workflow extracts markdown, then validates it with linters and Mermaid syntax checkers without the overhead of building HTML documentation.

Sources: README.md37 scripts/build-docs.sh:26-92

graph LR
    subgraph "Iteration Loop"
        Enable["Set MARKDOWN_ONLY=true"]
Run["docker run deepwiki-to-mdbook"]
Examine["Examine output/markdown/\nand output/raw_markdown/"]
Issue{"Found issue?"}
ModifyCode["Modify deepwiki-scraper.py\nor diagram processing"]
Rebuild["docker build -t deepwiki-to-mdbook ."]
end
    
    Complete["Issue resolved"]
FullBuild["Remove MARKDOWN_ONLY\nRun full build\nVerify HTML output"]
Enable --> Run
 
   Run --> Examine
 
   Examine --> Issue
 
   Issue -->|Yes| ModifyCode
 
   ModifyCode --> Rebuild
 
   Rebuild --> Run
 
   Issue -->|No| Complete
 
   Complete --> FullBuild

Debugging Workflow

A typical debugging workflow using markdown-only mode:

Figure 4: Iterative Debugging with MARKDOWN_ONLY

Benefits of this approach:

• Fast feedback : Markdown extraction completes in seconds vs. minutes for full builds
• Isolated testing : Changes to scraper logic can be verified without mdBook complications
• Raw comparison : raw_markdown/ shows pre-enhancement state for before/after analysis
• Incremental development : Iterate rapidly on extraction logic before testing full pipeline

Sources: scripts/build-docs.sh:68-92

Limitations and Considerations

Markdown-only mode has specific limitations that users should be aware of:

Table 3: Limitations

When NOT to use markdown-only mode:

• Final production builds destined for GitHub Pages
• Workflows requiring search functionality
• When testing template customizations
• Validation of cross-references and internal links

Sources: scripts/build-docs.sh26 scripts/build-docs.sh:68-92

Dismiss

Refresh this wiki

Enter email to refresh