Overview

Loading…

Overview

Relevant source files

• README.md

Purpose and Scope

This document introduces the DeepWiki-to-mdBook Converter, a containerized system that extracts wiki documentation from DeepWiki.com and transforms it into searchable HTML documentation using mdBook. This page covers the system’s purpose, core capabilities, and high-level architecture.

For detailed usage instructions, see Quick Start. For architecture details, see System Architecture. For configuration options, see Configuration Reference.

Sources: README.md:1-3

Problem Statement

DeepWiki.com provides AI-generated documentation for GitHub repositories as a web-based wiki. The system addresses the following limitations:

Sources: README.md:3-15

Core Capabilities

The system provides the following capabilities through environment variable configuration:

• Generic Repository Support : Works with any GitHub repository indexed by DeepWiki via REPO environment variable
• Auto-Detection : Extracts repository metadata from Git remotes when available
• Hierarchy Preservation : Maintains wiki page numbering and section structure
• Diagram Intelligence : Extracts ~461 total diagrams, matches ~48 with sufficient context using fuzzy matching
• Dual Output Modes : Full mdBook build or markdown-only extraction via MARKDOWN_ONLY flag
• No Authentication : Public HTTP scraping without API keys or credentials
• Containerized Deployment : Single Docker image with all dependencies

Sources: README.md:5-15 README.md:42-51

System Components

The system consists of three primary executable components coordinated by a shell orchestrator. The following diagram maps user interaction to specific code entities:

Component Architecture with Code Entities

graph TB
    DockerRun["docker run\nwith env vars"]
subgraph Container["/usr/local/bin/ executables"]
BuildDocs["build-docs.sh"]
Scraper["deepwiki-scraper.py\nmain()\nscrape_wiki()\nextract_mermaid_from_nextjs_data()"]
MdBook["mdbook binary\n(Rust)"]
MermaidPlugin["mdbook-mermaid binary\n(Rust)"]
end
    
    subgraph External["External HTTP Endpoints"]
DeepWikiAPI["deepwiki.com/$REPO"]
GitHubEdit["github.com/$REPO/edit/"]
end
    
    subgraph OutputVol["/output volume mount"]
MarkdownDir["markdown/\nnumbered .md files"]
RawMarkdownDir["raw_markdown/\npre-enhancement .md"]
BookDir["book/\nindex.html + search"]
ConfigFile["book.toml"]
SummaryFile["SUMMARY.md"]
end
    
 
   DockerRun -->|REPO BOOK_TITLE BOOK_AUTHORS MARKDOWN_ONLY| BuildDocs
    
 
   BuildDocs -->|python3 deepwiki-scraper.py| Scraper
 
   BuildDocs -->|mdbook init| MdBook
 
   BuildDocs -->|mdbook build| MdBook
 
   BuildDocs -->|generates| ConfigFile
 
   BuildDocs -->|generates| SummaryFile
    
 
   Scraper -->|requests.get| DeepWikiAPI
 
   Scraper -->|writes| RawMarkdownDir
 
   Scraper -->|writes enhanced| MarkdownDir
    
 
   MdBook -->|preprocessor chain| MermaidPlugin
 
   MdBook -->|generates| BookDir
    
 
   BookDir -.->|edit links point to| GitHubEdit

Executable Components

Sources: README.md:1-27 README.md:84-88 Diagram 1, Diagram 3

Processing Pipeline

The system executes a three-phase pipeline with conditional execution based on the MARKDOWN_ONLY environment variable. Each phase invokes specific executables and functions:

Three-Phase Execution Flow with Code Entities

stateDiagram-v2
    [*] --> ParseEnv : build-docs.sh reads env
    
    state ParseEnv {
        [*] --> ReadREPO : $REPO
        ReadREPO --> ReadBOOKTITLE : $BOOK_TITLE
        ReadBOOKTITLE --> ReadMARKDOWNONLY : $MARKDOWN_ONLY
        ReadMARKDOWNONLY --> [*]
    }
    
    ParseEnv --> Phase1 : python3 deepwiki-scraper.py
    
    state Phase1 {
        [*] --> scrape_wiki
        scrape_wiki --> BeautifulSoup4 : parse HTML
        BeautifulSoup4 --> html2text : convert to .md
        html2text --> extract_mermaid : extract_mermaid_from_nextjs_data()
        extract_mermaid --> normalize_mermaid : 7-step normalization
        normalize_mermaid --> inject_diagrams : inject_mermaid_diagrams_into_markdown()
        inject_diagrams --> write_files : /output/markdown/*.md
        write_files --> [*]
    }
    
    Phase1 --> CheckMode
    
    state CheckMode <<choice>>
    CheckMode --> Phase2 : if MARKDOWN_ONLY != true
    CheckMode --> Exit : if MARKDOWN_ONLY == true
    
    Phase2 --> GenerateBookToml : build-docs.sh writes book.toml
    GenerateBookToml --> GenerateSummary : build-docs.sh writes SUMMARY.md
    
    GenerateSummary --> Phase3
    
    state Phase3 {
        [*] --> mdbook_init : mdbook init
        mdbook_init --> mdbook_mermaid_install : mdbook-mermaid install
        mdbook_mermaid_install --> mdbook_build : mdbook build
        mdbook_build --> [*] : /output/book/
    }
    
    Phase3 --> Exit
    Exit --> [*]

Phase Execution Details

Sources: README.md:72-77 Diagram 2, Diagram 4

Input and Output

Input Requirements

Sources: README.md:42-51

Output Artifacts

Full Build Mode (MARKDOWN_ONLY=false or unset):

output/
├── markdown/
│   ├── 1-overview.md
│   ├── 2-quick-start.md
│   ├── section-3/
│   │   ├── 3-1-workspace.md
│   │   └── 3-2-parser.md
│   └── ...
├── book/
│   ├── index.html
│   ├── searchindex.json
│   ├── mermaid.min.js
│   └── ...
└── book.toml

Markdown-Only Mode (MARKDOWN_ONLY=true):

output/
└── markdown/
    ├── 1-overview.md
    ├── 2-quick-start.md
    └── ...

Sources: README.md:89-119

Technical Stack

The system combines multiple technology stacks in a single container using Docker multi-stage builds:

Runtime Dependencies

Build Architecture

The Dockerfile uses a two-stage build:

1. Stage 1 (rust:latest): Compiles mdbook and mdbook-mermaid binaries (~1.5 GB, discarded)
2. Stage 2 (python:3.12-slim): Copies binaries into Python runtime (~300-400 MB final)

Sources: README.md:146-157 Diagram 3

graph TB
    subgraph HostFS["Host Filesystem"]
HostOutput["./output/\n(bind mount)"]
end
    
    subgraph ContainerFS["Container Filesystem"]
BuildScript["/usr/local/bin/build-docs.sh"]
ScraperScript["/usr/local/bin/deepwiki-scraper.py"]
TmpWiki["/tmp/wiki_temp/\n(write buffer)"]
OutputMount["/output/\n(volume mount)"]
WorkspaceTemplates["/workspace/templates/\nheader.html, footer.html"]
end
    
    subgraph WriteOperations["File Write Operations"]
WriteRaw["write_markdown_file()\nraw .md to /tmp"]
WriteEnhanced["write enhanced .md\nafter diagram injection"]
AtomicMove["shutil.move()\nor mv command"]
CopyBook["cp -r mdbook_output/"]
end
    
    subgraph OutputStructure["/output/ final structure"]
OutMarkdown["/output/markdown/"]
OutRaw["/output/raw_markdown/"]
OutBook["/output/book/"]
OutConfig["/output/book.toml"]
end
    
 
   HostOutput -.->|-v bind mount| OutputMount
    
 
   ScraperScript -->|Phase 1| WriteRaw
 
   WriteRaw --> TmpWiki
 
   ScraperScript --> WriteEnhanced
 
   WriteEnhanced --> TmpWiki
    
 
   TmpWiki -->|atomic| AtomicMove
 
   AtomicMove --> OutMarkdown
 
   AtomicMove --> OutRaw
    
 
   BuildScript -->|Phase 2| OutConfig
 
   BuildScript -->|Phase 3| CopyBook
 
   CopyBook --> OutBook
    
 
   WorkspaceTemplates -.->|process-template.py reads| BuildScript
    
 
   OutMarkdown --> OutputMount
 
   OutRaw --> OutputMount
 
   OutBook --> OutputMount
 
   OutConfig --> OutputMount

File System Interaction

The system uses a temporary directory pattern to ensure atomic writes to the output volume:

Filesystem Write Pattern

Write Sequence

1. deepwiki-scraper.py writes raw markdown to /tmp/wiki_temp/ using write_markdown_file() function
2. After diagram injection via inject_mermaid_diagrams_into_markdown(), enhanced markdown moves to /output/markdown/
3. build-docs.sh generates /output/book.toml from environment variables
4. mdbook build writes HTML to internal directory, which build-docs.sh copies to /output/book/

This pattern ensures atomicity: partial writes never appear in /output/.

Sources: README.md:19-26 README.md:54-58 Diagram 3

Configuration Philosophy

The system operates on three configuration principles:

1. Environment-Driven : All customization via environment variables, no file editing required
2. Auto-Detection : Intelligent defaults from Git remotes (repository URL, author name)
3. Zero-Configuration : Minimal required inputs (REPO or auto-detect from current directory)

Minimal Example :

This single command triggers the complete extraction, transformation, and build pipeline.

For complete configuration options, see Configuration Reference. For deployment patterns, see Quick Start.

Sources: README.md:22-51 README.md:220-227

Dismiss

Refresh this wiki

Enter email to refresh

Quick Start

Loading…

Quick Start

Relevant source files

• README.md

This page provides step-by-step instructions for basic usage of the DeepWiki-to-mdBook converter. It covers Docker image building, container execution with environment variables, output inspection, and local serving. For comprehensive configuration options, see Configuration Reference. For understanding the internal pipeline, see System Architecture.

Prerequisites

Before starting, ensure you have the following installed:

• Docker (version 20.10 or later)
• Git (for repository cloning)
• Python 3 (for local serving)

The system requires no additional dependencies on the host machine; all build tools and Python packages are contained within the Docker image.

Sources: README.md:1-95

Basic Workflow

The typical workflow involves three steps: building the Docker image, running the container to generate documentation, and serving the output locally for preview.

flowchart TD
    Start["User initiates workflow"]
Clone["Clone repository\ngit clone"]
Build["Build Docker image\ndocker build -t deepwiki-to-mdbook ."]
Run["Run container with config\ndocker run --rm -e REPO=... -v ..."]
subgraph Container["Docker Container Execution"]
BuildScript["build-docs.sh"]
Scraper["deepwiki-scraper.py"]
Process["process-template.py"]
MdBook["mdbook build"]
end
    
    Output["Output directory\n/output mounted volume"]
Serve["Serve locally\npython3 -m http.server"]
View["View in browser\nhttp://localhost:8000"]
Start --> Clone
 
   Clone --> Build
 
   Build --> Run
 
   Run --> Container
 
   Container --> BuildScript
 
   BuildScript --> Scraper
 
   BuildScript --> Process
 
   BuildScript --> MdBook
 
   MdBook --> Output
 
   Output --> Serve
 
   Serve --> View

Workflow Diagram

Sources: README.md:12-29

Step 1: Build the Docker Image

Navigate to the repository root and build the Docker image:

This command:

• Reads the multi-stage Dockerfile at the repository root
• Compiles mdbook and mdbook-mermaid from source in the Rust builder stage
• Installs Python 3.12 dependencies in the final stage
• Tags the resulting image as deepwiki-to-mdbook

The build process typically takes 5-10 minutes on the first run due to Rust compilation. Subsequent builds use Docker layer caching.

Sources: README.md:14-16

Step 2: Run the Container

Execute the container with required environment variables and a volume mount for output:

Environment Variable Configuration

The container accepts configuration exclusively through environment variables:

*The REPO variable is auto-detected if the container is run in a Git repository context. For manual execution, it should be explicitly provided.

Volume Mount

The -v "$(pwd)/output:/output" mount maps the host’s ./output directory to the container’s /output directory. All generated artifacts are written here.

Sources: README.md:18-27 README.md:31-51

Step 3: Serve and View Output

After the container completes execution, serve the generated HTML documentation locally:

This command:

• Changes to the output directory
• Starts Python’s built-in HTTP server
• Serves files from the book subdirectory
• Listens on port 8000

Open http://localhost:8000 in a web browser to view the searchable documentation.

Output Directory Structure

The container generates four output artifacts:

Directory Descriptions:

• book/ : The final HTML output generated by mdbook build. This is the directory to serve for viewing documentation.
• markdown/ : Enhanced markdown files after diagram injection and template processing. Contains the source files used by mdBook.
• raw_markdown/ : Markdown files immediately after HTML-to-markdown conversion, before diagram enhancement. Useful for debugging the extraction phase.
• book.toml : The mdBook configuration file generated from environment variables.

Sources: README.md:26-27 README.md:53-58

flowchart LR
    CMD["Container CMD\nbuild-docs.sh"]
subgraph Phase1["Phase 1: Extraction"]
Fetch["fetch_and_convert_to_markdown()\ndeepwiki-scraper.py"]
RawOut["raw_markdown/\ndirectory"]
end
    
    subgraph Phase2["Phase 2: Enhancement"]
Extract["extract_mermaid_from_nextjs_data()\ndeepwiki-scraper.py"]
Normalize["normalize_mermaid_diagram()\ndeepwiki-scraper.py"]
Inject["inject_mermaid_diagrams()\ndeepwiki-scraper.py"]
Templates["process-template.py\n--input templates/header.html"]
MdOut["markdown/\ndirectory"]
end
    
    subgraph Phase3["Phase 3: Build"]
Summary["generate_summary()\nbuild-docs.sh"]
BookInit["mdbook init\nbinary"]
BookBuild["mdbook build\nbinary"]
BookOut["book/\ndirectory"]
end
    
 
   CMD --> Fetch
 
   Fetch --> RawOut
 
   RawOut --> Extract
 
   Extract --> Normalize
 
   Normalize --> Inject
 
   Inject --> Templates
 
   Templates --> MdOut
 
   MdOut --> Summary
 
   Summary --> BookInit
 
   BookInit --> BookBuild
 
   BookBuild --> BookOut

Container Execution Flow

The following diagram maps the container’s internal execution to specific code entities:

This diagram shows the execution path through the three phases of the pipeline, with references to actual functions and binaries. The build-docs.sh script orchestrates all three phases sequentially. For detailed information on each phase, see Three-Phase Pipeline.

Sources: README.md:72-77

Common Usage Patterns

Pattern 1: Custom Book Title and Authors

Pattern 2: Markdown-Only Mode

Skip the HTML build to inspect or further process the markdown files:

The markdown/ directory will contain all enhanced markdown files, but the book/ directory will not be created. See Markdown-Only Mode for more details.

Pattern 3: Custom Templates

Provide custom header and footer templates by mounting a templates directory:

Your my-templates/ directory should contain header.html and/or footer.html. See Template System for template variable syntax and Custom Templates for a comprehensive guide.

Sources: README.md:39-51

Minimal Example

For a minimal working example with auto-detected repository:

The REPO variable is auto-detected from git remote get-url origin, and BOOK_AUTHORS defaults to the repository owner.

Sources: README.md:12-29 README.md34

Verification Steps

After running the container, verify successful execution:

1. Check container logs : The container should print progress messages for each phase.
2. Inspect output directory : Ensure all four artifacts exist (book/, markdown/, raw_markdown/, book.toml).
3. Verify HTML structure : book/index.html should exist and contain the search interface.
4. Test local serving : The HTTP server should start without errors.
5. Browse documentation : Navigate to http://localhost:8000 and verify page rendering and search functionality.

Next Steps

After completing the Quick Start:

• Customize configuration : See Configuration Reference for all environment variables and options.
• Understand the pipeline : See System Architecture for how the system transforms DeepWiki content.
• Use in CI/CD : See GitHub Action to integrate into automated workflows.
• Customize templates : See Template System Details to brand your documentation.

Sources: README.md:1-95

Dismiss

Refresh this wiki

Enter email to refresh

Configuration Reference

Loading…

Configuration Reference

Relevant source files

• README.md
• action.yml
• scripts/build-docs.sh

This document provides a comprehensive reference for all configuration options available in the DeepWiki-to-mdBook Converter system. It covers environment variables, their default values, validation logic, auto-detection features, and how configuration flows through the system components.

For information about running the system with these configurations, see Quick Start. For details on how auto-detection works internally, see Auto-Detection Features.

Configuration System Overview

The DeepWiki-to-mdBook Converter uses environment variables as its sole configuration mechanism. All configuration is processed by the build-docs.sh orchestrator script at runtime, with no configuration files required. The system provides intelligent defaults and auto-detection capabilities to minimize required configuration.

Configuration Flow Diagram

flowchart TD
    User["User/CI System"]
Docker["docker run -e VAR=value"]
subgraph "build-docs.sh Configuration Processing"
        AutoDetect["Git Auto-Detection\n[build-docs.sh:8-19]"]
ParseEnv["Environment Variable Parsing\n[build-docs.sh:21-26]"]
Defaults["Default Value Assignment\n[build-docs.sh:43-45]"]
Validate["Validation\n[build-docs.sh:32-37]"]
end
    
    subgraph "Configuration Consumers"
        Scraper["deepwiki-scraper.py\nREPO parameter"]
BookToml["book.toml Generation\n[build-docs.sh:85-103]"]
SummaryGen["SUMMARY.md Generation\n[build-docs.sh:113-159]"]
end
    
 
   User -->|Set environment variables| Docker
 
   Docker -->|Container startup| AutoDetect
 
   AutoDetect -->|REPO detection| ParseEnv
 
   ParseEnv -->|Parse all vars| Defaults
 
   Defaults -->|Apply defaults| Validate
 
   Validate -->|REPO validated| Scraper
 
   Validate -->|BOOK_TITLE, BOOK_AUTHORS, GIT_REPO_URL| BookToml
 
   Validate -->|No direct config needed| SummaryGen

Sources: build-docs.sh:1-206 README.md:41-51

Environment Variables Reference

The following table lists all environment variables supported by the system:

Sources: build-docs.sh:21-26 README.md:44-51

Variable Details and Usage

REPO

Format: owner/repo (e.g., "facebook/react" or "microsoft/vscode")

Purpose: Identifies the GitHub repository to scrape from DeepWiki.com. This is the primary configuration variable that drives the entire system.

flowchart TD
    Start["build-docs.sh Startup"]
CheckEnv{"REPO environment\nvariable set?"}
UseEnv["Use provided REPO value\n[build-docs.sh:22]"]
CheckGit{"Git repository\ndetected?"}
GetRemote["Execute: git config --get\nremote.origin.url\n[build-docs.sh:12]"]
ParseURL["Extract owner/repo using regex:\n.*github\\.com[:/]([^/]+/[^/\\.]+)\n[build-docs.sh:16]"]
SetRepo["Set REPO variable\n[build-docs.sh:16]"]
ValidateRepo{"REPO is set?"}
Error["Exit with error\n[build-docs.sh:33-37]"]
Continue["Continue with\nREPO=$REPO_OWNER/$REPO_NAME"]
Start --> CheckEnv
 
   CheckEnv -->|Yes| UseEnv
 
   CheckEnv -->|No| CheckGit
 
   CheckGit -->|Yes| GetRemote
 
   CheckGit -->|No| ValidateRepo
 
   GetRemote --> ParseURL
 
   ParseURL --> SetRepo
 
   UseEnv --> ValidateRepo
 
   SetRepo --> ValidateRepo
 
   ValidateRepo -->|No| Error
 
   ValidateRepo -->|Yes| Continue

Auto-Detection Logic:

Sources: build-docs.sh:8-37

Validation: The system exits with an error if REPO is not set and cannot be auto-detected:

ERROR: REPO must be set or run from within a Git repository with a GitHub remote
Usage: REPO=owner/repo $0

Usage in System:

• Passed as first argument to deepwiki-scraper.py build-docs.sh58
• Used to derive REPO_OWNER and REPO_NAME build-docs.sh:40-41
• Used to construct GIT_REPO_URL default build-docs.sh45

BOOK_TITLE

Default: "Documentation"

Purpose: Sets the title of the generated mdBook documentation. This appears in the browser tab, navigation header, and book metadata.

Usage: Injected into book.toml configuration file build-docs.sh87:

Examples:

• BOOK_TITLE="React Documentation"
• BOOK_TITLE="VS Code Internals"
• BOOK_TITLE="Apache Arrow DataFusion Developer Guide"

Sources: build-docs.sh23 build-docs.sh87

BOOK_AUTHORS

Default: Repository owner extracted from REPO

Purpose: Sets the author name(s) in the mdBook documentation metadata.

Default Assignment Logic: build-docs.sh44

This uses shell parameter expansion to set BOOK_AUTHORS to REPO_OWNER only if BOOK_AUTHORS is unset or empty.

Usage: Injected into book.toml as an array build-docs.sh88:

Examples:

• If REPO="facebook/react" and BOOK_AUTHORS not set → BOOK_AUTHORS="facebook"
• Explicitly set: BOOK_AUTHORS="Meta Open Source"
• Multiple authors: BOOK_AUTHORS="John Doe, Jane Smith" (rendered as single string in array)

Sources: build-docs.sh24 build-docs.sh44 build-docs.sh88

GIT_REPO_URL

Default: https://github.com/{REPO}

Purpose: Provides the full GitHub repository URL used for “Edit this page” links in the generated mdBook documentation. Each page includes a link back to the source repository.

Default Assignment Logic: build-docs.sh45

Usage: Injected into book.toml configuration build-docs.sh95:

Notes:

• mdBook automatically appends /edit/main/ or similar paths based on its heuristics
• The URL must be a valid Git repository URL for the edit links to work correctly
• Can be overridden for non-standard Git hosting scenarios

Sources: build-docs.sh25 build-docs.sh45 build-docs.sh95

MARKDOWN_ONLY

Default: "false"

Type: Boolean string ("true" or "false")

Purpose: Controls whether the system executes the full three-phase pipeline or stops after Phase 2 (Markdown extraction with diagram enhancement). When set to "true", Phase 3 (mdBook build) is skipped.

flowchart TD
    Start["build-docs.sh Execution"]
Phase1["Phase 1: Scrape & Extract\n[build-docs.sh:56-58]"]
Phase2["Phase 2: Enhance Diagrams\n(within deepwiki-scraper.py)"]
CheckMode{"MARKDOWN_ONLY\n== 'true'?\n[build-docs.sh:61]"}
CopyMD["Copy markdown to /output/markdown\n[build-docs.sh:64-65]"]
ExitEarly["Exit (skipping mdBook build)\n[build-docs.sh:75]"]
Phase3Init["Phase 3: Initialize mdBook\n[build-docs.sh:79-106]"]
BuildBook["Build HTML documentation\n[build-docs.sh:176]"]
CopyAll["Copy all outputs\n[build-docs.sh:179-191]"]
Start --> Phase1
 
   Phase1 --> Phase2
 
   Phase2 --> CheckMode
 
   CheckMode -->|Yes| CopyMD
 
   CopyMD --> ExitEarly
 
   CheckMode -->|No| Phase3Init
 
   Phase3Init --> BuildBook
 
   BuildBook --> CopyAll
    
    style ExitEarly fill:#ffebee
    style CopyAll fill:#e8f5e9

Execution Flow with MARKDOWN_ONLY:

Sources: build-docs.sh26 build-docs.sh:61-76

Use Cases:

• Debugging diagram placement: Quickly iterate on diagram matching without waiting for mdBook build
• Markdown-only extraction: When you only need the Markdown source files
• Faster feedback loops: mdBook build adds significant time; skipping it speeds up testing
• Custom processing: Extract Markdown for processing with different documentation tools

Output Differences:

Performance Impact: Markdown-only mode is approximately 3-5x faster, as it skips:

• mdBook initialization build-docs.sh:79-106
• SUMMARY.md generation build-docs.sh:109-159
• File copying to book/src build-docs.sh:164-166
• mdbook-mermaid asset installation build-docs.sh:169-171
• mdBook HTML build build-docs.sh:174-176

Sources: build-docs.sh:61-76 README.md:55-76

Internal Configuration Variables

These variables are derived or used internally and are not meant to be configured by users:

Sources: build-docs.sh:27-30 build-docs.sh:40-41

Configuration Precedence and Inheritance

The system follows this precedence order for configuration values:

Sources: build-docs.sh:8-45

Example Scenarios:

1. User provides all values:

All explicit values used; no auto-detection occurs.

2. User provides only REPO:

   • REPO: "facebook/react" (explicit)
   • BOOK_TITLE: "Documentation" (default)
   • BOOK_AUTHORS: "facebook" (derived from REPO)
   • GIT_REPO_URL: "https://github.com/facebook/react" (derived)
   • MARKDOWN_ONLY: "false" (default)

3. User provides no values in Git repo:

   • REPO: Auto-detected from git config --get remote.origin.url
   • All other values derived or defaulted as above

Generated Configuration Files

The system generates configuration files dynamically based on environment variables:

book.toml

Location: Created at $BOOK_DIR/book.toml build-docs.sh85 copied to /output/book.toml build-docs.sh191

Template Structure:

Sources: build-docs.sh:85-103

Variable Substitution Mapping:

Hard-Coded Values:

• language = "en" build-docs.sh89
• default-theme = "rust" build-docs.sh94
• [preprocessor.mermaid] configuration build-docs.sh:97-98
• Sidebar folding enabled at level 1 build-docs.sh:100-102

SUMMARY.md

Location: Created at $BOOK_DIR/src/SUMMARY.md build-docs.sh159

Generation: Automatically generated from file structure in $WIKI_DIR, no direct environment variable input. See SUMMARY.md Generation for details.

Sources: build-docs.sh:109-159

Configuration Examples

Minimal Configuration

Results:

• REPO: "owner/repo"
• BOOK_TITLE: "Documentation"
• BOOK_AUTHORS: "owner"
• GIT_REPO_URL: "https://github.com/owner/repo"
• MARKDOWN_ONLY: "false"

Full Custom Configuration

Auto-Detected Configuration

Note: This only works if the current directory is a Git repository with a GitHub remote URL configured.

Debugging Configuration

Outputs only Markdown files to /output/markdown/, skipping the mdBook build phase.

Sources: README.md:28-88

Configuration Validation

The system performs validation on the REPO variable build-docs.sh:32-37:

Validation Rules:

• REPO must be non-empty after auto-detection
• No format validation is performed on REPO value (e.g., owner/repo pattern)
• Invalid REPO values will cause failures during scraping phase, not during validation

Other Variables:

• No validation performed on BOOK_TITLE, BOOK_AUTHORS, or GIT_REPO_URL
• MARKDOWN_ONLY is not validated; any value other than "true" is treated as false

Sources: build-docs.sh:32-37

Configuration Debugging

To debug configuration values, check the console output at startup build-docs.sh:47-53:

Configuration:
  Repository:    facebook/react
  Book Title:    React Documentation
  Authors:       Meta Open Source
  Git Repo URL:  https://github.com/facebook/react
  Markdown Only: false

This output shows the final resolved configuration values after auto-detection, derivation, and defaults are applied.

Sources: build-docs.sh:47-53

Dismiss

Refresh this wiki

Enter email to refresh

System Architecture

Loading…

System Architecture

Relevant source files

• Dockerfile
• README.md
• scripts/build-docs.sh

This page provides a comprehensive overview of the DeepWiki-to-mdBook converter’s architecture, including its component organization, execution model, and data flow patterns. The system is designed as a containerized pipeline that transforms DeepWiki content into searchable mdBook documentation through three distinct processing phases.

For detailed information about the three-phase transformation pipeline, see Three-Phase Pipeline. For Docker-specific implementation details, see Docker Multi-Stage Build.

Architectural Overview

The system follows a pipeline architecture with three sequential phases, orchestrated by a shell script and executed within a Docker container. All components are stateless and communicate through the filesystem, with no external dependencies required at runtime.

graph TB
    subgraph Docker["Docker Container (python:3.12-slim)"]
subgraph Executables["/usr/local/bin/"]
BuildScript["build-docs.sh"]
Scraper["deepwiki-scraper.py"]
TemplateProc["process-template.py"]
mdBook["mdbook"]
mdBookMermaid["mdbook-mermaid"]
end
        
        subgraph Workspace["/workspace/"]
Templates["templates/\nheader.html\nfooter.html"]
WorkingDirs["wiki/\nraw_markdown/\nbook/"]
end
        
        subgraph Output["/output/ (Volume Mount)"]
BookHTML["book/"]
MarkdownSrc["markdown/"]
RawMarkdown["raw_markdown/"]
BookConfig["book.toml"]
end
    end
    
    subgraph External["External Dependencies"]
DeepWiki["deepwiki.com"]
GitRemote["git remote"]
end
    
 
   BuildScript -->|executes| Scraper
 
   BuildScript -->|executes| TemplateProc
 
   BuildScript -->|executes| mdBook
 
   BuildScript -->|executes| mdBookMermaid
    
 
   Scraper -->|writes| WorkingDirs
 
   TemplateProc -->|reads| Templates
 
   TemplateProc -->|outputs HTML| BuildScript
 
   mdBook -->|reads| WorkingDirs
 
   mdBook -->|writes| BookHTML
    
 
   BuildScript -->|copies to| Output
    
 
   Scraper -->|HTTP GET| DeepWiki
 
   BuildScript -->|auto-detect| GitRemote
    
    style Executables fill:#f0f0f0
    style Workspace fill:#f0f0f0
    style Output fill:#e8f5e9

System Composition

Diagram: Container Internal Structure and Component Relationships

The container is structured into three main areas: executables in /usr/local/bin/, working files in /workspace/, and outputs in /output/. The build-docs.sh orchestrator coordinates all components, with persistent results written to the mounted volume.

Sources: Dockerfile:1-34 scripts/build-docs.sh:1-310

Core Components

The system consists of five primary components, each with a specific responsibility in the documentation generation pipeline.

Component Interaction Map

Diagram: Component Interaction and Data Flow

This diagram shows how build-docs.sh coordinates the three processing components sequentially, with data flowing through working directories before final output to the mounted volume.

Sources: scripts/build-docs.sh:1-310 Dockerfile:20-33

Execution Flow

The system follows a strictly sequential execution model, with each step depending on the output of the previous step. This design simplifies error handling and allows for debugging at intermediate stages.

Build Script Orchestration

The build-docs.sh script orchestrates the entire pipeline through seven distinct steps:

1. Configuration & Validation scripts/build-docs.sh:8-59

   • Auto-detects REPO from git remote if not provided
   • Sets defaults for BOOK_TITLE, BOOK_AUTHORS, GIT_REPO_URL
   • Validates required configuration
   • Computes derived URLs (DEEPWIKI_URL, badge URLs)

2. Wiki Scraping scripts/build-docs.sh:61-65

   • Executes deepwiki-scraper.py with repository identifier
   • Writes to /workspace/wiki/ and /workspace/raw_markdown/

3. Early Exit (Markdown-Only Mode) scripts/build-docs.sh:67-93

   • Optional: skip HTML build if MARKDOWN_ONLY=true
   • Copies markdown directly to output volume

4. mdBook Initialization scripts/build-docs.sh:95-122

   • Creates /workspace/book/ structure
   • Generates book.toml configuration
   • Initializes src/ directory

5. SUMMARY.md Generation scripts/build-docs.sh:124-188

   • Scans wiki directory for .md files
   • Sorts numerically by page number prefix
   • Builds hierarchical table of contents
   • Handles subsections in section-N/ directories

6. Template Processing & Injection scripts/build-docs.sh:190-261

   • Processes header.html and footer.html with variable substitution
   • Injects processed HTML into every markdown file
   • Copies enhanced markdown to book/src/

7. mdBook Build scripts/build-docs.sh:263-271

   • Installs mermaid assets via mdbook-mermaid install
   • Executes mdbook build
   • Generates searchable HTML in book/

8. Output Copying scripts/build-docs.sh:273-309

   • Copies all artifacts to /output/ volume mount
   • Preserves intermediate outputs for debugging

Diagram: build-docs.sh Sequential Execution Flow

Sources: scripts/build-docs.sh:1-310

Three-Phase Pipeline Architecture

The core transformation happens in three distinct phases, each with specific inputs, processing logic, and outputs. This separation allows for independent testing and debugging of each phase.

Phase Overview

Diagram: Three-Phase Pipeline with Key Functions

For detailed documentation of each phase, see Three-Phase Pipeline.

Sources: scripts/build-docs.sh:61-271 README.md:72-77

graph TB
    subgraph Stage1["Stage 1: Builder (rust:latest)"]
RustToolchain["Rust Toolchain\ncargo, rustc"]
CargoInstall["cargo install mdbook\ncargo install mdbook-mermaid"]
Binaries["/usr/local/cargo/bin/\nmdbook\nmdbook-mermaid"]
RustToolchain --> CargoInstall
 
       CargoInstall --> Binaries
    end
    
    subgraph Stage2["Stage 2: Runtime (python:3.12-slim)"]
PythonBase["Python 3.12 Runtime"]
PipInstall["pip install requirements"]
CopyBinaries["COPY --from=builder\nmdbook binaries"]
CopyScripts["COPY Python scripts\nCOPY Shell scripts\nCOPY Templates"]
FinalImage["Final Image\n~500MB"]
PythonBase --> PipInstall
 
       PipInstall --> CopyBinaries
 
       CopyBinaries --> CopyScripts
 
       CopyScripts --> FinalImage
    end
    
 
   Binaries -.->|copy only| CopyBinaries
    
    style Stage1 fill:#ffebee
    style Stage2 fill:#e8f5e9
    style FinalImage fill:#c8e6c9

Docker Multi-Stage Build

The Docker architecture uses a multi-stage build pattern to minimize final image size while compiling Rust-based tools from source. This approach separates build-time dependencies from runtime dependencies.

Stage Architecture

Diagram: Multi-Stage Build Process

The builder stage (approximately 2GB) is discarded after compilation, and only the compiled binaries (approximately 50MB) are copied to the final image, resulting in a significantly smaller runtime image.

Container Filesystem Layout

/usr/local/bin/
├── mdbook                    # Rust binary from builder stage
├── mdbook-mermaid            # Rust binary from builder stage
├── deepwiki-scraper.py       # Python script (executable)
├── process-template.py       # Python script (executable)
└── build-docs.sh             # Shell script (executable)

/workspace/
├── templates/
│   ├── header.html           # Default header template
│   └── footer.html           # Default footer template
├── wiki/                     # Created at runtime
├── raw_markdown/             # Created at runtime
└── book/                     # Created at runtime

/output/                      # Volume mount point
└── (user-provided volume)

For detailed Docker implementation information, see Docker Multi-Stage Build.

Sources: Dockerfile:1-34

Configuration Architecture

The system is configured entirely through environment variables and volume mounts , following the Twelve-Factor App methodology. No configuration files are required; all settings have sensible defaults.

Configuration Layers

1. Auto-Detection scripts/build-docs.sh:8-19

   • Extracts REPO from git remote get-url origin
   • Supports GitHub URLs in multiple formats (HTTPS, SSH)

2. Environment Variables scripts/build-docs.sh:21-26

   • User-provided overrides
   • Takes precedence over auto-detection

3. Computed Defaults scripts/build-docs.sh:40-51

   • Derives BOOK_AUTHORS from REPO owner
   • Constructs GIT_REPO_URL from REPO
   • Generates badge URLs

Diagram: Configuration Resolution Order

Key Configuration Variables

For complete configuration documentation, see Configuration Reference.

Sources: scripts/build-docs.sh:8-59 README.md:31-51

Output Artifacts

The system produces four distinct output artifacts, each serving a specific purpose in the documentation workflow:

The multi-artifact design allows users to inspect intermediate stages, debug transformation issues, or use the markdown files for alternative processing workflows.

For detailed output structure documentation, see Output Structure.

Sources: scripts/build-docs.sh:273-309 README.md:53-58

Extensibility Points

The architecture provides three primary extension mechanisms:

1. Custom Templates

Users can override default header/footer templates by mounting a custom directory:

The process-template.py script performs variable substitution on any mounted templates, supporting custom branding and layout.

Sources: scripts/build-docs.sh:195-234 Dockerfile26

2. Environment Variable Configuration

All behavior can be modified through environment variables without rebuilding the Docker image. This includes metadata, URLs, and operational modes.

Sources: scripts/build-docs.sh:8-59

3. Markdown-Only Mode

Setting MARKDOWN_ONLY=true allows users to skip the HTML build entirely, enabling alternative processing pipelines or custom mdBook configurations.

Sources: scripts/build-docs.sh:67-93

For advanced customization patterns, see Advanced Topics.

Summary

The DeepWiki-to-mdBook converter implements a pipeline architecture with three distinct phases (extraction, enhancement, build), orchestrated by a shell script within a multi-stage Docker container. The system is stateless, configuration-driven, and produces multiple output artifacts for different use cases. All components communicate through the filesystem, with no runtime dependencies beyond the container image.

Key architectural principles:

• Sequential processing : Each phase depends on the previous phase’s output
• Stateless execution : No persistent state between runs
• Configuration through environment : No config files required
• Multi-stage build : Minimized runtime image size
• Multiple outputs : Debugging and alternative workflows supported

Sources: Dockerfile:1-34 scripts/build-docs.sh:1-310 README.md:1-95

Dismiss

Refresh this wiki

Enter email to refresh

Three-Phase Pipeline

Loading…

Three-Phase Pipeline

Relevant source files

• python/deepwiki-scraper.py
• scripts/build-docs.sh

Purpose and Scope

This document describes the three-phase processing pipeline that transforms DeepWiki HTML pages into searchable mdBook documentation. The pipeline consists of Phase 1: Clean Markdown Extraction , Phase 2: Diagram Enhancement , and Phase 3: mdBook Build. Each phase has distinct responsibilities and uses different technology stacks.

For overall system architecture, see System Architecture. For detailed implementation of individual phases, see Phase 1: Markdown Extraction, Phase 2: Diagram Enhancement, and Phase 3: mdBook Build. For configuration that affects pipeline behavior, see Configuration Reference.

Pipeline Overview

The system processes content through three sequential phases, with an optional bypass mechanism for Phase 3. Each phase is implemented by different components and operates on files in specific directories.

Pipeline Execution Flow

stateDiagram-v2
    [*] --> Init["build-docs.sh\nParse env vars"]
    
    Init --> Phase1["Phase 1 : deepwiki-scraper.py"]
    
    state "Phase 1 : Markdown Extraction" as Phase1 {
        [*] --> ExtractStruct["extract_wiki_structure()"]
        ExtractStruct --> LoopPages["for page in pages"]
        LoopPages --> ExtractPage["extract_page_content(url)"]
        ExtractPage --> ConvertHTML["convert_html_to_markdown()"]
        ConvertHTML --> CleanFooter["clean_deepwiki_footer()"]
        CleanFooter --> WriteTemp["/workspace/wiki/*.md"]
        WriteTemp --> LoopPages
        LoopPages --> RawSnapshot["/workspace/raw_markdown/\n(debug snapshot)"]
    }
    
    Phase1 --> Phase2["Phase 2 : deepwiki-scraper.py"]
    
    state "Phase 2 : Diagram Enhancement" as Phase2 {
        [*] --> ExtractDiagrams["extract_and_enhance_diagrams()"]
        ExtractDiagrams --> FetchJS["Fetch JS payload\nextract_mermaid_from_nextjs_data()"]
        FetchJS --> NormalizeDiagrams["normalize_mermaid_diagram()\n7 normalization passes"]
        NormalizeDiagrams --> FuzzyMatch["Fuzzy match loop\n300/200/150/100/80 char chunks"]
        FuzzyMatch --> InjectFiles["Modify /workspace/wiki/*.md\nInsert ```mermaid blocks"]
    }
    
    Phase2 --> CheckMode{"MARKDOWN_ONLY\nenv var?"}
    
    CheckMode --> CopyMarkdown["build-docs.sh\ncp -r /workspace/wiki /output/markdown"] : true
    CheckMode --> Phase3["Phase 3 : build-docs.sh"] - false
    
    state "Phase 3 : mdBook Build" as Phase3 {
        [*] --> GenToml["Generate book.toml\n[book], [output.html]"]
        GenToml --> GenSummary["Generate src/SUMMARY.md\nScan .md files"]
        GenSummary --> CopyToSrc["cp -r /workspace/wiki/* src/"]
        CopyToSrc --> MermaidInstall["mdbook-mermaid install"]
        MermaidInstall --> MdBookBuild["mdbook build"]
        MdBookBuild --> OutputBook["/workspace/book/book/"]
    }
    
    Phase3 --> CopyAll["cp -r book /output/\ncp -r markdown /output/"]
    CopyMarkdown --> Done["/output directory\nready"]
    CopyAll --> Done
    Done --> [*]

Sources: scripts/build-docs.sh:61-93 python/deepwiki-scraper.py:1277-1408 python/deepwiki-scraper.py:880-1276

Phase Coordination

The build-docs.sh orchestrator coordinates all three phases and handles the decision point for markdown-only mode.

Orchestrator Control Flow

flowchart TD
    Start["Container entrypoint\nCMD: /usr/local/bin/build-docs.sh"]
Start --> ParseEnv["Parse environment\n$REPO, $BOOK_TITLE, $BOOK_AUTHORS\n$MARKDOWN_ONLY, $GIT_REPO_URL"]
ParseEnv --> CheckRepo{"$REPO\nset?"}
CheckRepo -->|No| GitDetect["git config --get remote.origin.url\nsed -E 's#.*github.com[:/]([^/]+/[^/.]+).*#\1#'"]
CheckRepo -->|Yes| SetVars["Set defaults:\nBOOK_AUTHORS=$REPO_OWNER\nGIT_REPO_URL=https://github.com/$REPO"]
GitDetect --> SetVars
    
 
   SetVars --> SetPaths["WORK_DIR=/workspace\nWIKI_DIR=/workspace/wiki\nRAW_DIR=/workspace/raw_markdown\nOUTPUT_DIR=/output\nBOOK_DIR=/workspace/book"]
SetPaths --> CallScraper["python3 /usr/local/bin/deepwiki-scraper.py $REPO $WIKI_DIR"]
CallScraper --> ScraperRuns["deepwiki-scraper.py executes:\nPhase 1: extract_wiki_structure()\nPhase 2: extract_and_enhance_diagrams()"]
ScraperRuns --> CheckMode{"$MARKDOWN_ONLY\n== 'true'?"}
CheckMode -->|Yes| QuickCopy["rm -rf $OUTPUT_DIR/markdown\nmkdir -p $OUTPUT_DIR/markdown\ncp -r $WIKI_DIR/. $OUTPUT_DIR/markdown/\nexit 0"]
CheckMode -->|No| MkdirBook["mkdir -p $BOOK_DIR\ncd $BOOK_DIR"]
MkdirBook --> GenToml["cat > book.toml <<EOF\n[book] title=$BOOK_TITLE\n[output.html] git-repository-url=$GIT_REPO_URL\n[preprocessor.mermaid]"]
GenToml --> MkdirSrc["mkdir -p src"]
MkdirSrc --> GenSummary["Generate src/SUMMARY.md\nls $WIKI_DIR/*.md /sort -t- -k1 -n for file: head -1 $file/ sed 's/^# //'"]
GenSummary --> CopyToSrc["cp -r $WIKI_DIR/* src/"]
CopyToSrc --> ProcessTemplates["python3 process-template.py header.html\npython3 process-template.py footer.html\nInject into src/*.md"]
ProcessTemplates --> InstallMermaid["mdbook-mermaid install $BOOK_DIR"]
InstallMermaid --> BuildBook["mdbook build"]
BuildBook --> CopyOutputs["mkdir -p $OUTPUT_DIR\ncp -r book $OUTPUT_DIR/\ncp -r $WIKI_DIR/. $OUTPUT_DIR/markdown/\ncp book.toml $OUTPUT_DIR/"]
QuickCopy --> Done["Exit 0"]
CopyOutputs --> Done

Sources: scripts/build-docs.sh:8-47 scripts/build-docs.sh:61-93 scripts/build-docs.sh:95-309

Phase 1: Clean Markdown Extraction

Phase 1 discovers the wiki structure and converts HTML pages to clean Markdown, writing files to a temporary directory (/workspace/wiki). This phase is implemented entirely in Python within deepwiki-scraper.py.

Phase 1 Data Flow

flowchart LR
    DeepWiki["https://deepwiki.com/$REPO"]
DeepWiki -->|session.get base_url| ExtractStruct["extract_wiki_structure(repo, session)"]
ExtractStruct -->|soup.find_all 'a', href=re.compile ...| ParseLinks["Parse sidebar links\nPattern: /$REPO/\d+"]
ParseLinks --> PageList["pages = [\n {'number': '1', 'title': 'Overview',\n 'url': '...', 'href': '...', 'level': 0},\n {'number': '2.1', 'title': 'Sub',\n 'url': '...', 'href': '...', 'level': 1},\n ...\n]\nsorted by page number"]
PageList --> Loop["for page in pages:]
 Loop --> FetchPage[fetch_page(url, session)\nUser-Agent header\n3 retries with timeout=30"]
FetchPage --> ParseHTML["BeautifulSoup(response.text)\nRemove: nav, header, footer, aside\nFind: article or main or body"]
ParseHTML --> ConvertMD["h = html2text.HTML2Text()\nh.body_width = 0\nmarkdown = h.handle(html_content)"]
ConvertMD --> CleanFooter["clean_deepwiki_footer(markdown)\nRegex patterns:\n'Dismiss', 'Refresh this wiki',\n'On this page', 'Edit Wiki'"]
CleanFooter --> FixLinks["fix_wiki_link(match)\nRegex: /owner/repo/(\d+(?:\.\d+)*)-(.+)\nConvert to: section-N/N-M-slug.md"]
FixLinks --> ResolvePath["resolve_output_path(page_number, title)\nnormalized_number_parts()\nsanitize_filename()"]
ResolvePath --> WriteFile["filepath.write_text(markdown)\nMain: /workspace/wiki/N-slug.md\nSub: /workspace/wiki/section-N/N-M-slug.md"]
WriteFile --> Loop

Sources: python/deepwiki-scraper.py:116-163 python/deepwiki-scraper.py:751-877 python/deepwiki-scraper.py:213-228 python/deepwiki-scraper.py:165-211 python/deepwiki-scraper.py:22-26 python/deepwiki-scraper.py:28-53

Key Functions and Their Roles

File Organization Logic

flowchart TD
    PageNum["page['number']\n(from DeepWiki)"]
PageNum --> Normalize["normalized_number_parts(page_number)\nSplit on '.', shift main number down by 1\nDeepWiki '1' → []\nDeepWiki '2' → ['1']\nDeepWiki '2.1' → ['1', '1']"]
Normalize --> CheckParts{"len(parts)?"}
CheckParts -->|0 was page 1| RootOverview["Filename: overview.md\nPath: $WIKI_DIR/overview.md\nNo section dir"]
CheckParts -->|1 main page| RootMain["Filename: N-slug.md\nExample: 1-quick-start.md\nPath: $WIKI_DIR/1-quick-start.md\nNo section dir"]
CheckParts -->|2+ subsection| ExtractSection["main_section = parts[0]\nsection_dir = f'section-{main_section}'"]
ExtractSection --> CreateDir["section_path = Path($WIKI_DIR) / section_dir\nsection_path.mkdir(exist_ok=True)"]
CreateDir --> SubFile["Filename: N-M-slug.md\nExample: 1-1-installation.md\nPath: $WIKI_DIR/section-1/1-1-installation.md"]

The system organizes files hierarchically based on page numbering. DeepWiki pages are numbered starting from 1, but the system shifts them down by 1 so that the first page becomes unnumbered (the overview).

Sources: python/deepwiki-scraper.py:28-43 python/deepwiki-scraper.py:45-63 python/deepwiki-scraper.py:1332-1338

Phase 2: Diagram Enhancement

Phase 2 extracts Mermaid diagrams from the JavaScript payload and uses fuzzy matching to intelligently place them in the appropriate Markdown files. This phase operates on files in the temporary directory (/workspace/wiki).

Phase 2 Algorithm Flow

flowchart TD
    Start["extract_and_enhance_diagrams(repo, temp_dir, session, diagram_source_url)"]
Start --> FetchJS["response = session.get(diagram_source_url)\nhtml_text = response.text"]
FetchJS --> ExtractRegex["pattern = r'```mermaid(?:\\r\\\n/\\/\r?\)(.*?)(?:\\r\\\n/\\/\r?\)```'\ndiagram_matches = re.finditer(pattern, html_text, re.DOTALL)"]
ExtractRegex --> CountTotal["print(f'Found {len(diagram_matches)}
total diagrams')"]
CountTotal --> ExtractContext["for match in diagram_matches:\ncontext_start = max(0, match.start() - 2000)\ncontext_before = html_text[context_start:match.start()]"]
ExtractContext --> Unescape["Unescape escape sequences:\nreplace('\\\n', '\\n')\nreplace('\\	', '\	')\nreplace('\\\"', '\"')\nreplace('\<', '<')\nreplace('\>', '>')"]
Unescape --> ParseContext["context_lines = [l for l in context.split('\\n')
if l.strip()]\nFind last_heading (line starting with #)\nExtract anchor_text (last 2-3 non-heading lines, max 300 chars)"]
ParseContext --> Normalize["normalize_mermaid_diagram(diagram)\n7 normalization passes:\nnormalize_mermaid_edge_labels()\nnormalize_mermaid_state_descriptions()\nnormalize_flowchart_nodes()\nnormalize_statement_separators()\nnormalize_empty_node_labels()\nnormalize_gantt_diagram()"]
Normalize --> BuildContexts["diagram_contexts.append({\n 'last_heading': last_heading,\n 'anchor_text': anchor_text[-300:],\n 'diagram': normalized_diagram\n})"]
BuildContexts --> ScanFiles["md_files = list(temp_dir.glob('**/*.md'))\nfor md_file in md_files:]
    
 
   ScanFiles --> SkipExisting{re.search(r'^\s*`{3,}\s*mermaid\b',\ncontent)?"}
SkipExisting -->|Yes| ScanFiles
 
   SkipExisting -->|No| NormalizeContent["content_normalized = content.lower()\ncontent_normalized = ' '.join(content_normalized.split())"]
NormalizeContent --> MatchLoop["for idx, item in enumerate(diagram_contexts):]
 MatchLoop --> TryChunks[for chunk_size in [300, 200, 150, 100, 80]:\ntest_chunk = anchor_normalized[-chunk_size:]\npos = content_normalized.find(test_chunk)"]
TryChunks --> FoundMatch{"pos != -1?"}
FoundMatch -->|Yes| ConvertToLine["char_count = 0\nfor line_num, line in enumerate(lines):\n char_count += len(' '.join(line.split())) + 1\n if char_count >= pos: best_match_line = line_num"]
FoundMatch -->|No| TryHeading["for line_num, line in enumerate(lines):\nif heading_normalized in line_normalized:\n best_match_line = line_num"]
TryHeading --> FoundMatch2{"best_match_line != -1?"}
FoundMatch2 -->|Yes| ConvertToLine
 
   FoundMatch2 -->|No| MatchLoop
    
 
   ConvertToLine --> CheckScore{"best_match_score >= 80?"}
CheckScore -->|Yes| FindInsert["insert_line = best_match_line + 1\nSkip blank lines, skip paragraph/list"]
CheckScore -->|No| MatchLoop
    
 
   FindInsert --> QueueInsert["pending_insertions.append(\n (insert_line, diagram, score, idx)\n)\ndiagrams_used.add(idx)"]
QueueInsert --> MatchLoop
    
 
   MatchLoop --> SortInsert["pending_insertions.sort(key=lambda x: x[0], reverse=True)"]
SortInsert --> InsertLoop["for insert_line, diagram, score, idx in pending_insertions:\nlines.insert(insert_line, '')\nlines.insert(insert_line, f'{fence}mermaid')\nlines.insert(insert_line, diagram)\nlines.insert(insert_line, fence)\nlines.insert(insert_line, '')"]
InsertLoop --> WriteFile["with open(md_file, 'w')
as f:\n f.write('\\n'.join(lines))"]
WriteFile --> ScanFiles

Sources: python/deepwiki-scraper.py:880-1276 python/deepwiki-scraper.py:899-1088 python/deepwiki-scraper.py:1149-1273 python/deepwiki-scraper.py:230-393

Fuzzy Matching Algorithm

The algorithm uses progressively shorter anchor text chunks to find the best match location for each diagram. The score threshold of 80 ensures only high-confidence matches are inserted.

Sources: python/deepwiki-scraper.py:1184-1218

flowchart LR
    AnchorText["anchor_text\n(last 300 chars from context)"]
AnchorText --> NormalizeA["anchor_normalized = anchor.lower()\nanchor_normalized = ' '.join(anchor_normalized.split())"]
MDFile["markdown file content"]
MDFile --> NormalizeC["content_normalized = content.lower()\ncontent_normalized = ' '.join(content_normalized.split())"]
NormalizeA --> Loop["for chunk_size in [300, 200, 150, 100, 80]:]
 NormalizeC --> Loop
 Loop --> Extract[if len(anchor_normalized) >= chunk_size:\n test_chunk = anchor_normalized[-chunk_size:]"]
Extract --> Find["pos = content_normalized.find(test_chunk)"]
Find --> FoundPos{"pos != -1?"}
FoundPos -->|Yes| CharToLine["char_count = 0\nfor line_num, line in enumerate(lines):\n char_count += len(' '.join(line.split())) + 1\n if char_count >= pos:\n best_match_line = line_num\n best_match_score = chunk_size"]
FoundPos -->|No| Loop
    
 
   CharToLine --> CheckThresh{"best_match_score >= 80?"}
CheckThresh -->|Yes| Accept["Accept match\nQueue for insertion"]
CheckThresh -->|No| HeadingFallback["Try heading_normalized in line_normalized\nbest_match_score = 50"]
HeadingFallback --> CheckThresh2{"best_match_score >= 80?"}
CheckThresh2 -->|Yes| Accept
 
   CheckThresh2 -->|No| Reject["Reject match\nSkip diagram"]

Diagram Extraction from JavaScript

Diagrams are extracted from the Next.js JavaScript payload embedded in the HTML response. DeepWiki stores all diagrams for all pages in a single JavaScript bundle, which requires fuzzy matching to place each diagram in the correct file.

Extraction Method

The primary extraction pattern captures fenced Mermaid blocks with various newline representations:

Unescape Sequence

Each diagram block undergoes unescaping to convert JavaScript string literals to actual text:

Sources: python/deepwiki-scraper.py:899-901 python/deepwiki-scraper.py:1039-1047

Phase 3: mdBook Build

Phase 3 generates mdBook configuration, creates the table of contents, and builds the final HTML documentation. This phase is orchestrated by build-docs.sh and invokes Rust tools (mdbook, mdbook-mermaid).

Phase 3 Component Interactions

flowchart TD
    Entry["build-docs.sh line 95\nPhase 3 starts"]
Entry --> MkdirBook["mkdir -p $BOOK_DIR\ncd $BOOK_DIR\n$BOOK_DIR=/workspace/book"]
MkdirBook --> GenToml["cat > book.toml <<EOF\n[book]\ntitle = \"$BOOK_TITLE\"\nauthors = [\"$BOOK_AUTHORS\"]\n[output.html]\ndefault-theme = \"rust\"\ngit-repository-url = \"$GIT_REPO_URL\"\n[preprocessor.mermaid]\ncommand = \"mdbook-mermaid\"\n[output.html.fold]\nenable = true"]
GenToml --> MkdirSrc["mkdir -p src"]
MkdirSrc --> InitSummary["{ echo '# Summary'; echo ''; } > src/SUMMARY.md"]
InitSummary --> FindOverview["main_pages_list=$(ls $WIKI_DIR/*.md)\noverview_file=$(printf '%s\ ' $main_pages_list / grep -Ev '^[0-9]' / head -1)\ntitle=$(head -1 $WIKI_DIR/$overview_file /sed 's/^# //'"]
FindOverview --> WriteOverview["echo \"[$title] $overview_file \" >> src/SUMMARY.md echo '' >> src/SUMMARY.md"]
WriteOverview --> SortPages["main_pages=$ printf '%s\ ' $main_pages_list/ awk -F/ '{print $NF}'\n /grep -E '^[0-9]'/ sort -t- -k1 -n)"]
SortPages --> LoopPages["echo \"$main_pages\" |while read -r file; do"]
LoopPages --> ExtractTitle["filename=$ basename \"$file\" title=$ head -1 \"$file\"| sed 's/^# //')"]
ExtractTitle --> ExtractNum["section_num=$(echo \"$filename\" |grep -oE '^[0-9]+' "]
ExtractNum --> CheckSubdir{"[ -d \"$WIKI_DIR/section-$section_num\" ]?"}
CheckSubdir -->|Yes|WriteMain["echo \"- [$title] $filename \" >> src/SUMMARY.md"]
WriteMain --> LoopSubs["ls $WIKI_DIR/section-$section_num/*.md/ awk -F/ '{print $NF}'\n /sort -t- -k1 -n/ while read subname; do"]
LoopSubs --> WriteSub["subfile=\"section-$section_num/$subname\"\nsubtitle=$(head -1 \"$subfile\" |sed 's/^# //' echo \" - [$subtitle] $subfile \" >> src/SUMMARY.md"]
WriteSub --> LoopSubs
 CheckSubdir -->|No| WriteStandalone["echo \"- [$title]($filename)\" >> src/SUMMARY.md"]
LoopSubs --> LoopPages
 
   WriteStandalone --> LoopPages
    
 
   LoopPages --> CopySrc["cp -r $WIKI_DIR/* src/"]
CopySrc --> ProcessTemplates["python3 /usr/local/bin/process-template.py $HEADER_TEMPLATE\npython3 /usr/local/bin/process-template.py $FOOTER_TEMPLATE\nInject into src/*.md and src/*/*.md"]
ProcessTemplates --> MermaidInstall["mdbook-mermaid install $BOOK_DIR"]
MermaidInstall --> MdBookBuild["mdbook build\nReads book.toml and src/SUMMARY.md\nProcesses src/*.md files\nGenerates book/index.html"]
MdBookBuild --> CopyOut["mkdir -p $OUTPUT_DIR\ncp -r book $OUTPUT_DIR/\ncp -r $WIKI_DIR/. $OUTPUT_DIR/markdown/\ncp book.toml $OUTPUT_DIR/"]

Sources: scripts/build-docs.sh:95-309 scripts/build-docs.sh:124-188 scripts/build-docs.sh:190-261 scripts/build-docs.sh:263-271

book.toml Generation

The orchestrator dynamically generates book.toml with runtime configuration from environment variables:

Sources: scripts/build-docs.sh:102-119

flowchart TD
    Start["Generate src/SUMMARY.md\n{ echo '# Summary'; echo ''; } > src/SUMMARY.md"]
Start --> ListFiles["main_pages_list=$(ls $WIKI_DIR/*.md 2>/dev/null // true)"]
ListFiles --> FindOverview["overview_file=$(printf '%s\ ' $main_pages_list\n /awk -F/ '{print $NF}'/ grep -Ev '^[0-9]'\n /head -1"]
FindOverview --> WriteOverview["if [ -n \"$overview_file\" ]; then title=$ head -1 $WIKI_DIR/$overview_file| sed 's/^# //')\n  echo \"[${title:-Overview}]($overview_file)\" >> src/SUMMARY.md\n  echo '' >> src/SUMMARY.md\nfi"]
WriteOverview --> FilterMain["main_pages=$(printf '%s\ ' $main_pages_list\n /awk -F/ '{print $NF}'/ grep -E '^[0-9]'\n /sort -t- -k1 -n"]
FilterMain --> LoopMain["echo \"$main_pages\"| while read -r file; do"]
LoopMain --> CheckFile{"[ -f \"$file\" ]?"}
CheckFile -->|Yes| GetFilename["filename=$(basename \"$file\")\ntitle=$(head -1 \"$file\" |sed 's/^# //' "]
CheckFile -->|No|LoopMain
 GetFilename --> ExtractNum["section_num=$ echo \"$filename\"| grep -oE '^[0-9]+' || true)\nsection_dir=\"$WIKI_DIR/section-$section_num\""]
ExtractNum --> CheckSubdir{"[ -n \"$section_num\" ] &&\n[ -d \"$section_dir\" ]?"}
CheckSubdir -->|Yes| WriteMainWithSubs["echo \"- [$title]($filename)\" >> src/SUMMARY.md"]
WriteMainWithSubs --> ListSubs["ls $section_dir/*.md 2>/dev/null\n /awk -F/ '{print $NF}'/ sort -t- -k1 -n"]
ListSubs --> LoopSubs["while read subname; do"]
LoopSubs --> CheckSubFile{"[ -f \"$subfile\" ]?"}
CheckSubFile -->|Yes| WriteSub["subfile=\"$section_dir/$subname\"\nsubfilename=$(basename \"$subfile\")\nsubtitle=$(head -1 \"$subfile\" |sed 's/^# //' echo \" - [$subtitle] section-$section_num/$subfilename \" >> src/SUMMARY.md"]
CheckSubFile -->|No|LoopSubs
 WriteSub --> LoopSubs
 CheckSubdir -->|No| WriteStandalone["echo \"- [$title]($filename)\" >> src/SUMMARY.md"]
LoopSubs --> LoopMain
 
   WriteStandalone --> LoopMain
    
 
   LoopMain --> CountEntries["echo \"Generated SUMMARY.md with $(grep -c '\\[' src/SUMMARY.md)
entries\""]

SUMMARY.md Generation Algorithm

The table of contents is generated by scanning the actual file structure in /workspace/wiki and extracting titles from the first line of each file:

Sources: scripts/build-docs.sh:124-188

mdBook and mdbook-mermaid Execution

The build process invokes two Rust binaries installed via the Docker multi-stage build:

mdbook Build Process:

The mdbook build command performs the following operations:

1. Parse Structure : Read src/SUMMARY.md to determine page hierarchy and navigation order
2. Process Files : For each .md file referenced in SUMMARY.md:
   • Parse Markdown with CommonMark parser
   • Process Mermaid fenced code blocks via mdbook-mermaid preprocessor
   • Apply rust theme styles (configurable via default-theme in book.toml)
   • Generate sidebar navigation
3. Generate HTML : Create HTML files with:
   • Responsive navigation sidebar
   • Client-side search functionality (elasticlunr.js)
   • “Edit this page” links using git-repository-url from book.toml
   • Syntax highlighting for code blocks
4. Copy Assets : Bundle theme assets, fonts, and JavaScript libraries

Sources: scripts/build-docs.sh:263-271 scripts/build-docs.sh:102-119

Data Transformation Summary

Each phase transforms data in specific ways, with temporary directories used for intermediate work:

Working Directories:

Final Output Structure:

/output/
├── book/                          # Static HTML site (from /workspace/book/book/)
│   ├── index.html
│   ├── overview.html               # First page (unnumbered)
│   ├── 1-quick-start.html         # Main pages
│   ├── section-1/
│   │   ├── 1-1-installation.html  # Subsections
│   │   └── ...
│   ├── mermaid.min.js             # Installed by mdbook-mermaid
│   ├── mermaid-init.js            # Installed by mdbook-mermaid
│   └── ...
├── markdown/                       # Enhanced Markdown source (from /workspace/wiki/)
│   ├── overview.md
│   ├── 1-quick-start.md
│   ├── section-1/
│   │   ├── 1-1-installation.md
│   │   └── ...
│   └── ...
├── raw_markdown/                   # Pre-enhancement snapshot (from /workspace/raw_markdown/)
│   ├── overview.md                 # Same structure as markdown/ but without diagrams
│   └── ...
└── book.toml                      # mdBook configuration (from /workspace/book/book.toml)

Sources: scripts/build-docs.sh:273-294 python/deepwiki-scraper.py:1358-1366

flowchart TD
    Phase1["Phase 1: Extraction\n(deepwiki-scraper.py)"]
Phase2["Phase 2: Enhancement\n(deepwiki-scraper.py)"]
Phase1 --> Phase2
    
 
   Phase2 --> Check{"MARKDOWN_ONLY\n== true?"}
Check -->|Yes| FastPath["cp -r /workspace/wiki/* /output/markdown/\nExit (fast path)"]
Check -->|No| Phase3["Phase 3: mdBook Build\n(build-docs.sh)"]
Phase3 --> FullOutput["Copy book/ and markdown/ to /output/\nExit (full build)"]
FastPath --> End[/"Build complete"/]
 
   FullOutput --> End

Conditional Execution: MARKDOWN_ONLY Mode

The MARKDOWN_ONLY environment variable allows bypassing Phase 3 for faster iteration during development:

When MARKDOWN_ONLY=true:

• Execution time: ~30-60 seconds (scraping + diagram matching only)
• Output: /output/markdown/ only
• Use case: Debugging diagram placement, testing content extraction

When MARKDOWN_ONLY=false (default):

• Execution time: ~60-120 seconds (full pipeline)
• Output: /output/book/, /output/markdown/, /output/book.toml
• Use case: Production documentation builds

Sources: build-docs.sh:60-76 README.md:55-76

Dismiss

Refresh this wiki

Enter email to refresh

Docker Multi-Stage Build

Loading…

Docker Multi-Stage Build

Relevant source files

• Dockerfile

Purpose and Scope

This document explains the Docker multi-stage build strategy used to create the deepwiki-scraper container image. It details how the system combines a Rust toolchain for compiling documentation tools with a Python runtime for web scraping, while optimizing the final image size.

For information about how the container orchestrates the build process, see build-docs.sh Orchestrator. For details on the Python scraper implementation, see deepwiki-scraper.py.

Multi-Stage Build Strategy

The Dockerfile implements a two-stage build pattern that separates compilation from runtime. Stage 1 uses a full Rust development environment to compile mdBook binaries from source. Stage 2 creates a minimal Python runtime and extracts only the compiled binaries, discarding the build toolchain.

Build Stages Flow

graph TD
    subgraph Stage1["Stage 1: Rust Builder (rust:latest)"]
RustBase["rust:latest base image\n~1.5 GB with toolchain"]
CargoInstall["cargo install mdbook\ncargo install mdbook-mermaid"]
Binaries["/usr/local/cargo/bin/\nmdbook\nmdbook-mermaid"]
RustBase --> CargoInstall
 
       CargoInstall --> Binaries
    end
    
    subgraph Stage2["Stage 2: Python Runtime (python:3.12-slim)"]
PyBase["python:3.12-slim base\n~150 MB"]
UVInstall["Copy uv from\nghcr.io/astral-sh/uv:latest"]
PipInstall["uv pip install --system\nrequirements.txt"]
CopyBinaries["COPY --from=builder\n/usr/local/cargo/bin/"]
CopyScripts["COPY tools/ and\nbuild-docs.sh"]
PyBase --> UVInstall
 
       UVInstall --> PipInstall
 
       PipInstall --> CopyBinaries
 
       CopyBinaries --> CopyScripts
    end
    
 
   Binaries -.->|Extract only binaries| CopyBinaries
    
 
   CopyScripts --> FinalImage["Final Image\n~300-400 MB"]
Stage1 -.->|Discarded after build| Discard["Discard"]
style RustBase fill:#f5f5f5
    style PyBase fill:#f5f5f5
    style FinalImage fill:#e8e8e8
    style Discard fill:#fff,stroke-dasharray: 5 5

Sources: Dockerfile:1-33

Stage 1: Rust Builder

Stage 1 uses rust:latest as the base image, providing the complete Rust toolchain including cargo, the Rust package manager and build tool.

Rust Builder Configuration

The cargo install commands fetch mdBook and mdbook-mermaid source from crates.io, compile them from source, and install the resulting binaries to /usr/local/cargo/bin/.

flowchart LR
    subgraph BuilderStage["builder stage"]
CratesIO["crates.io\n(package registry)"]
CargoFetch["cargo fetch\n(download sources)"]
CargoCompile["cargo build --release\n(compile to binary)"]
CargoInstallBin["Install to\n/usr/local/cargo/bin/"]
CratesIO --> CargoFetch
 
       CargoFetch --> CargoCompile
 
       CargoCompile --> CargoInstallBin
    end
    
 
   CargoInstallBin --> MdBookBin["mdbook binary"]
CargoInstallBin --> MermaidBin["mdbook-mermaid binary"]
MdBookBin -.->|Copied to Stage 2| NextStage["NextStage"]
MermaidBin -.->|Copied to Stage 2| NextStage

Sources: Dockerfile:1-5

Stage 2: Python Runtime Assembly

Stage 2 builds the final runtime image starting from python:3.12-slim, a minimal Python base image that omits development headers and unnecessary packages.

Python Runtime Components

graph TB
    subgraph PythonBase["python:3.12-slim"]
PyInterpreter["Python 3.12 interpreter"]
PyStdlib["Python standard library"]
BaseUtils["Essential utilities\n(bash, sh, coreutils)"]
end
    
    subgraph InstalledTools["Installed via COPY"]
UV["uv package manager\n/bin/uv, /bin/uvx"]
PyDeps["Python packages\n(requests, beautifulsoup4, html2text)"]
RustBins["Rust binaries\n(mdbook, mdbook-mermaid)"]
Scripts["Application scripts\n(deepwiki-scraper.py, build-docs.sh)"]
end
    
 
   PythonBase --> UV
 
   UV --> PyDeps
 
   PythonBase --> RustBins
 
   PythonBase --> Scripts
    
 
   PyDeps --> Runtime["Runtime Environment"]
RustBins --> Runtime
 
   Scripts --> Runtime

The installation sequence follows a specific order:

1. Copy uv Dockerfile13 - Multi-stage copy from ghcr.io/astral-sh/uv:latest
2. Install Python dependencies Dockerfile:16-17 - Uses uv pip install --system --no-cache
3. Copy Rust binaries Dockerfile:20-21 - Extracts from builder stage
4. Copy application scripts Dockerfile:24-29 - Adds Python scraper and orchestrator

Sources: Dockerfile:8-29

Binary Extraction and Integration

The critical optimization occurs at Dockerfile:20-21 where the COPY --from=builder directive extracts only the compiled binaries without any build dependencies.

Binary Extraction Pattern

flowchart LR
    subgraph BuilderFS["Builder Filesystem"]
CargoDir["/usr/local/cargo/bin/"]
MdBookSrc["mdbook\n(compiled binary)"]
MermaidSrc["mdbook-mermaid\n(compiled binary)"]
CargoDir --> MdBookSrc
 
       CargoDir --> MermaidSrc
    end
    
    subgraph RuntimeFS["Runtime Filesystem"]
BinDir["/usr/local/bin/"]
MdBookDst["mdbook\n(extracted)"]
MermaidDst["mdbook-mermaid\n(extracted)"]
BinDir --> MdBookDst
 
       BinDir --> MermaidDst
    end
    
 
   MdBookSrc -.->|COPY --from=builder| MdBookDst
 
   MermaidSrc -.->|COPY --from=builder| MermaidDst
    
    subgraph Discarded["Discarded (not copied)"]
RustToolchain["rustc compiler"]
CargoTool["cargo build tool"]
SourceFiles["mdBook source files"]
BuildCache["cargo build cache"]
end

Both binaries are statically linked or contain all necessary Rust runtime dependencies, allowing them to execute in the Python base image without the Rust toolchain.

Sources: Dockerfile:19-21

Python Dependency Installation

Python dependencies are installed using uv, a fast Python package installer written in Rust. The dependencies are defined in tools/requirements.txt:1-4

Python Dependencies

The installation command Dockerfile17 uses these flags:

• --system: Install to system Python (not virtualenv)
• --no-cache: Don’t cache downloaded packages (reduces image size)
• -r /tmp/requirements.txt: Read dependencies from file

Sources: Dockerfile:16-17 tools/requirements.txt:1-4

graph LR
    subgraph Approach1["Single-Stage Approach (Hypothetical)"]
Single["rust:latest + Python\n~2+ GB"]
end
    
    subgraph Approach2["Multi-Stage Approach (Actual)"]
Builder["Stage 1: rust:latest\n~1.5 GB\n(discarded)"]
Runtime["Stage 2: python:3.12-slim\n+ binaries + dependencies\n~300-400 MB"]
Builder -.->|Extract binaries only| Runtime
    end
    
 
   Single -->|Contains unnecessary build toolchain| Waste["Wasted Space"]
Runtime -->|Contains only runtime essentials| Efficient["Efficient"]
style Single fill:#f5f5f5
    style Builder fill:#f5f5f5
    style Runtime fill:#e8e8e8
    style Waste fill:#fff,stroke-dasharray: 5 5
    style Efficient fill:#fff,stroke-dasharray: 5 5

Image Size Optimization

The multi-stage strategy achieves significant size reduction by discarding the build environment.

Size Comparison

Size Breakdown of Final Image

Sources: Dockerfile:1-33 README.md156

graph TB
    subgraph Filesystem["/usr/local/bin/"]
BuildScript["build-docs.sh\n(orchestrator)"]
ScraperScript["deepwiki-scraper.py\n(Python scraper)"]
MdBookBin["mdbook\n(Rust binary)"]
MermaidBin["mdbook-mermaid\n(Rust binary)"]
UVBin["uv\n(Python installer)"]
end
    
    subgraph SystemPython["/usr/local/lib/python3.12/"]
Requests["requests package"]
BS4["beautifulsoup4 package"]
Html2Text["html2text package"]
end
    
    subgraph Execution["Execution Flow"]
Docker["docker run\n(CMD)"]
Docker --> BuildScript
 
       BuildScript -->|python| ScraperScript
 
       BuildScript -->|subprocess| MdBookBin
 
       MdBookBin -->|preprocessor| MermaidBin
 
       ScraperScript --> Requests
 
       ScraperScript --> BS4
 
       ScraperScript --> Html2Text
    end

Runtime Environment Structure

The final image contains a hybrid Python-Rust runtime where Python scripts can execute Rust binaries as subprocesses.

Runtime Component Locations

The entrypoint Dockerfile32 executes /usr/local/bin/build-docs.sh, which orchestrates calls to both Python and Rust components. The script can execute:

• python /usr/local/bin/deepwiki-scraper.py for web scraping
• mdbook init for initialization
• mdbook build for HTML generation
• mdbook-mermaid install for asset installation

Sources: Dockerfile:28-32 build-docs.sh

Container Execution Model

When the container runs, Docker executes the CMD Dockerfile32 which invokes build-docs.sh. This shell script has access to all binaries in /usr/local/bin/ (automatically on $PATH).

Process Tree During Execution

graph TD
    Docker["docker run\n(container init)"]
Docker --> CMD["CMD: build-docs.sh"]
CMD --> Phase1["Phase 1:\npython deepwiki-scraper.py"]
CMD --> Phase2["Phase 2: mdbook init"]
CMD --> Phase3["Phase 3: mdbook-mermaid install"]
CMD --> Phase4["Phase 4: mdbook build"]
Phase1 --> PyProc["Python 3.12 process"]
PyProc --> ReqLib["requests.get()"]
PyProc --> BS4Lib["BeautifulSoup()"]
PyProc --> H2TLib["html2text.HTML2Text()"]
Phase2 --> MdBookProc1["mdbook binary process"]
Phase3 --> MermaidProc["mdbook-mermaid binary process"]
Phase4 --> MdBookProc2["mdbook binary process"]
MdBookProc2 --> MermaidPreproc["mdbook-mermaid\n(as preprocessor)"]

Sources: Dockerfile32 README.md:122-145

Dismiss

Refresh this wiki

Enter email to refresh

Component Reference

Loading…

Component Reference

Relevant source files

• README.md
• scripts/build-docs.sh

Purpose and Scope

This page provides a high-level overview of the major components in the DeepWiki-to-mdBook converter system and their responsibilities. Each component is introduced with its primary function, key files, and relationships to other components.

For detailed information about specific components:

• Shell orchestration logic: see build-docs.sh Orchestrator
• Content extraction and diagram processing: see deepwiki-scraper.py
• Header and footer customization: see Template System
• Final HTML generation: see mdBook Integration

System Component Map

The following diagram shows all major components and their organizational relationships:

Sources: scripts/build-docs.sh:1-310 README.md:84-88

graph TB
    subgraph "Entry Points"
        Dockerfile["Dockerfile\n(multi-stage build)"]
ActionYAML["action.yml\n(GitHub Action)"]
end
    
    subgraph "Orchestration Layer"
        BuildScript["build-docs.sh\n(main orchestrator)"]
end
    
    subgraph "Python Components"
        Scraper["deepwiki-scraper.py\n(content extraction)"]
TemplateProc["process-template.py\n(template rendering)"]
end
    
    subgraph "Build Tools"
        mdBook["mdbook\n(HTML generator)"]
mdBookMermaid["mdbook-mermaid\n(diagram renderer)"]
end
    
    subgraph "Configuration Assets"
        HeaderTemplate["templates/header.html"]
FooterTemplate["templates/footer.html"]
BookToml["book.toml\n(generated)"]
SummaryMd["SUMMARY.md\n(generated)"]
end
    
    subgraph "Data Directories"
        WikiDir["/workspace/wiki\n(enhanced markdown)"]
RawDir["/workspace/raw_markdown\n(pre-enhancement)"]
BookSrc["/workspace/book/src\n(mdBook input)"]
OutputDir["/output\n(final artifacts)"]
end
    
 
   Dockerfile -->|builds| BuildScript
 
   Dockerfile -->|installs| Scraper
 
   Dockerfile -->|installs| TemplateProc
 
   Dockerfile -->|compiles| mdBook
 
   Dockerfile -->|compiles| mdBookMermaid
 
   ActionYAML -->|invokes| Dockerfile
    
 
   BuildScript -->|executes| Scraper
 
   BuildScript -->|executes| TemplateProc
 
   BuildScript -->|executes| mdBook
 
   BuildScript -->|executes| mdBookMermaid
 
   BuildScript -->|generates| BookToml
 
   BuildScript -->|generates| SummaryMd
    
 
   Scraper -->|writes to| WikiDir
 
   Scraper -->|writes to| RawDir
 
   TemplateProc -->|reads| HeaderTemplate
 
   TemplateProc -->|reads| FooterTemplate
 
   TemplateProc -->|outputs HTML| BuildScript
    
 
   BuildScript -->|copies| WikiDir
 
   WikiDir -->|to| BookSrc
 
   BuildScript -->|injects templates into| BookSrc
    
 
   mdBook -->|reads| BookToml
 
   mdBook -->|reads| SummaryMd
 
   mdBook -->|reads| BookSrc
 
   mdBook -->|builds to| OutputDir
 
   mdBookMermaid -->|preprocesses| BookSrc

Core Components

build-docs.sh

Type: Shell script orchestrator
Location: scripts/build-docs.sh:1-310
Entry Point: Docker container CMD instruction

The main orchestration script that coordinates the entire build process. It performs seven sequential steps:

Key Responsibilities:

• Environment variable validation and default assignment
• Git repository auto-detection from remote URLs
• Orchestrating execution order of Python scripts
• Dynamic SUMMARY.md generation with numeric sorting
• Template injection into all markdown files
• Output directory management

Sources: scripts/build-docs.sh:1-310

deepwiki-scraper.py

Type: Python script
Location: python/deepwiki-scraper.py
Invocation: scripts/build-docs.sh65

The content extraction component that scrapes DeepWiki wiki pages and converts them to markdown with embedded diagrams.

Key Responsibilities:

• Fetch wiki HTML from https://deepwiki.com/{REPO}
• Parse Next.js data payload to discover wiki structure
• Convert HTML to markdown using html2text library
• Extract Mermaid diagrams from JavaScript payload
• Normalize diagrams for Mermaid 11 compatibility (7-step pipeline)
• Match diagrams to pages using fuzzy text matching
• Write enhanced markdown to /workspace/wiki
• Write pre-enhancement snapshot to /workspace/raw_markdown

The scraper is covered in detail in deepwiki-scraper.py.

Sources: scripts/build-docs.sh65 README.md74

process-template.py

Type: Python script
Location: python/process-template.py
Invocation: scripts/build-docs.sh:205-213 scripts/build-docs.sh:222-230

A template rendering utility that processes HTML template files with variable substitution.

Key Responsibilities:

• Read template file from path argument
• Parse variable assignments from command-line arguments (format: KEY=value)
• Substitute {{VARIABLE}} placeholders with values
• Handle conditional rendering with {{#if VARIABLE}}...{{/if}} blocks
• Output processed HTML to stdout

Template Variables Supported:

• REPO - Repository identifier (e.g., “owner/repo”)
• BOOK_TITLE - Documentation title
• BOOK_AUTHORS - Author names
• GIT_REPO_URL - Full GitHub repository URL
• DEEPWIKI_URL - DeepWiki page URL
• DEEPWIKI_BADGE_URL - Badge image URL
• GITHUB_BADGE_URL - GitHub badge URL
• GENERATION_DATE - Build timestamp

See Template System for comprehensive documentation.

Sources: scripts/build-docs.sh:195-234 README.md51

Template Files

Type: HTML configuration files
Location: templates/header.html, templates/footer.html
Default Path: /workspace/templates/
Custom Mount: -v "$(pwd)/my-templates:/workspace/templates"

Static HTML template files that are processed by process-template.py and injected into every markdown file.

File Responsibilities:

Injection Logic:

[Header HTML]
<blank line>
[Original Markdown Content]
<blank line>
[Footer HTML]

Templates are injected at scripts/build-docs.sh:240-261 after all markdown files are copied to the book source directory.

Sources: scripts/build-docs.sh:195-234 scripts/build-docs.sh:240-261 README.md:39-51

mdBook and mdbook-mermaid

Type: External build tools (Rust binaries)
Location: /usr/local/bin/mdbook, /usr/local/bin/mdbook-mermaid
Compilation: Dockerfile multi-stage build

Pre-compiled tools that generate the final HTML output.

mdBook Responsibilities:

• Read configuration from book.toml scripts/build-docs.sh:102-119
• Parse SUMMARY.md to build navigation structure
• Convert markdown files to HTML with search index
• Apply theme (rust theme by default)
• Generate table of contents sidebar
• Create chapter navigation links

mdbook-mermaid Responsibilities:

• Act as mdBook preprocessor scripts/build-docs.sh:113-114
• Detect mermaid code blocks in markdown
• Install JavaScript rendering libraries scripts/build-docs.sh266
• Configure client-side diagram rendering

See mdBook Integration for detailed integration documentation.

Sources: scripts/build-docs.sh:113-114 scripts/build-docs.sh266 scripts/build-docs.sh271

sequenceDiagram
    participant Docker
    participant BuildScript as "build-docs.sh"
    participant Scraper as "deepwiki-scraper.py"
    participant TemplateProc as "process-template.py"
    participant mdBook as "mdbook"
    participant FileSystem as "/output"
    
    Docker->>BuildScript: Execute CMD
    
    BuildScript->>BuildScript: Validate REPO env var
    BuildScript->>BuildScript: Auto-detect from git remote
    BuildScript->>BuildScript: Set defaults (BOOK_AUTHORS, etc)
    
    BuildScript->>Scraper: Execute with REPO arg
    Scraper->>Scraper: Fetch DeepWiki HTML
    Scraper->>Scraper: Extract wiki structure
    Scraper->>Scraper: Convert HTML to markdown
    Scraper->>Scraper: Process Mermaid diagrams
    Scraper->>FileSystem: Write /workspace/wiki/*.md
    Scraper->>FileSystem: Write /workspace/raw_markdown/*.md
    Scraper-->>BuildScript: Exit 0
    
    alt MARKDOWN_ONLY=true
        BuildScript->>FileSystem: Copy markdown to /output
        BuildScript->>Docker: Exit 0
    end
    
    BuildScript->>BuildScript: Create /workspace/book/
    BuildScript->>BuildScript: Generate book.toml
    BuildScript->>BuildScript: Scan wiki files
    BuildScript->>BuildScript: Generate SUMMARY.md
    
    BuildScript->>TemplateProc: Process header.html
    TemplateProc-->>BuildScript: Return HTML string
    BuildScript->>TemplateProc: Process footer.html
    TemplateProc-->>BuildScript: Return HTML string
    
    BuildScript->>BuildScript: Copy wiki/*.md to book/src/
    BuildScript->>BuildScript: Inject header into each .md
    BuildScript->>BuildScript: Inject footer into each .md
    
    BuildScript->>mdBook: mdbook-mermaid install
    BuildScript->>mdBook: mdbook build
    mdBook->>mdBook: Parse SUMMARY.md
    mdBook->>mdBook: Convert markdown to HTML
    mdBook->>mdBook: Build search index
    mdBook->>FileSystem: Write /workspace/book/book/
    mdBook-->>BuildScript: Exit 0
    
    BuildScript->>FileSystem: Copy book/ to /output/book/
    BuildScript->>FileSystem: Copy wiki/ to /output/markdown/
    BuildScript->>FileSystem: Copy raw_markdown/ to /output/raw_markdown/
    BuildScript->>FileSystem: Copy book.toml to /output/
    
    BuildScript-->>Docker: Exit 0

Component Execution Flow

This diagram shows the runtime execution sequence and data flow between components:

Sources: scripts/build-docs.sh:1-310

File System Organization

The following table maps logical component names to their physical locations in the Docker container and output directory:

Sources: scripts/build-docs.sh:27-31 scripts/build-docs.sh:274-294

graph TD
    subgraph "Docker Build Stage 1"
        RustBase["rust:latest base image"]
CargoInstall["cargo install"]
RustBase --> CargoInstall
 
       CargoInstall --> mdBookBin["mdbook binary"]
CargoInstall --> mdBookMermaidBin["mdbook-mermaid binary"]
end
    
    subgraph "Docker Build Stage 2"
        PythonBase["python:3.12-slim base image"]
PipInstall["pip install"]
PythonBase --> PipInstall
 
       PipInstall --> RequestsLib["requests library"]
PipInstall --> Html2TextLib["html2text library"]
PipInstall --> RapidFuzzLib["rapidfuzz library"]
end
    
    subgraph "Runtime Dependencies"
        BuildScript["build-docs.sh"]
ScraperPy["deepwiki-scraper.py"]
TemplatePy["process-template.py"]
BuildScript --> ScraperPy
 
       BuildScript --> TemplatePy
 
       BuildScript --> mdBookBin
 
       BuildScript --> mdBookMermaidBin
        
 
       ScraperPy --> RequestsLib
 
       ScraperPy --> Html2TextLib
 
       ScraperPy --> RapidFuzzLib
    end
    
    subgraph "Environment Inputs"
        EnvREPO["REPO env var"]
EnvBOOK_TITLE["BOOK_TITLE env var"]
EnvMARKDOWN_ONLY["MARKDOWN_ONLY env var"]
GitRemote["git remote origin"]
GitRemote -.fallback.-> EnvREPO
 
       EnvREPO --> BuildScript
 
       EnvBOOK_TITLE --> BuildScript
 
       EnvMARKDOWN_ONLY --> BuildScript
    end
    
    mdBookBin -.copied from.-> RustBase
    mdBookMermaidBin -.copied from.-> RustBase

Component Dependencies

This diagram maps the dependency relationships between components, showing which components require which other components:

Sources: scripts/build-docs.sh:8-19 README.md:14-27

Component Communication Patterns

Inter-Process Communication

All component communication uses standard Unix patterns:

Sources: scripts/build-docs.sh2 scripts/build-docs.sh65 scripts/build-docs.sh:205-213

File System Communication

Components communicate via shared file system locations:

Sources: scripts/build-docs.sh:27-31 scripts/build-docs.sh237 scripts/build-docs.sh:274-294

Configuration Communication

Environment variables flow unidirectionally from the container entry point to all components:

Sources: scripts/build-docs.sh:8-60 scripts/build-docs.sh:200-230

Component Responsibilities Matrix

The following table summarizes what each component is and is not responsible for:

Sources: scripts/build-docs.sh:1-310 README.md:72-77

Dismiss

Refresh this wiki

Enter email to refresh

build-docs.sh Orchestrator

Loading…

build-docs.sh Orchestrator

Relevant source files

• scripts/build-docs.sh

Purpose and Scope

The build-docs.sh script is the main orchestration layer for the DeepWiki-to-mdBook conversion system. It coordinates all components of the three-phase pipeline, manages configuration, handles environment variable processing, and produces the final output artifacts. This document covers the script’s responsibilities, execution flow, configuration management, and integration with other system components.

For details on the components orchestrated by this script, see deepwiki-scraper.py, Template System, and mdBook Integration. For information on the three-phase architecture, see Three-Phase Pipeline.

Role and Responsibilities

The orchestrator serves as the single entry point for the documentation build process. It is invoked as the Docker container’s default command and coordinates all system components in a sequential, deterministic manner.

Key Responsibilities:

Sources: scripts/build-docs.sh:1-310

Architecture Overview

The following diagram maps the orchestrator’s workflow to actual code entities and directory paths used in the script:

Diagram: Orchestrator Component Integration

graph TB
    Entry["Entry Point\nbuild-docs.sh"]
subgraph "Configuration Phase"
        AutoDetect["Git Auto-detection\nlines 8-19"]
EnvVars["Environment Variables\nREPO, BOOK_TITLE, etc.\nlines 21-26"]
Defaults["Default Generation\nBOOK_AUTHORS, GIT_REPO_URL\nlines 44-51"]
Validate["Validation\nREPO check\nlines 33-38"]
end
    
    subgraph "Execution Phase"
        Step1["Step 1: deepwiki-scraper.py\n$REPO → $WIKI_DIR\nlines 61-65"]
Decision{{"MARKDOWN_ONLY\n?\nlines 68"}}
MarkdownExit["Copy to /output/markdown\nlines 69-93"]
Step2["Step 2: mkdir $BOOK_DIR\nCreate book.toml\nlines 95-119"]
Step3["Step 3: Generate SUMMARY.md\nDiscover structure\nlines 124-188"]
Step4["Step 4: process-template.py\nInject headers/footers\nlines 190-261"]
Step5["Step 5: mdbook-mermaid install\nlines 263-266"]
Step6["Step 6: mdbook build\nlines 268-271"]
Step7["Step 7: Copy to /output\nlines 273-295"]
end
    
 
   Entry --> AutoDetect
 
   AutoDetect --> EnvVars
 
   EnvVars --> Defaults
 
   Defaults --> Validate
 
   Validate --> Step1
 
   Step1 --> Decision
 
   Decision -->|true| MarkdownExit
 
   Decision -->|false| Step2
 
   Step2 --> Step3
 
   Step3 --> Step4
 
   Step4 --> Step5
 
   Step5 --> Step6
 
   Step6 --> Step7
    
 
   MarkdownExit --> OutputMarkdown["/output/markdown/"]
MarkdownExit --> OutputRaw["/output/raw_markdown/"]
Step7 --> OutputBook["/output/book/"]
Step7 --> OutputMarkdown
 
   Step7 --> OutputRaw
 
   Step7 --> OutputConfig["/output/book.toml"]

Sources: scripts/build-docs.sh:1-310

Configuration Management

The script implements a sophisticated configuration system with automatic detection, environment variable overrides, and sensible defaults.

Auto-Detection Logic

The script attempts to automatically detect the repository from Git metadata if REPO is not explicitly set:

Diagram: Repository Auto-Detection Flow

flowchart TD
    Start["Check if REPO\nenvironment variable set"]
Start -->|Not set| CheckGit["Check if .git directory exists\ngit rev-parse --git-dir"]
Start -->|Set| UseProvided["Use provided REPO value"]
CheckGit -->|Yes| GetRemote["Get remote.origin.url\ngit config --get remote.origin.url"]
CheckGit -->|No| RequireManual["Require manual REPO setting"]
GetRemote -->|Found| ExtractOwnerRepo["Extract owner/repo using sed\nPattern: github.com[:/]owner/repo"]
GetRemote -->|Not found| RequireManual
    
 
   ExtractOwnerRepo --> SetRepo["Set REPO variable"]
UseProvided --> SetRepo
 
   SetRepo --> Validate["Validate REPO is not empty"]
RequireManual --> Validate
    
 
   Validate -->|Empty| Error["Exit with error\nlines 34-37"]
Validate -->|Valid| Continue["Continue execution"]

The regex pattern at scripts/build-docs.sh16 handles multiple GitHub URL formats:

• https://github.com/owner/repo.git
• git@github.com:owner/repo.git
• https://github.com/owner/repo

Sources: scripts/build-docs.sh:8-19 scripts/build-docs.sh:33-38

Configuration Variables

The following table documents all configuration variables managed by the orchestrator:

Computed variables derived from $REPO:

Sources: scripts/build-docs.sh:21-51

sequenceDiagram
    participant Script as build-docs.sh
    participant Scraper as deepwiki-scraper.py
    participant FileSystem as File System
    participant Templates as process-template.py
    participant MDBook as mdbook
    participant Mermaid as mdbook-mermaid
    
    Note over Script: Configuration Phase
    Script->>Script: Auto-detect REPO
    Script->>Script: Set defaults
    Script->>Script: Validate configuration
    
    Note over Script: Step 1: Scraping
    Script->>FileSystem: rm -rf $RAW_DIR
    Script->>Scraper: python3 deepwiki-scraper.py $REPO $WIKI_DIR
    Scraper-->>FileSystem: Write markdown to $WIKI_DIR
    Scraper-->>FileSystem: Write raw snapshots to $RAW_DIR
    
    alt MARKDOWN_ONLY == true
        Note over Script: Markdown-Only Exit Path
        Script->>FileSystem: cp $WIKI_DIR to /output/markdown
        Script->>FileSystem: cp $RAW_DIR to /output/raw_markdown
        Script->>Script: Exit (skip HTML build)
    else MARKDOWN_ONLY == false
        Note over Script: Step 2: mdBook Initialization
        Script->>FileSystem: mkdir -p $BOOK_DIR/src
        Script->>FileSystem: Create book.toml
        
        Note over Script: Step 3: SUMMARY.md Generation
        Script->>FileSystem: Scan $WIKI_DIR for .md files
        Script->>FileSystem: Generate src/SUMMARY.md
        
        Note over Script: Step 4: Template Processing
        Script->>Templates: process-template.py $HEADER_TEMPLATE
        Templates-->>Script: Processed HEADER_HTML
        Script->>Templates: process-template.py $FOOTER_TEMPLATE
        Templates-->>Script: Processed FOOTER_HTML
        Script->>FileSystem: cp $WIKI_DIR/* to src/
        Script->>FileSystem: Inject header/footer into all .md files
        
        Note over Script: Step 5: Mermaid Installation
        Script->>Mermaid: mdbook-mermaid install $BOOK_DIR
        Mermaid-->>FileSystem: Install mermaid.js assets
        
        Note over Script: Step 6: Build
        Script->>MDBook: mdbook build
        MDBook-->>FileSystem: Generate book/ directory
        
        Note over Script: Step 7: Output Collection
        Script->>FileSystem: cp book to /output/book
        Script->>FileSystem: cp $WIKI_DIR to /output/markdown
        Script->>FileSystem: cp $RAW_DIR to /output/raw_markdown
        Script->>FileSystem: cp book.toml to /output/book.toml
    end

Execution Flow

The orchestrator follows a seven-step execution sequence, with conditional branching for markdown-only mode:

Diagram: Step-by-Step Execution Sequence

Sources: scripts/build-docs.sh:61-310

Step Details

Step 1: Wiki Scraping

Lines: scripts/build-docs.sh:61-65

Invokes the Python scraper to fetch and convert DeepWiki content:

The scraper writes output to two locations:

• $WIKI_DIR (/workspace/wiki): Enhanced markdown with injected diagrams
• $RAW_DIR (/workspace/raw_markdown): Pre-enhancement markdown snapshots for debugging

For details on the scraper’s operation, see deepwiki-scraper.py.

Sources: scripts/build-docs.sh:61-65

Step 2: mdBook Structure Initialization

Lines: scripts/build-docs.sh:95-119

Skipped if: MARKDOWN_ONLY=true

Creates the mdBook directory structure and generates book.toml configuration:

$BOOK_DIR/
├── book.toml
└── src/

The book.toml file is generated using a heredoc with variable substitution:

Sources: scripts/build-docs.sh:95-119

Step 3: SUMMARY.md Generation

Lines: scripts/build-docs.sh:124-188

flowchart TD
    Start["Start SUMMARY.md Generation"]
Start --> WriteHeader["Write '# Summary' header"]
WriteHeader --> FindOverview["Find overview file\ngrep -Ev '^[0-9]'"]
FindOverview -->|Found| WriteOverview["Write overview entry\nExtract title from first line"]
FindOverview -->|Not found| ListMain
 
   WriteOverview --> ListMain["List all main pages\nls $WIKI_DIR/*.md"]
ListMain --> FilterOverview["Filter out overview file"]
FilterOverview --> NumericSort["Sort numerically\nsort -t- -k1 -n"]
NumericSort --> ProcessLoop["For each file"]
ProcessLoop --> ExtractTitle["Extract title\nhead -1 /sed 's/^# //'"]
ExtractTitle --> GetSectionNum["Extract section number grep -oE '^[0-9]+'"]
GetSectionNum --> CheckSubdir{"Subsection directory section-N exists?"}
CheckSubdir -->|Yes|WriteSection["Write section entry - [title] filename"]
WriteSection --> ListSubs["List subsection files ls section-N/*.md"]
ListSubs --> SortSubs["Sort numerically sort -t- -k1 -n"]
SortSubs --> WriteSubLoop["For each subsection: - [subtitle] section-N/file"]
WriteSubLoop --> NextFile
 CheckSubdir -->|No|WriteStandalone["Write standalone entry - [title] filename"]
WriteStandalone --> NextFile{"More files?"}
NextFile -->|Yes|ProcessLoop
 NextFile -->|No| Complete["Complete src/SUMMARY.md"]

Dynamically generates the table of contents by discovering the file structure in $WIKI_DIR. This step implements numeric sorting and hierarchical organization.

Diagram: SUMMARY.md Generation Algorithm

Key implementation details:

Overview Page Detection: scripts/build-docs.sh:136-144

• Searches for files without numeric prefix
• Typically matches Overview.md or similar

Numeric Sorting: scripts/build-docs.sh:147-155

• Uses sort -t- -k1 -n to sort by numeric prefix
• Handles formats like 1-Title.md, 2.1-Subtopic.md

Hierarchy Detection: scripts/build-docs.sh:165-180

• Checks for section-N/ directories for each numeric section
• Creates indented entries for subsections

Sources: scripts/build-docs.sh:124-188

Step 4: Template Processing and File Copying

Lines: scripts/build-docs.sh:190-261

flowchart LR
    subgraph "Template Loading"
        HeaderPath["$HEADER_TEMPLATE\n/workspace/templates/header.html"]
FooterPath["$FOOTER_TEMPLATE\n/workspace/templates/footer.html"]
GenDate["GENERATION_DATE\ndate -u command"]
end
    
    subgraph "Variable Substitution"
        ProcessH["process-template.py\n$HEADER_TEMPLATE"]
ProcessF["process-template.py\n$FOOTER_TEMPLATE"]
Vars["Variables passed:\nDEEPWIKI_URL\nDEEPWIKI_BADGE_URL\nGIT_REPO_URL\nGITHUB_BADGE_URL\nREPO\nBOOK_TITLE\nBOOK_AUTHORS\nGENERATION_DATE"]
end
    
    subgraph "Injection"
        CopyFiles["cp $WIKI_DIR/* to src/"]
InjectLoop["For each .md file:\nsrc/*.md src/*/*.md"]
CreateTemp["Create temp file:\nHEADER + content + FOOTER"]
Replace["mv temp to original"]
end
    
 
   HeaderPath --> ProcessH
 
   FooterPath --> ProcessF
 
   GenDate --> Vars
 
   Vars --> ProcessH
 
   Vars --> ProcessF
 
   ProcessH --> HeaderHTML["HEADER_HTML variable"]
ProcessF --> FooterHTML["FOOTER_HTML variable"]
CopyFiles --> InjectLoop
 
   HeaderHTML --> InjectLoop
 
   FooterHTML --> InjectLoop
 
   InjectLoop --> CreateTemp
 
   CreateTemp --> Replace

Processes header and footer templates and injects them into all markdown files.

Template Processing Flow:

Diagram: Template Processing Pipeline

The template processor is invoked with all configuration variables as arguments: scripts/build-docs.sh:205-213 scripts/build-docs.sh:222-230

File injection pattern: scripts/build-docs.sh:243-257

• Processes all .md files in src/ and src/*/
• Creates temporary file with header + original content + footer
• Replaces original with modified version

For details on the template system and variable substitution, see Template System.

Sources: scripts/build-docs.sh:190-261

Step 5: Mermaid Installation

Lines: scripts/build-docs.sh:263-266

Installs mdbook-mermaid preprocessor assets into the book directory:

This command installs the mermaid.js library and initialization code required for client-side diagram rendering in the final HTML output.

Sources: scripts/build-docs.sh:263-266

Step 6: Book Build

Lines: scripts/build-docs.sh:268-271

Executes the mdBook build process:

Build Process:

1. Reads book.toml configuration
2. Processes src/SUMMARY.md to determine structure
3. Applies mermaid preprocessor to all markdown files
4. Converts markdown to HTML with search indexing
5. Outputs to $BOOK_DIR/book/ directory

For more information on mdBook integration, see mdBook Integration.

Sources: scripts/build-docs.sh:268-271

Step 7: Output Collection

Lines: scripts/build-docs.sh:273-295

Copies all build artifacts to the /output volume mount for persistence:

The script ensures clean output by removing existing directories before copying: scripts/build-docs.sh:282-290

Sources: scripts/build-docs.sh:273-295

Markdown-Only Mode

When MARKDOWN_ONLY=true, the orchestrator follows a shortened execution path that skips HTML generation:

Execution Path:

1. Step 1: Scrape wiki (normal)
2. Copy $WIKI_DIR to /output/markdown/
3. Copy $RAW_DIR to /output/raw_markdown/ (if exists)
4. Exit with success

Use Cases:

• Debugging the scraper output without full build overhead
• Extracting markdown for alternative processing pipelines
• CI/CD test workflows that only validate markdown generation
• Custom post-processing before HTML generation

Implementation: scripts/build-docs.sh:68-93

Sources: scripts/build-docs.sh:68-93

Error Handling

The orchestrator implements fail-fast error handling:

Error Handling Mechanisms:

The script does not use explicit error trapping; instead, it relies on Bash’s set -e behavior to immediately exit if any command returns a non-zero status. This ensures that failures in any component (scraper, template processor, mdBook) halt execution and propagate to the Docker container exit code.

Sources: scripts/build-docs.sh2 scripts/build-docs.sh:33-38 scripts/build-docs.sh:215-216 scripts/build-docs.sh:232-233

graph TB
    Orchestrator["build-docs.sh"]
subgraph "Python Components"
        Scraper["deepwiki-scraper.py\nArgs: REPO, WIKI_DIR"]
Templates["process-template.py\nArgs: template_path, var1=val1, ..."]
end
    
    subgraph "Build Tools"
        MDBook["mdbook build\nWorking dir: $BOOK_DIR"]
Mermaid["mdbook-mermaid install\nArgs: $BOOK_DIR"]
end
    
    subgraph "File System"
        Input["Input:\n/workspace/templates/"]
Working["Working:\n$WIKI_DIR\n$RAW_DIR\n$BOOK_DIR"]
Output["Output:\n/output/"]
end
    
    subgraph "Environment"
        EnvVars["Environment Variables:\nREPO, BOOK_TITLE,\nBOOK_AUTHORS, etc."]
end
    
 
   EnvVars --> Orchestrator
 
   Input --> Templates
    
 
   Orchestrator -->|python3| Scraper
 
   Orchestrator -->|python3| Templates
 
   Orchestrator -->|mdbook| MDBook
 
   Orchestrator -->|mdbook-mermaid| Mermaid
    
 
   Scraper --> Working
 
   Templates --> Orchestrator
 
   Orchestrator --> Working
 
   MDBook --> Working
 
   Mermaid --> Working
    
 
   Orchestrator --> Output

Integration Points

The orchestrator integrates with multiple system components through well-defined interfaces:

Diagram: Component Integration Interfaces

Interface Specifications:

deepwiki-scraper.py:

• Invocation: python3 /usr/local/bin/deepwiki-scraper.py $REPO $WIKI_DIR
• Input: Repository identifier (e.g., "jzombie/deepwiki-to-mdbook")
• Output: Markdown files in $WIKI_DIR, raw snapshots in $RAW_DIR
• Documentation: deepwiki-scraper.py

process-template.py:

• Invocation: python3 /usr/local/bin/process-template.py $TEMPLATE_PATH var1=val1 var2=val2 ...
• Input: Template file path and variable assignments
• Output: Processed HTML string to stdout
• Documentation: Template System

mdbook:

• Invocation: mdbook build (in $BOOK_DIR)
• Input: book.toml and src/ directory structure
• Output: HTML in book/ subdirectory
• Documentation: mdBook Integration

mdbook-mermaid:

• Invocation: mdbook-mermaid install $BOOK_DIR
• Input: Book directory path
• Output: Mermaid assets installed in book directory
• Documentation: mdBook Integration

Sources: scripts/build-docs.sh65 scripts/build-docs.sh:205-213 scripts/build-docs.sh:222-230 scripts/build-docs.sh266 scripts/build-docs.sh271

Output Artifacts

The orchestrator produces a structured output directory with multiple artifact types:

/output/
├── book/                    # Searchable HTML documentation (Step 7)
│   ├── index.html
│   ├── searchindex.js
│   ├── mermaid.min.js
│   └── ...
├── markdown/                # Enhanced markdown files (Step 7 or markdown-only)
│   ├── Overview.md
│   ├── 1-First-Section.md
│   ├── section-1/
│   │   └── 1.1-Subsection.md
│   └── ...
├── raw_markdown/            # Pre-enhancement snapshots (if available)
│   ├── Overview.md
│   ├── 1-First-Section.md
│   └── ...
└── book.toml                # Book configuration reference (Step 7)

Artifact Generation Timeline:

Sources: scripts/build-docs.sh:273-295

Dismiss

Refresh this wiki

Enter email to refresh

deepwiki-scraper.py

Loading…

deepwiki-scraper.py

Relevant source files

• python/deepwiki-scraper.py

Purpose and Scope

The deepwiki-scraper.py script is the primary data extraction and transformation component that converts DeepWiki wiki content into enhanced markdown files. It orchestrates a three-phase pipeline: (1) extracting clean markdown from DeepWiki HTML, (2) enhancing files with normalized Mermaid diagrams using fuzzy matching, and (3) moving completed files to the output directory.

This page documents the script’s architecture, execution model, and key algorithms. For information about how this script is invoked by the build system, see build-docs.sh Orchestrator. For detailed explanations of the extraction and enhancement phases, see Phase 1: Markdown Extraction and Phase 2: Diagram Enhancement.

Sources: python/deepwiki-scraper.py:1-11

Command-Line Interface

The script requires two positional arguments:

• Repository identifier : Format owner/repo (e.g., jzombie/deepwiki-to-mdbook)
• Output directory : Destination path for generated markdown files

The repository identifier is validated using the regex pattern ^[\w-]+/[\w-]+$ at python/deepwiki-scraper.py:1287-1289 The script exits with status code 1 if validation fails or if the wiki structure cannot be extracted.

Sources: python/deepwiki-scraper.py:1-10 python/deepwiki-scraper.py:1277-1289

Three-Phase Execution Model

Figure 1: Three-Phase Execution Pipeline

The main() function at python/deepwiki-scraper.py:1277-1410 implements a three-phase workflow:

A temporary directory is created at python/deepwiki-scraper.py:1295-1296 using Python’s tempfile.TemporaryDirectory context manager. This ensures automatic cleanup even if the script fails. A raw markdown snapshot is saved to raw_markdown/ at python/deepwiki-scraper.py:1358-1366 before diagram enhancement for debugging purposes.

Sources: python/deepwiki-scraper.py:1277-1410 python/deepwiki-scraper.py:1298-1371

Wiki Structure Discovery

Figure 2: Structure Discovery Algorithm Using extract_wiki_structure

The extract_wiki_structure function at python/deepwiki-scraper.py:116-163 discovers all wiki pages by parsing the main wiki index page. It uses a compiled regex pattern to find all links matching ^/{repo_pattern}/\d+ at python/deepwiki-scraper.py:128-129

The page numbering scheme distinguishes main pages from subsections using dot notation:

• Level 0 : Main pages (e.g., 1, 2, 3) - pages with no dots
• Level 1 : Subsections (e.g., 2.1, 2.2) - pages with one dot
• Level N : Deeper subsections (e.g., 2.1.3) - pages with N dots

The level is calculated at python/deepwiki-scraper.py145 as page_num.count('.'). Pages are sorted using a custom key function at python/deepwiki-scraper.py:157-159 that splits the page number by dots and converts each component to an integer, ensuring proper numerical ordering (e.g., 2.10 comes after 2.9).

Each page dictionary contains:

• number: Page number string (e.g., "2.1")
• title: Extracted link text
• url: Full URL to the page
• href: Relative path (used for link rewriting)
• level: Nesting depth based on dot count

Sources: python/deepwiki-scraper.py:116-163 python/deepwiki-scraper.py145 python/deepwiki-scraper.py:157-161

Path Resolution and Numbering Normalization

Figure 3: Path Resolution Using normalized_number_parts and resolve_output_path

The path resolution system normalizes DeepWiki’s numbering scheme to match mdBook’s conventions. The normalized_number_parts function at python/deepwiki-scraper.py:28-43 shifts page numbers down by one so that DeepWiki’s page 1 becomes unnumbered (the index page), and subsequent pages start at 1.

The resolve_output_path function at python/deepwiki-scraper.py:45-53 combines normalized numbers with sanitized titles. Subsections (with len(parts) > 1) are placed in directories named section-{main_number} at python/deepwiki-scraper.py52 The sanitize_filename function at python/deepwiki-scraper.py:22-26 strips special characters and normalizes whitespace using regex patterns r'[^\w\s-]' and r'[-\s]+'.

The build_target_path function at python/deepwiki-scraper.py:55-63 constructs full relative paths for link rewriting, used by the link fixing logic at python/deepwiki-scraper.py:854-875

Sources: python/deepwiki-scraper.py:28-63 python/deepwiki-scraper.py:22-26 python/deepwiki-scraper.py:854-875

Content Extraction and HTML-to-Markdown Conversion

Figure 4: Content Extraction Pipeline Using extract_page_content

The extract_page_content function at python/deepwiki-scraper.py:751-877 implements a multi-stage HTML cleaning and conversion pipeline. BeautifulSoup selectors at python/deepwiki-scraper.py:761-762 remove navigation elements before content extraction.

The content finder at python/deepwiki-scraper.py:765-779 tries a prioritized list of selectors: article, main, .wiki-content, .content, #content, .markdown-body, and finally falls back to body. DeepWiki-specific UI elements are removed at python/deepwiki-scraper.py:786-795 by searching for text patterns like “Index your code with Devin” and “Edit Wiki”.

Navigation list removal at python/deepwiki-scraper.py:799-806 detects and removes <ul> elements containing more than 5 links where 80%+ are internal wiki links.

The convert_html_to_markdown function at python/deepwiki-scraper.py:213-228 uses the html2text library with configuration:

• ignore_links = False - preserve all links
• body_width = 0 - disable line wrapping to prevent formatting issues

Note at python/deepwiki-scraper.py:221-223 explicitly documents that Mermaid diagram processing is disabled during HTML conversion because diagrams from ALL pages are mixed together in the JavaScript payload.

The clean_deepwiki_footer function at python/deepwiki-scraper.py:165-211 removes DeepWiki UI elements using compiled regex patterns for text like “Dismiss”, “Refresh this wiki”, and “On this page”. It scans backwards from the end of the file up to 50 lines to find footer markers at python/deepwiki-scraper.py:187-191

Link rewriting at python/deepwiki-scraper.py:854-875 converts DeepWiki URLs to relative markdown paths, handling both same-section and cross-section references by calculating relative paths based on the source file’s section directory.

Sources: python/deepwiki-scraper.py:751-877 python/deepwiki-scraper.py:213-228 python/deepwiki-scraper.py:165-211 python/deepwiki-scraper.py:395-406

Diagram Extraction from JavaScript Payload

Figure 5: Diagram Extraction Using extract_and_enhance_diagrams

The extract_and_enhance_diagrams function at python/deepwiki-scraper.py:880-1275 extracts all Mermaid diagrams from DeepWiki’s Next.js JavaScript payload. The regex pattern at python/deepwiki-scraper.py899 matches fenced code blocks with various newline formats: \\r\\n, \\n, or actual newline characters.

Context extraction at python/deepwiki-scraper.py:903-1087 captures up to 2000 characters before each diagram to enable fuzzy matching. For each diagram, the context is parsed to extract:

1. Last heading : The most recent line starting with # (searched backwards from diagram position)
2. Anchor text : The last 2-3 non-heading lines exceeding 20 characters in length, concatenated and truncated to 300 characters

The context extraction logic at python/deepwiki-scraper.py:1066-1081 searches backwards through context lines to find the last heading, then collects up to 3 substantial non-heading lines as anchor text.

The unescaping phase at python/deepwiki-scraper.py:1039-1046 handles JavaScript string escapes:

The merge_multiline_labels function at python/deepwiki-scraper.py:907-1009 collapses wrapped Mermaid labels into literal \n sequences. This is crucial because DeepWiki sometimes wraps long labels across multiple lines in the HTML, but Mermaid 11 expects these to be explicitly marked with \n tokens.

Sources: python/deepwiki-scraper.py:880-1087 python/deepwiki-scraper.py899 python/deepwiki-scraper.py:1039-1046 python/deepwiki-scraper.py:907-1009

Seven-Step Mermaid Normalization Pipeline

Figure 6: Seven-Step Normalization Pipeline Using normalize_mermaid_diagram

The normalize_mermaid_diagram function at python/deepwiki-scraper.py:385-393 applies seven normalization passes to ensure Mermaid 11 compatibility:

Step 1: normalize_mermaid_edge_labels

Function at python/deepwiki-scraper.py:230-251 Applies only to graphs and flowcharts (detected by checking if first line starts with graph or flowchart). Uses regex r'\|([^|]*)\|' to find edge labels and flattens any containing \n, \\n, (, or ) by:

• Replacing \\n and \n with spaces
• Removing parentheses
• Collapsing whitespace with re.sub(r'\s+', ' ', cleaned).strip()

Step 2: normalize_mermaid_state_descriptions

Function at python/deepwiki-scraper.py:253-277 Applies only to state diagrams. Ensures state descriptions use the syntax State : Description by:

• Skipping lines with :: (already valid)
• Splitting on single : and cleaning suffix
• Replacing colons in description with -
• Rebuilding as {prefix.rstrip()} : {cleaned_suffix}

Step 3: normalize_flowchart_nodes

Function at python/deepwiki-scraper.py:279-301 Applies to graphs and flowcharts. Uses regex r'\["([^"]*)"\]' to find node labels and:

• Replaces pipe characters with forward slashes
• Collapses whitespace
• Inserts newlines between consecutive statements using regex at python/deepwiki-scraper.py:298-299

Step 4: normalize_statement_separators

Function at python/deepwiki-scraper.py:313-328 Applies to graphs and flowcharts. The STATEMENT_BREAK_PATTERN at python/deepwiki-scraper.py:309-311 detects consecutive statements on one line and inserts newlines between them while preserving indentation.

Step 5: normalize_empty_node_labels

Function at python/deepwiki-scraper.py:330-341 Uses regex r'(\b[A-Za-z0-9_]+)\[""\]' to find nodes with empty labels. Generates fallback label from node ID by replacing underscores/hyphens with spaces.

Step 6: normalize_gantt_diagram

Function at python/deepwiki-scraper.py:343-383 Applies only to Gantt diagrams. Detects task lines missing IDs using pattern r'^(\s*"[^"]+"\s*):\s*(.+)$' and inserts synthetic IDs (task1, task2, etc.) when the first token after the colon is not an ID or after reference.

Sources: python/deepwiki-scraper.py:385-393 python/deepwiki-scraper.py:230-251 python/deepwiki-scraper.py:253-277 python/deepwiki-scraper.py:279-301 python/deepwiki-scraper.py:313-328 python/deepwiki-scraper.py:330-341 python/deepwiki-scraper.py:343-383

Fuzzy Matching and Diagram Injection

Figure 7: Fuzzy Matching Algorithm for Diagram Injection

The fuzzy matching algorithm at python/deepwiki-scraper.py:1150-1275 pairs each diagram with its correct markdown file by matching context against file contents. The algorithm uses a progressive chunk size strategy at python/deepwiki-scraper.py1188 to find matches:

The matching loop at python/deepwiki-scraper.py:1170-1238 attempts anchor text matching first. The anchor text (last 2-3 lines of context before the diagram) is normalized to lowercase with whitespace collapsed at python/deepwiki-scraper.py:1185-1186 For each chunk size, the algorithm searches for the test chunk at the end of the anchor text (anchor_normalized[-chunk_size:]) in the normalized file content.

If anchor matching fails (score < 80), the algorithm falls back to heading matching at python/deepwiki-scraper.py:1204-1216 This compares the last_heading from diagram context against all headings in the file after normalizing both by removing # symbols and collapsing whitespace.

Only matches with best_match_score >= 80 are accepted at python/deepwiki-scraper.py1218 This threshold balances precision (avoiding false matches) with recall (ensuring most diagrams are placed).

Insertion Point Logic

The insertion point finder at python/deepwiki-scraper.py:1220-1236 behaves differently based on match type:

After heading match :

1. Skip blank lines after heading
2. Skip through the following paragraph
3. Insert after the paragraph ends (blank line or next heading)

After paragraph match :

1. Find end of current paragraph
2. Insert when encountering blank line or heading

Content Guards

The enforce_content_start function at python/deepwiki-scraper.py:1138-1147 and advance_past_lists function at python/deepwiki-scraper.py:1125-1137 implement content guards to prevent diagram insertion in protected areas:

Protected prefix (detected by protected_prefix_end at python/deepwiki-scraper.py:1101-1115):

• Title line (first line starting with #)
• “Relevant source files” section and its list items
• Blank lines in these sections

List blocks (detected by is_list_line at python/deepwiki-scraper.py:1117-1123):

• Lines starting with -, *, +
• Lines matching \d+[.)]\s (numbered lists)

Diagrams are never inserted inside list blocks. If the insertion point lands in a list, advance_past_lists moves the insertion point to after the list ends.

Dynamic Fence Length

The insertion logic at python/deepwiki-scraper.py:1249-1266 calculates a dynamic fence length to handle diagrams containing backticks. It scans the diagram text for the longest run of consecutive backticks and sets fence_len = max(3, max_backticks + 1). This ensures the fence markers (````mermaid`) always properly delimit the diagram content.

Sources: python/deepwiki-scraper.py:1150-1275 python/deepwiki-scraper.py:1170-1238 python/deepwiki-scraper.py:1101-1147 python/deepwiki-scraper.py:1249-1266

Error Handling and Retry Logic

Figure 8: Retry Logic in fetch_page Function

The fetch_page function at python/deepwiki-scraper.py:65-80 implements a 3-attempt retry strategy with exponential backoff. The retry loop at python/deepwiki-scraper.py:71-80 catches all exceptions using a bare except Exception as e clause and retries with a 2-second delay using time.sleep(2).

Browser-like headers are set at python/deepwiki-scraper.py:67-69 to avoid bot detection:

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

The timeout is set to 30 seconds at python/deepwiki-scraper.py73 After a successful fetch, response.raise_for_status() validates the HTTP status code.

The main extraction loop at python/deepwiki-scraper.py:1328-1353 catches exceptions per-page and continues processing remaining pages:

This ensures that a single page failure doesn’t abort the entire scraping process. The success count is reported at python/deepwiki-scraper.py1355

The top-level try-except block at python/deepwiki-scraper.py:1310-1407 catches any unhandled exceptions and exits with status code 1, signaling failure to the calling build script.

Sources: python/deepwiki-scraper.py:65-80 python/deepwiki-scraper.py:1328-1353 python/deepwiki-scraper.py:1310-1407

Session Management and Rate Limiting

The script uses a requests.Session object created at python/deepwiki-scraper.py:1305-1308 with persistent headers:

Session reuse provides connection pooling and persistent cookies across requests. The session is passed to all HTTP functions: extract_wiki_structure, extract_page_content, and extract_and_enhance_diagrams.

Rate limiting is implemented at python/deepwiki-scraper.py1350 with a 1-second sleep between page extractions:

This prevents overwhelming the DeepWiki server and reduces the risk of rate limiting or IP blocking. The comment at python/deepwiki-scraper.py1349 explicitly states “Be nice to the server”.

Sources: python/deepwiki-scraper.py:1305-1308 python/deepwiki-scraper.py:1349-1350

Key Function Reference

Sources: python/deepwiki-scraper.py:1-1411

Dismiss

Refresh this wiki

Enter email to refresh

Template System

Loading…

Template System

Relevant source files

• python/process-template.py
• templates/README.md
• templates/footer.html
• templates/header.html

The template system provides customizable header and footer content that is injected into every markdown file during the build process. This system uses a simple variable substitution syntax with conditional rendering support, allowing users to customize the appearance and metadata of generated documentation without modifying the core build scripts.

For information about how templates are injected during the build process, see Template Injection. For comprehensive documentation on template variables, see Template Variables.

Sources: templates/README.md:1-77

Template Files

The system uses two HTML template files that define content to be injected into each markdown file:

Both templates are processed through the same variable substitution engine before injection.

Sources: templates/README.md:6-8 templates/header.html:1-9 templates/footer.html:1-11

Default Header Template

The default header template displays project badges and attribution information:

The template uses inline conditionals to prevent mdBook from wrapping links in separate paragraph tags, which would break the styling.

Sources: templates/header.html:1-9

Default Footer Template

The default footer template displays generation metadata and repository information:

Sources: templates/footer.html:1-11

Template Syntax

The template system supports three syntactic features: variable substitution, conditional rendering, and HTML comments.

Variable Substitution

Variables use double-brace syntax: {{VARIABLE_NAME}}. The processor replaces these with the corresponding variable value, or an empty string if the variable is not defined.

Variable names must match the pattern \w+ (alphanumeric and underscore characters).

Sources: python/process-template.py:38-45 templates/README.md:12-15

Conditional Rendering

Conditional blocks use {{#if VARIABLE}}...{{/if}} syntax. The content between the tags is included only if the variable exists and is non-empty.

The conditional pattern matches \{\{#if\s+(\w+)\}\}(.*?)\{\{/if\}\} and evaluates whether the variable is truthy.

Sources: python/process-template.py:24-36 templates/README.md:17-22

HTML Comments

HTML comments are automatically stripped from the output during processing:

Sources: python/process-template.py:47-48 templates/README.md:24-28

Template Processing Engine

Diagram: Template Processing Flow

Sources: python/process-template.py:1-82

The process-template.py script implements a two-pass processing algorithm:

1. Conditional Processing (first pass): Evaluates {{#if}} blocks using regular expression matching and removes or includes content based on variable truthiness python/process-template.py:24-36
2. Variable Substitution (second pass): Replaces {{VAR}} placeholders with actual values python/process-template.py:38-45
3. Comment Removal (cleanup): Strips HTML comments from the final output python/process-template.py:47-48

Command-Line Interface

The script accepts a template file path and variable assignments in KEY=value format:

Arguments are parsed at python/process-template.py:66-70 where each KEY=value pair is split and stored in a dictionary for substitution.

Sources: python/process-template.py:53-82

Available Template Variables

The following variables are provided by the build system and available in all templates:

All variables are optional. If a variable is undefined, it is replaced with an empty string during substitution.

Sources: templates/README.md:30-39

Customization

Diagram: Template Customization Architecture

Sources: templates/README.md:42-56

Volume Mount Customization

Custom templates can be provided by mounting a local directory or individual files into the Docker container:

Mount entire template directory:

Mount individual template files:

Sources: templates/README.md:45-51

Environment Variable Customization

Template locations can be overridden via environment variables:

Example:

Sources: templates/README.md:53-56

Integration with Build Process

Diagram: Template System in Build Pipeline

Sources: templates/README.md:1-77 Diagram 2 from high-level system architecture

The template system is invoked during Phase 3 of the build pipeline, after markdown files have been generated and enhanced with diagrams. The build-docs.sh orchestrator script processes each template file through process-template.py, then injects the processed content into every markdown file before generating SUMMARY.md and running mdbook build.

Template processing occurs in the following sequence:

1. Template Resolution : Determine paths to header and footer templates using environment variables or defaults
2. Variable Collection : Gather all template variables from environment and runtime context
3. Template Processing : Invoke process-template.py for each template file with collected variables
4. Content Injection : Prepend processed header and append processed footer to each markdown file
5. Build Continuation : Proceed with SUMMARY.md generation and mdBook build

This design ensures that all documentation pages share consistent branding, navigation, and metadata without requiring manual edits to individual markdown files.

Sources: templates/README.md:1-77

Example Custom Templates

Minimal Header Example

Sources: templates/README.md:60-68

Custom Footer Example

Sources: templates/README.md:70-76

Conditional Badge Example

This example demonstrates conditional rendering of badges based on available repository and documentation URLs. If neither GIT_REPO_URL nor DEEPWIKI_URL is defined, the template produces no output.

Sources: templates/header.html:2-4 templates/README.md:17-22

Dismiss

Refresh this wiki

Enter email to refresh

mdBook Integration

Loading…

mdBook Integration

Relevant source files

• README.md
• scripts/build-docs.sh

Purpose and Scope

This page documents how the system integrates with mdBook and mdbook-mermaid to generate the final HTML documentation. It covers the configuration generation process, automatic table of contents creation, mermaid diagram support installation, and the build execution. For information about the overall three-phase pipeline, see Three-Phase Pipeline. For template injection specifics, see Template Injection.

mdBook and mdbook-mermaid Tools

The system uses two Rust-based tools compiled during the Docker build:

Both tools are compiled from source in the Rust builder stage of the Docker image and copied to the final Python-based runtime container for size optimization.

mdBook Build Pipeline

graph TB
    subgraph "Input_Preparation"
        WikiDir["/workspace/wiki/\nEnhanced markdown files"]
Templates["Processed templates\nHEADER_HTML, FOOTER_HTML"]
end
    
    subgraph "Configuration_Generation"
        BookToml["book.toml generation\n[scripts/build-docs.sh:102-119]"]
SummaryGen["SUMMARY.md generation\n[scripts/build-docs.sh:124-186]"]
EnvVars["Environment variables:\nBOOK_TITLE, BOOK_AUTHORS\nGIT_REPO_URL"]
end
    
    subgraph "Structure_Creation"
        BookDir["/workspace/book/\nmdBook project root"]
SrcDir["/workspace/book/src/\nMarkdown source"]
CopyFiles["Copy wiki files to src/\n[scripts/build-docs.sh:237]"]
InjectTemplates["Inject header/footer\n[scripts/build-docs.sh:239-261]"]
end
    
    subgraph "mdBook_Processing"
        MermaidInstall["mdbook-mermaid install\n[scripts/build-docs.sh:266]"]
MdBookBuild["mdbook build\n[scripts/build-docs.sh:271]"]
Preprocessor["mdbook-mermaid preprocessor\nConverts ```mermaid blocks"]
end
    
    subgraph "Output"
        BookHTML["/workspace/book/book/\nBuilt HTML documentation"]
OutputCopy["Copy to /output/book/\n[scripts/build-docs.sh:279]"]
end
    
 
   EnvVars --> BookToml
 
   EnvVars --> SummaryGen
 
   WikiDir --> CopyFiles
 
   Templates --> InjectTemplates
    
 
   BookToml --> BookDir
 
   SummaryGen --> SrcDir
 
   CopyFiles --> SrcDir
 
   InjectTemplates --> SrcDir
    
 
   BookDir --> MermaidInstall
 
   MermaidInstall --> MdBookBuild
 
   SrcDir --> MdBookBuild
 
   MdBookBuild --> Preprocessor
 
   Preprocessor --> BookHTML
 
   BookHTML --> OutputCopy

Sources: scripts/build-docs.sh:95-295

Configuration Generation (book.toml)

The book.toml configuration file is dynamically generated at scripts/build-docs.sh:102-119 using environment variables.

book.toml Structure

Sources: scripts/build-docs.sh:102-119

Configuration Sections

The git-repository-url setting automatically adds an “Edit on GitHub” button to the top-right of each page, pointing to the repository specified in $GIT_REPO_URL (defaults to https://github.com/$REPO).

Sources: scripts/build-docs.sh:102-119

SUMMARY.md Generation

The table of contents is automatically generated by analyzing the file structure in /workspace/wiki/. The algorithm at scripts/build-docs.sh:124-186 creates a hierarchical navigation structure.

SUMMARY.md Generation Algorithm

graph TB
    subgraph "File_Discovery"
        WikiDir["/workspace/wiki/"]
ListFiles["ls *.md\n[scripts/build-docs.sh:135]"]
FilterNumeric["grep -E '^[0-9]'\n[scripts/build-docs.sh:150]"]
NumericSort["sort -t- -k1 -n\n[scripts/build-docs.sh:151]"]
end
    
    subgraph "Overview_Detection"
        OverviewFile["Find non-numeric file\n[scripts/build-docs.sh:138]"]
ExtractTitle["head -1 /sed 's/^# //' [scripts/build-docs.sh:140]"]
WriteOverview["Write [Title] filename [scripts/build-docs.sh:141]"]
end subgraph "Main_Page_Processing" IteratePages["Iterate main pages [scripts/build-docs.sh:158-185]"]
ExtractPageTitle["Extract '# Title' from file [scripts/build-docs.sh:163]"]
CheckSubsections["Check section-N directory [scripts/build-docs.sh:166-169]"]
end subgraph "Subsection_Handling" ListSubsections["ls section-N/*.md [scripts/build-docs.sh:174]"]
SortSubsections["sort -t- -k1 -n [scripts/build-docs.sh:174]"]
ExtractSubTitle["Extract '# Title' from subfile [scripts/build-docs.sh:178]"]
WriteSubsection["Write ' - [Title] section-N/file ' [scripts/build-docs.sh:179]"]
end subgraph "Output" SummaryMd["/workspace/book/src/SUMMARY.md"]
end
 WikiDir --> ListFiles
 ListFiles --> OverviewFile
 OverviewFile --> ExtractTitle
 ExtractTitle --> WriteOverview
 ListFiles --> FilterNumeric
 FilterNumeric --> NumericSort
 NumericSort --> IteratePages
 IteratePages --> ExtractPageTitle
 ExtractPageTitle --> CheckSubsections
 CheckSubsections -->|Has subsections|ListSubsections
 ListSubsections --> SortSubsections
 SortSubsections --> ExtractSubTitle
 ExtractSubTitle --> WriteSubsection
 CheckSubsections -->|No subsections| WriteStandalone["Write '- [Title](file)'"]
WriteOverview --> SummaryMd
 
   WriteSubsection --> SummaryMd
 
   WriteStandalone --> SummaryMd

Sources: scripts/build-docs.sh:124-186

SUMMARY.md Structure

The generated SUMMARY.md follows this format:

Generation Logic

1. Overview Detection scripts/build-docs.sh:136-144: Finds the first non-numeric markdown file and writes it as the top-level overview link.

2. Main Page Sorting scripts/build-docs.sh:147-155: Extracts numeric prefixes (e.g., 1, 2, 10) and sorts numerically using sort -t- -k1 -n, ensuring 10-file.md comes after 2-file.md.

3. Subsection Detection scripts/build-docs.sh:166-180: For each main page with numeric prefix N, checks if directory section-N/ exists. If found, lists and sorts subsection files, writing them indented with two spaces.

4. Title Extraction scripts/build-docs.sh:163-178: Reads the first line of each markdown file and removes the # prefix using sed 's/^# //'.

Sources: scripts/build-docs.sh:124-186

mdbook-mermaid Installation

Before building the book, the mermaid preprocessor assets must be installed. This is performed by mdbook-mermaid install at scripts/build-docs.sh266

mdbook-mermaid Installation Process

Sources: scripts/build-docs.sh:263-266

The mdbook-mermaid install command:

1. Downloads the mermaid.js library (version compatible with Mermaid 11)
2. Creates initialization scripts to configure mermaid rendering
3. Adds CSS stylesheets for diagram theming
4. Modifies the HTML templates to include the necessary <script> tags

During the subsequent mdbook build, the mermaid preprocessor:

1. Detects code blocks with ````mermaid` fence
2. Wraps them in <pre class="mermaid"> tags
3. Leaves the mermaid syntax intact for client-side rendering

Sources: scripts/build-docs.sh:263-271

Build Execution

The actual mdBook build occurs at scripts/build-docs.sh271 with a simple mdbook build command.

mdBook Build Process

Sources: scripts/build-docs.sh:268-271

Build Steps

1. Configuration Parsing : mdBook reads book.toml to load title, authors, theme, and preprocessor configuration.

2. Chapter Loading : mdBook parses SUMMARY.md to determine navigation structure and file order.

3. Markdown Processing : Each markdown file is read from the src/ directory.

4. Preprocessor Execution : The mdbook-mermaid preprocessor transforms mermaid code blocks into render-ready HTML.

5. Theme Application : The rust theme provides CSS styling, JavaScript functionality, and HTML templates.

6. Search Index : mdBook generates a searchable index from all content, enabling the built-in search feature.

7. Output Generation : HTML files are written to the book/ subdirectory within the project root.

Sources: scripts/build-docs.sh:268-271

Output Structure

After the build completes, the system copies outputs to /output/ at scripts/build-docs.sh:274-295

Output Directory Structure

Sources: scripts/build-docs.sh:274-295

Output Artifacts

The book/ directory contains:

• index.html - Main entry point with navigation sidebar
• print.html - Single-page version for printing
• *.html - Individual chapter pages
• searchindex.json - Search index data
• searchindex.js - Search functionality
• CSS and JavaScript assets for theming and interactivity

Sources: scripts/build-docs.sh:274-295 README.md:53-58

Local Serving

The generated HTML can be served locally using Python’s built-in HTTP server, as documented in README.md26:

This serves the documentation at http://localhost:8000, providing:

• Full-text search functionality
• Interactive navigation sidebar
• Rendered Mermaid diagrams (client-side rendering)
• Theme switching (light/dark/rust/coal/navy/ayu)
• Print-friendly single-page view

Sources: README.md:26-29 scripts/build-docs.sh:307-308

Dismiss

Refresh this wiki

Enter email to refresh

Phase 1: Markdown Extraction

Loading…

Phase 1: Markdown Extraction

Relevant source files

• python/deepwiki-scraper.py
• scripts/build-docs.sh

This page documents Phase 1 of the three-phase processing pipeline, which handles the extraction and initial conversion of wiki content from DeepWiki.com into clean Markdown files. Phase 1 produces raw Markdown files in a temporary directory before diagram enhancement (Phase 2, see #7) and mdBook HTML generation (Phase 3, see #8).

For detailed information about specific sub-processes within Phase 1, see:

• Wiki structure discovery algorithm: #6.1
• HTML parsing and Markdown conversion: #6.2

Scope and Objectives

Phase 1 accomplishes the following:

1. Discover all wiki pages and their hierarchical structure from DeepWiki
2. Fetch HTML content for each page via HTTP requests
3. Parse HTML to extract main content and remove UI elements
4. Convert cleaned HTML to Markdown using html2text
5. Organize output files into a hierarchical directory structure
6. Save to a temporary directory for subsequent processing

This phase is implemented entirely in Python within deepwiki-scraper.py and operates independently of Phases 2 and 3.

Sources: README.md:121-128 tools/deepwiki-scraper.py:790-876

Phase 1 Execution Flow

The following diagram shows the complete execution flow of Phase 1, mapping high-level steps to specific functions in the codebase:

Sources: tools/deepwiki-scraper.py:790-876 tools/deepwiki-scraper.py:78-125 tools/deepwiki-scraper.py:453-594

flowchart TD
    Start["main()
Entry Point"]
CreateTemp["Create tempfile.TemporaryDirectory()"]
CreateSession["requests.Session()
with User-Agent"]
DiscoverPhase["Structure Discovery Phase"]
ExtractWiki["extract_wiki_structure(repo, session)"]
ParseLinks["BeautifulSoup: find_all('a', href=pattern)"]
SortPages["sort by page number (handle dots)"]
ExtractionPhase["Content Extraction Phase"]
LoopPages["For each page in pages list"]
FetchContent["extract_page_content(url, session, page_info)"]
FetchHTML["fetch_page(url, session)
with retries"]
ParseHTML["BeautifulSoup(response.text)"]
RemoveNav["Remove nav/header/footer/aside elements"]
FindContent["Find main content: article/main/[role='main']"]
ConvertPhase["Conversion Phase"]
ConvertMD["convert_html_to_markdown(html_content)"]
HTML2Text["html2text.HTML2Text with body_width=0"]
CleanFooter["clean_deepwiki_footer(markdown)"]
FixLinks["Regex replace: wiki links → .md paths"]
SavePhase["File Organization Phase"]
DetermineLevel{"page['level'] == 0?"}
SaveRoot["Save to temp_dir/NUM-title.md"]
CreateSubdir["Create temp_dir/section-N/"]
SaveSubdir["Save to section-N/NUM-title.md"]
NextPage{"More pages?"}
Complete["Phase 1 Complete: temp_dir contains all .md files"]
Start --> CreateTemp
 
   CreateTemp --> CreateSession
 
   CreateSession --> DiscoverPhase
    
 
   DiscoverPhase --> ExtractWiki
 
   ExtractWiki --> ParseLinks
 
   ParseLinks --> SortPages
 
   SortPages --> ExtractionPhase
    
 
   ExtractionPhase --> LoopPages
 
   LoopPages --> FetchContent
 
   FetchContent --> FetchHTML
 
   FetchHTML --> ParseHTML
 
   ParseHTML --> RemoveNav
 
   RemoveNav --> FindContent
 
   FindContent --> ConvertPhase
    
 
   ConvertPhase --> ConvertMD
 
   ConvertMD --> HTML2Text
 
   HTML2Text --> CleanFooter
 
   CleanFooter --> FixLinks
 
   FixLinks --> SavePhase
    
 
   SavePhase --> DetermineLevel
 
   DetermineLevel -->|Yes: Main Page| SaveRoot
 
   DetermineLevel -->|No: Subsection| CreateSubdir
 
   CreateSubdir --> SaveSubdir
 
   SaveRoot --> NextPage
 
   SaveSubdir --> NextPage
    
 
   NextPage -->|Yes| LoopPages
 
   NextPage -->|No| Complete

Core Components and Data Flow

Structure Discovery Pipeline

The structure discovery process identifies all wiki pages and builds a hierarchical page list:

Sources: tools/deepwiki-scraper.py:78-125 tools/deepwiki-scraper.py:90-116 tools/deepwiki-scraper.py:118-123

flowchart LR
    subgraph Input
        BaseURL["Base URL\ndeepwiki.com/owner/repo"]
end
    
    subgraph extract_wiki_structure
        FetchMain["fetch_page(base_url)"]
ParseSoup["BeautifulSoup(response.text)"]
FindLinks["soup.find_all('a', href=regex)"]
ExtractInfo["Extract page number & title\nRegex: /(\d+(?:\.\d+)*)-(.+)$"]
CalcLevel["Calculate level from dots\nlevel = page_num.count('.')"]
BuildPages["Build pages list with metadata"]
SortFunc["Sort by sort_key(page)\nparts = [int(x)
for x in num.split('.')]"]
end
    
    subgraph Output
        PagesList["List[Dict]\n{'number': '2.1',\n'title': 'Section',\n'url': full_url,\n'href': path,\n'level': 1}"]
end
    
 
   BaseURL --> FetchMain
 
   FetchMain --> ParseSoup
 
   ParseSoup --> FindLinks
 
   FindLinks --> ExtractInfo
 
   ExtractInfo --> CalcLevel
 
   CalcLevel --> BuildPages
 
   BuildPages --> SortFunc
 
   SortFunc --> PagesList

Content Extraction and Cleaning

Each page undergoes a multi-step cleaning process to remove DeepWiki UI elements:

Sources: tools/deepwiki-scraper.py:27-42 tools/deepwiki-scraper.py:453-594 tools/deepwiki-scraper.py:175-190 tools/deepwiki-scraper.py:127-173

flowchart TD
    subgraph fetch_page
        MakeRequest["requests.get(url, headers)\nUser-Agent: Mozilla/5.0..."]
RetryLogic["Retry up to 3 times\n2 second delay between attempts"]
CheckStatus["response.raise_for_status()"]
end
    
    subgraph extract_page_content
        ParsePage["BeautifulSoup(response.text)"]
RemoveUnwanted["Decompose: nav, header, footer,\naside, .sidebar, script, style"]
FindMain["Try selectors in order:\narticle → main → .wiki-content\n→ [role='main'] → body"]
RemoveUI["Remove DeepWiki UI elements:\n'Edit Wiki', 'Last indexed:',\n'Index your code with Devin'"]
RemoveNavLists["Remove navigation <ul> lists\n(80%+ internal wiki links)"]
end
    
    subgraph convert_html_to_markdown
        HTML2TextInit["h = html2text.HTML2Text()\nh.ignore_links = False\nh.body_width = 0"]
HandleContent["markdown = h.handle(html_content)"]
CleanFooterCall["clean_deepwiki_footer(markdown)"]
end
    
    subgraph clean_deepwiki_footer
        SplitLines["lines = markdown.split('\\n')"]
ScanBackward["Scan last 50 lines backward\nfor footer patterns"]
MatchPatterns["Regex patterns:\n'Dismiss', 'Refresh this wiki',\n'On this page', 'Edit Wiki'"]
TruncateLines["lines = lines[:footer_start]"]
RemoveEmpty["Remove trailing empty lines"]
end
    
 
   MakeRequest --> RetryLogic
 
   RetryLogic --> CheckStatus
 
   CheckStatus --> ParsePage
    
 
   ParsePage --> RemoveUnwanted
 
   RemoveUnwanted --> FindMain
 
   FindMain --> RemoveUI
 
   RemoveUI --> RemoveNavLists
 
   RemoveNavLists --> HTML2TextInit
    
 
   HTML2TextInit --> HandleContent
 
   HandleContent --> CleanFooterCall
    
 
   CleanFooterCall --> SplitLines
 
   SplitLines --> ScanBackward
 
   ScanBackward --> MatchPatterns
 
   MatchPatterns --> TruncateLines
 
   TruncateLines --> RemoveEmpty

Link Rewriting Logic

Phase 1 transforms internal DeepWiki links into relative Markdown file paths. The rewriting logic accounts for hierarchical directory structure:

Sources: tools/deepwiki-scraper.py:549-592

flowchart TD
    subgraph Input
        WikiLink["DeepWiki Link\n[text](/owner/repo/2-1-section)"]
SourcePage["Current Page Info\n{level: 1, number: '2.1'}"]
end
    
    subgraph fix_wiki_link
        ExtractPath["Regex: /(\d+(?:\.\d+)*)-(.+)$"]
ParseNumbers["Extract: page_num='2.1', slug='section'"]
ConvertNum["file_num = page_num.replace('.', '-')\nResult: '2-1'"]
CheckTarget{"Is target\nsubsection?\n(has '.')"}
CheckSource{"Is source\nsubsection?\n(level > 0)"}
CheckSame{"Same main\nsection?"}
PathSameSection["Relative path:\nfile_num-slug.md"]
PathDiffSection["Full path:\nsection-N/file_num-slug.md"]
PathToMain["Up one level:\n../file_num-slug.md"]
PathMainToMain["Same level:\nfile_num-slug.md"]
end
    
    subgraph Output
        MDLink["Markdown Link\n[text](2-1-section.md)\nor [text](section-2/2-1-section.md)\nor [text](../2-1-section.md)"]
end
    
 
   WikiLink --> ExtractPath
 
   ExtractPath --> ParseNumbers
 
   ParseNumbers --> ConvertNum
 
   ConvertNum --> CheckTarget
    
 
   CheckTarget -->|Yes| CheckSource
 
   CheckTarget -->|No: Main Page| CheckSource
    
 
   CheckSource -->|Target: Sub, Source: Sub| CheckSame
 
   CheckSource -->|Target: Sub, Source: Main| PathDiffSection
 
   CheckSource -->|Target: Main, Source: Sub| PathToMain
 
   CheckSource -->|Target: Main, Source: Main| PathMainToMain
    
 
   CheckSame -->|Yes| PathSameSection
 
   CheckSame -->|No| PathDiffSection
    
 
   PathSameSection --> MDLink
 
   PathDiffSection --> MDLink
 
   PathToMain --> MDLink
 
   PathMainToMain --> MDLink

File Organization Strategy

Phase 1 organizes output files into a hierarchical directory structure based on page levels:

Directory Structure Rules

File Organization Implementation

Sources: tools/deepwiki-scraper.py:21-25 tools/deepwiki-scraper.py:845-868

HTTP Session Configuration

Phase 1 uses a persistent requests.Session with browser-like headers and retry logic:

Session Setup

Retry Strategy

Sources: tools/deepwiki-scraper.py:27-42 tools/deepwiki-scraper.py:817-821

Data Structures

Page Metadata Dictionary

Each page discovered by extract_wiki_structure() is represented as a dictionary:

Sources: tools/deepwiki-scraper.py:109-115

BeautifulSoup Content Selectors

Phase 1 attempts multiple selector strategies to find main content, in priority order:

Sources: tools/deepwiki-scraper.py:472-484

Error Handling and Robustness

Page Extraction Error Handling

Phase 1 implements graceful degradation for individual page failures:

Sources: tools/deepwiki-scraper.py:841-876

Content Extraction Fallbacks

If primary content selectors fail, Phase 1 applies fallback strategies:

1. Content Selector Fallback Chain : Try 8 different selectors (see table above)
2. Empty Content Check : Raises exception if no content element found tools/deepwiki-scraper.py:486-487
3. HTTP Retry Logic : 3 attempts with exponential backoff
4. Session Persistence : Reuses TCP connections for efficiency

Sources: tools/deepwiki-scraper.py:472-487 tools/deepwiki-scraper.py:27-42

Output Format

Temporary Directory Structure

At the end of Phase 1, the temporary directory contains the following structure:

temp_dir/
├── 1-overview.md                    # Main page (level 0)
├── 2-architecture.md                # Main page (level 0)
├── 3-components.md                  # Main page (level 0)
├── section-2/                       # Subsections of page 2
│   ├── 2-1-workspace-and-crates.md  # Subsection (level 1)
│   └── 2-2-dependency-graph.md      # Subsection (level 1)
└── section-4/                       # Subsections of page 4
    ├── 4-1-logical-planning.md
    └── 4-2-physical-planning.md

Markdown File Format

Each generated Markdown file has the following characteristics:

• Title : Always starts with # {Page Title} heading
• Content : Cleaned HTML converted to Markdown via html2text
• Links : Internal wiki links rewritten to relative .md paths
• No Diagrams : Diagrams are added in Phase 2 (see #7)
• No Footer : DeepWiki UI elements removed via clean_deepwiki_footer()
• Encoding : UTF-8

Sources: tools/deepwiki-scraper.py:862-868 tools/deepwiki-scraper.py:127-173

Phase 1 Completion Criteria

Phase 1 is considered complete when:

1. All pages discovered by extract_wiki_structure() have been processed
2. Each page’s Markdown file has been written to the temporary directory
3. Directory structure (main pages + section-N/ subdirectories) has been created
4. Success count is reported: "✓ Successfully extracted N/M pages to temp directory"

The temporary directory is then passed to Phase 2 for diagram enhancement.

Sources: tools/deepwiki-scraper.py877 tools/deepwiki-scraper.py:596-788

Dismiss

Refresh this wiki

Enter email to refresh

Wiki Structure Discovery

Loading…

Wiki Structure Discovery

Relevant source files

• python/deepwiki-scraper.py

Purpose and Scope

This document describes the wiki structure discovery mechanism in Phase 1 of the processing pipeline. The system analyzes the main DeepWiki repository page to identify all available wiki pages and their hierarchical relationships. This discovery phase produces a structured page list that drives subsequent content extraction.

For the HTML-to-Markdown conversion that follows discovery, see HTML to Markdown Conversion. For the overall Phase 1 process, see Phase 1: Markdown Extraction.

Overview

The discovery process fetches the main wiki page for a repository and parses its HTML to extract all wiki page references. The system identifies both main pages (e.g., 1, 2, 3) and subsections (e.g., 2.1, 2.2, 3.1) by analyzing link patterns. The output is a sorted list of page metadata that includes page numbers, titles, URLs, and hierarchical levels.

flowchart TD
 
   Start["main()
entry point"] --> ValidateRepo["Validate repo format\n(owner/repo)"]
ValidateRepo --> CreateSession["Create requests.Session\nwith User-Agent headers"]
CreateSession --> CallExtract["extract_wiki_structure(repo, session)"]
CallExtract --> FetchMain["Fetch https://deepwiki.com/{repo}"]
FetchMain --> ParseHTML["BeautifulSoup(response.text)"]
ParseHTML --> FindLinks["soup.find_all('a', href=regex)"]
FindLinks --> IterateLinks["Iterate over all links"]
IterateLinks --> ExtractPattern["Regex: /(\d+(?:\.\d+)*)-(.+)$"]
ExtractPattern --> BuildPageDict["Build page dict:\n{number, title, url, href, level}"]
BuildPageDict --> CheckDupe{"href in seen_urls?"}
CheckDupe -->|Yes| IterateLinks
 
   CheckDupe -->|No| AddToList["pages.append(page_dict)"]
AddToList --> IterateLinks
    
 
   IterateLinks -->|Done| SortPages["Sort by numeric parts:\nsort_key([int(x)
for x in num.split('.')])"]
SortPages --> ReturnPages["Return pages list"]
ReturnPages --> ProcessPages["Process each page\nin main loop"]
style CallExtract fill:#f9f,stroke:#333,stroke-width:2px
    style ExtractPattern fill:#f9f,stroke:#333,stroke-width:2px
    style SortPages fill:#f9f,stroke:#333,stroke-width:2px

Discovery Flow Diagram

Sources: tools/deepwiki-scraper.py:78-125 tools/deepwiki-scraper.py:790-831

Main Discovery Function

The extract_wiki_structure function performs the core discovery logic. It accepts a repository identifier (e.g., "jzombie/deepwiki-to-mdbook") and an HTTP session object, then returns a list of page dictionaries.

Function Signature and Entry Point

Sources: tools/deepwiki-scraper.py:78-79

HTTP Request and HTML Parsing

The function constructs the base URL and fetches the main wiki page:

The fetch_page helper includes retry logic (3 attempts) and browser-like headers to handle transient failures.

Sources: tools/deepwiki-scraper.py:80-84 tools/deepwiki-scraper.py:27-42

Link Pattern Matching

Regex-Based Link Discovery

The system uses a compiled regex pattern to find all wiki page links:

This pattern matches URLs like:

• /jzombie/deepwiki-to-mdbook/1-overview
• /jzombie/deepwiki-to-mdbook/2-quick-start
• /jzombie/deepwiki-to-mdbook/2-1-basic-usage

Sources: tools/deepwiki-scraper.py:88-90

Page Information Extraction

For each matched link, the system extracts page metadata using a detailed regex pattern:

The regex r'/(\d+(?:\.\d+)*)-(.+)$' captures:

• Group 1: Page number with optional dots (e.g., 1, 2.1, 3.2.1)
• Group 2: URL slug (e.g., overview, basic-usage)

Sources: tools/deepwiki-scraper.py:98-107

Link Extraction Data Flow

Sources: tools/deepwiki-scraper.py:98-115

Deduplication and Sorting

Deduplication Strategy

The system maintains a seen_urls set to prevent duplicate page entries:

Sources: tools/deepwiki-scraper.py:92-116

Hierarchical Sorting

Pages are sorted by their numeric components to maintain proper ordering:

This ensures ordering like: 1 → 2 → 2.1 → 2.2 → 3 → 3.1

Sources: tools/deepwiki-scraper.py:118-123

Sorting Example

Page Data Structure

Page Dictionary Schema

Each discovered page is represented as a dictionary:

Sources: tools/deepwiki-scraper.py:109-115

Level Calculation

The level field indicates hierarchical depth:

Sources: tools/deepwiki-scraper.py:106-114

Discovery Result Processing

Output Statistics

After discovery, the system categorizes pages and reports statistics:

Sources: tools/deepwiki-scraper.py:824-837

Integration with Content Extraction

The discovered page list drives the extraction loop in main():

Sources: tools/deepwiki-scraper.py:841-860

Alternative Discovery Method (Unused)

Subsection Probing Function

The codebase includes a discover_subsections function that uses HTTP HEAD requests to probe for subsections, but this function is not invoked in the current implementation:

This function attempts to discover subsections by making HEAD requests to potential URLs (e.g., /repo/2-1-, /repo/2-2-). However, the actual implementation relies entirely on parsing links from the main wiki page.

Sources: tools/deepwiki-scraper.py:44-76

Discovery Method Comparison

Sources: tools/deepwiki-scraper.py:44-76 tools/deepwiki-scraper.py:78-125

Error Handling

No Pages Found

The system validates that at least one page was discovered:

Sources: tools/deepwiki-scraper.py:828-830

Network Failures

The fetch_page function includes retry logic:

Sources: tools/deepwiki-scraper.py:33-42

Summary

The wiki structure discovery process provides a robust mechanism for identifying all pages in a DeepWiki repository through a single HTML parse operation. The resulting page list is hierarchically organized and drives all subsequent extraction operations in Phase 1.

Sources: tools/deepwiki-scraper.py:78-125 tools/deepwiki-scraper.py:790-831

Dismiss

Refresh this wiki

Enter email to refresh

HTML to Markdown Conversion

Loading…

HTML to Markdown Conversion

Relevant source files

• python/deepwiki-scraper.py

This page documents the HTML to Markdown conversion process in Phase 1 of the pipeline. After the wiki structure is discovered (see Wiki Structure Discovery), each page’s HTML content is fetched, cleaned, and converted to Markdown format. This conversion prepares the content for diagram enhancement in Phase 2 (see Phase 2: Diagram Enhancement).

Conversion Pipeline

The HTML to Markdown conversion follows a multi-step pipeline that progressively cleans and transforms the content. The process is orchestrated by the extract_page_content function and involves HTML parsing, element removal, conversion, and post-processing.

Conversion Pipeline Flow

graph TB
    fetch["fetch_page()\n[65-80]"]
parse["BeautifulSoup()\nHTML Parser"]
remove1["Remove Navigation\nElements [761-762]"]
remove2["Find Main Content\nArea [765-782]"]
remove3["Remove DeepWiki UI\nElements [786-806]"]
convert["convert_html_to_markdown()\n[213-228]"]
clean["clean_deepwiki_footer()\n[165-211]"]
format["format_source_references()\n[397-406]"]
links["Fix Internal Links\n[854-875]"]
output["Cleaned Markdown\nOutput"]
fetch --> parse
 
   parse --> remove1
 
   remove1 --> remove2
 
   remove2 --> remove3
 
   remove3 --> convert
 
   convert --> clean
 
   clean --> format
 
   format --> links
 
   links --> output

Sources: python/deepwiki-scraper.py:751-877

HTML Parsing and Content Extraction

The conversion begins by fetching the HTML page using a requests.Session with browser-like headers to avoid bot detection. BeautifulSoup parses the HTML into a navigable tree structure.

Content Area Detection

The system uses a cascading selector strategy to locate the main content area, trying multiple selectors in order of preference:

Content Detection Logic

Sources: python/deepwiki-scraper.py:765-782

DeepWiki UI Element Removal

Before conversion, the system removes DeepWiki-specific navigation and UI elements that would pollute the final documentation. This occurs in two stages: pre-processing element removal and footer cleanup.

Pre-Processing Element Removal

The first stage removes structural elements and UI components using CSS selectors:

The second stage removes DeepWiki-specific text-based UI elements by scanning for characteristic strings:

Sources: python/deepwiki-scraper.py:761-762 python/deepwiki-scraper.py:786-795

Navigation List Removal

DeepWiki pages include navigation lists that link to all wiki pages. The system detects and removes these by identifying unordered lists (<ul>) with the following characteristics:

1. Contains more than 5 links
2. At least 80% of links are internal (start with /)

graph LR
    ul["Find all ul\nElements"]
count["Count Links\nin List"]
check1{"More than\n5 links?"}
check2{"80%+ are\ninternal?"}
remove["Remove ul\nElement"]
keep["Keep ul\nElement"]
ul --> count
 
   count --> check1
 
   check1 -->|Yes| check2
 
   check1 -->|No| keep
 
   check2 -->|Yes| remove
 
   check2 -->|No| keep

Navigation List Detection

Sources: python/deepwiki-scraper.py:799-806

html2text Conversion

After cleaning the HTML, the system uses the html2text library to convert HTML to Markdown. The conversion is configured with specific settings to preserve link structure and prevent line wrapping.

html2text Configuration

The body_width = 0 setting is critical because it prevents the converter from introducing artificial line breaks that would break code blocks and formatted content.

Important: Mermaid diagram extraction is explicitly disabled at this stage. DeepWiki’s Next.js payload contains diagrams from ALL pages mixed together, making per-page extraction unreliable. Diagrams are handled separately in Phase 2 using fuzzy matching (see Fuzzy Matching Algorithm).

Sources: python/deepwiki-scraper.py:213-228

graph TB
    scan["Scan Last 50 Lines\nBackwards"]
patterns["Check Against\nFooter Patterns"]
found{"Pattern\nMatch?"}
backward["Scan Backward\n20 More Lines"]
content{"Hit Real\nContent?"}
cut["Cut Lines from\nFooter Start"]
trim["Trim Trailing\nEmpty Lines"]
scan --> patterns
 
   patterns --> found
 
   found -->|Yes| backward
 
   found -->|No| trim
 
   backward --> content
 
   content -->|Yes| cut
 
   content -->|No| backward
 
   cut --> trim

Footer Cleanup

The clean_deepwiki_footer function removes DeepWiki’s footer UI elements that appear at the end of each page. It uses regex patterns to detect footer markers and removes everything from that point onward.

Footer Detection Patterns

The footer patterns are compiled regex expressions:

Sources: python/deepwiki-scraper.py:165-211

Post-Processing Steps

After initial conversion, two post-processing steps refine the Markdown output: source reference formatting and internal link rewriting.

Source Reference Formatting

The format_source_references function inserts colons between filenames and line numbers in source code references. This transforms patterns like [path/to/file:10-20] into [path/to/file:10-20].

Pattern Matching:

• Regex: \[([A-Za-z0-9._/-]+?)(\d+-\d+)\]
• Capture Group 1: Filename path
• Capture Group 2: Line number range
• Output: [filename:linerange]

Sources: python/deepwiki-scraper.py:395-406

graph TB
    find["Find Link Pattern:\n/owner/repo/page"]
extract["Extract page_num\nand slug"]
normalize["normalized_number_parts()\n[28-43]"]
build["build_target_path()\n[55-63]"]
relative{"Source in\nSubsection?"}
same{"Target in Same\nSection?"}
diff["Prefix: ../\nDifferent section"]
none["Prefix: ../\nTop level"]
local["No prefix\nSame section"]
find --> extract
 
   extract --> normalize
 
   normalize --> build
 
   build --> relative
 
   relative -->|Yes| same
 
   relative -->|No| done["Return\nRelative Path"]
same -->|Yes| local
 
   same -->|No| diff
 
   diff --> done
 
   local --> done
 
   none --> done

Internal Link Rewriting

DeepWiki uses absolute URLs for internal wiki links (e.g., /owner/repo/4-query-planning). The system rewrites these to relative Markdown file paths using the build_target_path function.

Link Rewriting Process

Path Resolution Examples:

Sources: python/deepwiki-scraper.py:854-875 python/deepwiki-scraper.py:55-63 python/deepwiki-scraper.py:28-43

Duplicate Content Removal

The final cleanup step removes duplicate titles and stray “Menu” text that may appear in the converted Markdown. The system tracks whether a title has been seen and skips subsequent occurrences if they match the first title exactly.

Cleanup Rules:

1. Skip standalone “Menu” lines
2. Keep first # Title occurrence
3. Skip duplicate titles that match the first title
4. Preserve all other content

Sources: python/deepwiki-scraper.py:820-841

Output Format

The final output is clean Markdown with the following characteristics:

• Title guaranteed to be present (added if missing)
• No DeepWiki UI elements
• No artificial line wrapping
• Relative internal links
• Formatted source references
• Stripped trailing whitespace

The output is written to temporary storage before diagram enhancement in Phase 2. A snapshot of this raw Markdown (without diagrams) is saved to raw_markdown/ for debugging purposes.

Sources: python/deepwiki-scraper.py:1357-1366

Dismiss

Refresh this wiki

Enter email to refresh

Phase 2: Diagram Enhancement

Loading…

Phase 2: Diagram Enhancement

Relevant source files

• python/deepwiki-scraper.py
• scripts/build-docs.sh

Purpose and Scope

Phase 2 performs intelligent diagram extraction and placement after Phase 1 has generated clean markdown files. This phase extracts Mermaid diagrams from DeepWiki’s JavaScript payload, matches them to appropriate locations in the markdown content using fuzzy text matching, and inserts them contextually after relevant paragraphs.

For information about the initial markdown extraction that precedes this phase, see Phase 1: Markdown Extraction. For details on the specific fuzzy matching algorithm implementation, see Fuzzy Diagram Matching Algorithm. For information about the extraction patterns used, see Diagram Extraction from Next.js.

Sources: README.md:130-136 tools/deepwiki-scraper.py:596-789

The Client-Side Rendering Problem

DeepWiki renders diagrams client-side using JavaScript, making them invisible to traditional HTML scraping. All Mermaid diagrams are embedded in a JavaScript payload (self.__next_f.push) that contains diagram code for all pages in the wiki , not just the current page. This creates a matching problem: given ~461 diagrams in a single payload and individual markdown files, how do we determine which diagrams belong in which files?

Key challenges:

• Diagrams are escaped JavaScript strings (\n, \t, \")
• No metadata associates diagrams with specific pages
• html2text conversion changes text formatting from the original JavaScript context
• Must avoid false positives (placing diagrams in wrong locations)

Sources: tools/deepwiki-scraper.py:458-461 README.md:131-136

Architecture Overview

Diagram: Phase 2 Processing Pipeline

Sources: tools/deepwiki-scraper.py:596-789

Diagram Extraction Process

The extraction process reads the JavaScript payload from any DeepWiki page and locates all Mermaid diagram blocks using regex pattern matching.

flowchart TD
    Start["extract_and_enhance_diagrams()"]
FetchURL["Fetch https://deepwiki.com/repo/1-overview"]
subgraph "Pattern Matching"
        Pattern1["Pattern: r'```mermaid\\\\\n(.*?)```'\n(re.DOTALL)"]
Pattern2["Pattern: r'([^`]{500,}?)```mermaid\\\\ (.*?)```'\n(with context)"]
FindAll["re.findall() → all_diagrams list"]
FindIter["re.finditer() → diagram_contexts with context"]
end
    
    subgraph "Unescaping"
        ReplaceNewline["Replace '\\\\\n' → newline"]
ReplaceTab["Replace '\\\\ ' → tab"]
ReplaceQuote["Replace '\\\\\"' → double-quote"]
ReplaceUnicode["Replace Unicode escapes:\n\\\< → '<'\n\\\> → '>'\n\\\& → '&'"]
end
    
    subgraph "Context Processing"
        Last500["Extract last 500 chars of context"]
FindHeading["Scan for last heading starting with #"]
ExtractAnchor["Extract last 2-3 non-heading lines\n(min 20 chars each)"]
BuildDict["Build dict: {last_heading, anchor_text, diagram}"]
end
    
 
   Start --> FetchURL
 
   FetchURL --> Pattern1
 
   FetchURL --> Pattern2
 
   Pattern1 --> FindAll
 
   Pattern2 --> FindIter
    
 
   FindAll --> ReplaceNewline
 
   FindIter --> ReplaceNewline
 
   ReplaceNewline --> ReplaceTab
 
   ReplaceTab --> ReplaceQuote
 
   ReplaceQuote --> ReplaceUnicode
    
 
   ReplaceUnicode --> Last500
 
   Last500 --> FindHeading
 
   FindHeading --> ExtractAnchor
 
   ExtractAnchor --> BuildDict
    
 
   BuildDict --> Output["Returns:\n- all_diagrams count\n- diagram_contexts list"]

Extraction Function Flow

Diagram: Diagram Extraction and Context Building

Sources: tools/deepwiki-scraper.py:604-674

Key Implementation Details

Sources: tools/deepwiki-scraper.py:614-674

Fuzzy Matching Algorithm

The fuzzy matching algorithm determines where each diagram should be inserted by finding the best match between the diagram’s context and the markdown file’s content.

flowchart TD
    Start["For each diagram_contexts[idx]"]
CheckUsed["idx in diagrams_used?"]
Skip["Skip to next diagram"]
subgraph "Text Normalization"
        NormFile["Normalize file content:\ncontent.lower()\n' '.join(content.split())"]
NormAnchor["Normalize anchor_text:\nanchor.lower()\n' '.join(anchor.split())"]
NormHeading["Normalize heading:\nheading.lower().replace('#', '').strip()"]
end
    
    subgraph "Progressive Chunk Matching"
        Try300["Try chunk_size=300"]
Try200["Try chunk_size=200"]
Try150["Try chunk_size=150"]
Try100["Try chunk_size=100"]
Try80["Try chunk_size=80"]
ExtractChunk["test_chunk = anchor_normalized[-chunk_size:]"]
FindPos["pos = content_normalized.find(test_chunk)"]
CheckPos["pos != -1?"]
ConvertLine["Convert char position to line number"]
RecordMatch["Record best_match_line, best_match_score"]
end
    
    subgraph "Heading Fallback"
        IterLines["For each line in markdown"]
CheckHeadingLine["line.strip().startswith('#')?"]
NormalizeLinе["Normalize line heading"]
CheckContains["heading_normalized in line_normalized?"]
RecordHeadingMatch["best_match_line = line_num\nbest_match_score = 50"]
end
    
 
   Start --> CheckUsed
 
   CheckUsed -->|Yes| Skip
 
   CheckUsed -->|No| NormFile
    
 
   NormFile --> NormAnchor
 
   NormAnchor --> Try300
 
   Try300 --> ExtractChunk
 
   ExtractChunk --> FindPos
 
   FindPos --> CheckPos
 
   CheckPos -->|Found| ConvertLine
 
   CheckPos -->|Not found| Try200
 
   ConvertLine --> RecordMatch
    
 
   Try200 --> Try150
 
   Try150 --> Try100
 
   Try100 --> Try80
 
   Try80 -->|All failed| IterLines
    
 
   RecordMatch --> Success["Return match with score"]
IterLines --> CheckHeadingLine
 
   CheckHeadingLine -->|Yes| NormalizeLinе
 
   NormalizeLinе --> CheckContains
 
   CheckContains -->|Yes| RecordHeadingMatch
 
   RecordHeadingMatch --> Success

Matching Strategy

Diagram: Progressive Chunk Matching with Fallback

Sources: tools/deepwiki-scraper.py:708-746

Chunk Size Progression

The algorithm tries progressively smaller chunk sizes to accommodate variations in text formatting between the JavaScript context and the html2text-converted markdown:

The algorithm stops at the first successful match, prioritizing larger chunks for higher confidence.

Sources: tools/deepwiki-scraper.py:716-730 README.md134

flowchart TD
    Start["Found best_match_line"]
CheckType["lines[best_match_line].strip().startswith('#')?"]
subgraph "Heading Case"
        H1["insert_line = best_match_line + 1"]
H2["Skip blank lines after heading"]
H3["Skip through paragraph content"]
H4["Stop at next blank line or heading"]
end
    
    subgraph "Paragraph Case"
        P1["insert_line = best_match_line + 1"]
P2["Find end of current paragraph"]
P3["Stop at next blank line or heading"]
end
    
    subgraph "Insertion Format"
        I1["Insert: empty line"]
I2["Insert: ```mermaid"]
I3["Insert: diagram code"]
I4["Insert: ```"]
I5["Insert: empty line"]
end
    
 
   Start --> CheckType
 
   CheckType -->|Heading| H1
 
   CheckType -->|Paragraph| P1
    
 
   H1 --> H2
 
   H2 --> H3
 
   H3 --> H4
    
 
   P1 --> P2
 
   P2 --> P3
    
 
   H4 --> I1
 
   P3 --> I1
    
 
   I1 --> I2
 
   I2 --> I3
 
   I3 --> I4
 
   I4 --> I5
    
 
   I5 --> Append["Append to pending_insertions list:\n(insert_line, diagram, score, idx)"]

Insertion Point Logic

After finding a match, the system determines the precise line number where the diagram should be inserted.

Insertion Algorithm

Diagram: Insertion Point Calculation

Sources: tools/deepwiki-scraper.py:747-768

graph LR
    Collect["Collect all\npending_insertions"]
Sort["Sort by insert_line\n(descending)"]
Insert["Insert from bottom to top\npreserves line numbers"]
Write["Write enhanced file\nto temp_dir"]
Collect --> Sort
 
   Sort --> Insert
 
   Insert --> Write

Batch Insertion Strategy

Diagrams are inserted in descending line order to avoid invalidating insertion points:

Diagram: Batch Insertion Order

Implementation:

Sources: tools/deepwiki-scraper.py:771-783

sequenceDiagram
    participant Main as extract_and_enhance_diagrams()
    participant Glob as temp_dir.glob('**/*.md')
    participant File as Individual .md file
    participant Matcher as Fuzzy Matcher
    participant Writer as File Writer
    
    Main->>Main: Extract all diagram_contexts
    Main->>Glob: Find all markdown files
    
    loop For each md_file
        Glob->>File: Open and read content
        File->>File: Check if '```mermaid' already present
        
        alt Already has diagrams
            File->>Glob: Skip (continue)
        else No diagrams
            File->>Matcher: Normalize content
            
            loop For each diagram_context
                Matcher->>Matcher: Try progressive chunk matching
                Matcher->>Matcher: Try heading fallback
                Matcher->>Matcher: Record best match
            end
            
            Matcher->>File: Return pending_insertions list
            File->>File: Sort insertions (descending)
            File->>File: Insert diagrams bottom-up
            File->>Writer: Write enhanced content
            Writer->>Main: Increment enhanced_count
        end
    end
    
    Main->>Main: Print summary

File Processing Workflow

Phase 2 operates on files in the temporary directory created by Phase 1, enhancing them in-place before they are moved to the final output directory.

Processing Loop

Diagram: File Processing Sequence

Sources: tools/deepwiki-scraper.py:676-788

Performance Characteristics

Extraction Statistics

From a typical wiki with ~10 pages:

Sources: README.md:132-133 tools/deepwiki-scraper.py674 tools/deepwiki-scraper.py788

Matching Performance

The progressive chunk size strategy balances precision and recall:

• High precision matches (300-200 chars) : Strong contextual alignment
• Medium precision matches (150-100 chars) : Acceptable with some risk
• Low precision matches (80 chars) : Risk of false positives
• Heading-only matches (score: 50) : Last resort fallback

The algorithm prefers to skip a diagram rather than place it incorrectly, prioritizing documentation quality over diagram count.

Sources: tools/deepwiki-scraper.py:716-745

Integration with Phases 1 and 3

Input Requirements (from Phase 1)

• Clean markdown files in temp_dir
• Files must not already contain \```mermaid blocks
• Proper heading structure for fallback matching
• Normalized link structure

Sources: tools/deepwiki-scraper.py:810-877

Output Guarantees (for Phase 3)

• Enhanced markdown files in temp_dir
• Diagrams inserted with proper fencing: \```mermaid…`````
• Blank lines before and after diagrams for proper rendering
• Original file structure preserved (section-N directories maintained)
• Atomic file operations (write complete file or skip)

Sources: tools/deepwiki-scraper.py:781-786 tools/deepwiki-scraper.py:883-908

Workflow Integration

Diagram: Three-Phase Integration

Sources: README.md:123-145 tools/deepwiki-scraper.py:810-916

Error Handling and Edge Cases

Skipped Files

Files are skipped if they already contain Mermaid diagrams to avoid duplicate insertion:

Sources: tools/deepwiki-scraper.py:686-687

Failed Matches

When a diagram cannot be matched:

• The diagram is not inserted (conservative approach)
• No error is raised (continues processing other diagrams)
• File is left unmodified if no diagrams match

Sources: tools/deepwiki-scraper.py:699-746

Network Errors

If diagram extraction fails (network error, changed HTML structure):

• Warning is printed but Phase 2 continues
• Phase 1 files remain valid
• System can still proceed to Phase 3 without diagrams

Sources: tools/deepwiki-scraper.py:610-612

Diagram Quality Thresholds

Sources: tools/deepwiki-scraper.py648 tools/deepwiki-scraper.py712 tools/deepwiki-scraper.py661

Summary

Phase 2 implements a sophisticated fuzzy matching system that:

1. Extracts all Mermaid diagrams from DeepWiki’s JavaScript payload using regex patterns
2. Processes diagram context to extract heading and anchor text metadata
3. Matches diagrams to markdown files using progressive chunk size comparison (300→80 chars)
4. Inserts diagrams after relevant paragraphs with proper formatting
5. Validates through conservative matching to avoid false positives

The phase operates entirely on files in the temporary directory, leaving Phase 1’s output intact while preparing enhanced files for Phase 3’s mdBook build process.

Sources: README.md:130-136 tools/deepwiki-scraper.py:596-789

Dismiss

Refresh this wiki

Enter email to refresh

Mermaid Normalization

Loading…

Mermaid Normalization

Relevant source files

• python/deepwiki-scraper.py
• python/tests/test_mermaid_normalization.py

The Mermaid normalization pipeline transforms diagrams extracted from DeepWiki’s JavaScript payload into syntax that is compatible with Mermaid 11. DeepWiki’s diagrams often contain formatting issues, legacy syntax, and multiline constructs that newer Mermaid parsers reject. This seven-step normalization process ensures that all diagrams render correctly in mdBook’s Mermaid renderer.

For information about how diagrams are extracted from the JavaScript payload, see Phase 2: Diagram Enhancement. For information about the fuzzy matching algorithm that places diagrams in the correct markdown files, see Fuzzy Matching Algorithm.

Purpose and Scope

This page documents the seven normalization functions that transform raw Mermaid diagram code into Mermaid 11-compatible syntax. Each normalization step addresses a specific category of syntax errors or incompatibilities. The pipeline is applied to every diagram before it is injected into markdown files.

The normalization pipeline handles:

• Multiline edge labels that span multiple lines
• State diagram description syntax variations
• Flowchart node labels containing reserved characters
• Missing statement separators between consecutive nodes
• Empty node labels that lack fallback text
• Gantt chart tasks missing required task identifiers
• Additional edge case transformations (quote stripping, label merging)

Normalization Pipeline Architecture

The normalization pipeline is orchestrated by the normalize_mermaid_diagram function, which applies seven normalization passes in sequence. Each pass is idempotent and focuses on a specific syntax issue.

Pipeline Flow Diagram

graph TD
    Input["Raw Diagram Text\nfrom Next.js Payload"]
Step1["normalize_mermaid_edge_labels()\nFlatten multiline edge labels"]
Step2["normalize_mermaid_state_descriptions()\nFix state syntax"]
Step3["normalize_flowchart_nodes()\nClean node labels"]
Step4["normalize_statement_separators()\nInsert newlines"]
Step5["normalize_empty_node_labels()\nAdd fallback labels"]
Step6["normalize_gantt_diagram()\nAdd synthetic task IDs"]
Output["Normalized Diagram\nMermaid 11 Compatible"]
Input --> Step1
 
   Step1 --> Step2
 
   Step2 --> Step3
 
   Step3 --> Step4
 
   Step4 --> Step5
 
   Step5 --> Step6
 
   Step6 --> Output

Sources: python/deepwiki-scraper.py:385-393 python/deepwiki-scraper.py:230-383

Function Name to Normalization Step Mapping

Sources: python/deepwiki-scraper.py:385-393

Step 1: Edge Label Normalization

The normalize_mermaid_edge_labels function collapses multiline edge labels into single-line labels with escaped newline sequences. Mermaid 11 rejects edge labels that span multiple physical lines.

Function : normalize_mermaid_edge_labels(diagram_text: str) -> str

Pattern Matched : Edge labels enclosed in pipes: |....|

Transformations Applied :

• Replace literal newline characters with spaces
• Replace escaped \n sequences with spaces
• Remove parentheses from labels (invalid syntax)
• Collapse multiple spaces into single spaces

Implementation Details :

• Only processes diagrams starting with graph or flowchart keywords
• Uses regex pattern \|([^|]*)\| to match edge labels
• Checks for presence of \n, (, or ) before applying cleanup
• Preserves labels that are already properly formatted

Sources: python/deepwiki-scraper.py:230-251

Step 2: State Description Normalization

The normalize_mermaid_state_descriptions function ensures state diagram descriptions follow the strict State : Description syntax required by Mermaid 11.

Function : normalize_mermaid_state_descriptions(diagram_text: str) -> str

Pattern Matched : State declarations with colons in state diagrams

Transformations Applied :

• Ensure single space after state name before colon
• Replace newlines in descriptions with spaces
• Replace additional colons in description with -
• Collapse multiple spaces to single space

Implementation Details :

• Only processes diagrams starting with statediagram keyword
• Skips lines containing :: (double colon, used for class names)
• Splits each line on first colon occurrence
• Requires both prefix and suffix to be non-empty after stripping

Sources: python/deepwiki-scraper.py:253-277

Step 3: Flowchart Node Normalization

The normalize_flowchart_nodes function removes reserved characters (especially pipe |) from flowchart node labels and adds statement separators.

Function : normalize_flowchart_nodes(diagram_text: str) -> str

Pattern Matched : Node labels in brackets: ["..."]

Transformations Applied :

• Replace pipe characters | with forward slash /
• Collapse multiple spaces to single space
• Insert newlines between consecutive statements on same line

Implementation Details :

• Only processes diagrams starting with graph or flowchart keywords
• Uses regex \["([^"]*)"\] to match quoted node labels
• Inserts newlines after closing brackets/braces/parens using regex: (\"]|\}|\))\s+(?=[A-Za-z0-9_])
• Preserves indentation when splitting statements

Sources: python/deepwiki-scraper.py:279-301

Step 4: Statement Separator Normalization

The normalize_statement_separators function inserts newlines between consecutive Mermaid statements that have been flattened onto a single line.

Function : normalize_statement_separators(diagram_text: str) -> str

Connector Tokens Recognized :

--> ==> -.-> --x x-- o--> o-> x-> *--> <--> <-.-> <-- --o

Pattern Matched : Whitespace before a node identifier that precedes a connector

Regex Pattern : STATEMENT_BREAK_PATTERN

Implementation Details :

• Only processes diagrams starting with graph or flowchart keywords
• Defines FLOW_CONNECTORS list of all Mermaid connector tokens
• Builds regex pattern by escaping and joining connector tokens
• Pattern: (?<!\n)([ \t]+)(?=[A-Za-z0-9_][\w\-]*(?:\s*\[[^\]]*\])?\s*(?:CONNECTORS)(?:\|[^|]*\|)?\s*)
• Preserves indentation length when inserting newlines
• Converts tabs to 4 spaces for consistent indentation

Sources: python/deepwiki-scraper.py:303-328 python/deepwiki-scraper.py:309-311

Step 5: Empty Node Label Normalization

The normalize_empty_node_labels function provides fallback text for nodes with empty labels, which Mermaid 11 rejects.

Function : normalize_empty_node_labels(diagram_text: str) -> str

Pattern Matched : Empty quoted labels: NodeId[""]

Transformation Applied :

• Use node ID as fallback label text
• Replace underscores and hyphens with spaces
• Preserve original node ID for connections

Implementation Details :

• Regex pattern: (\b[A-Za-z0-9_]+)\[""\]
• Converts underscores/hyphens to spaces for readable label: re.sub(r'[_\-]+', ' ', node_id)
• Falls back to raw node_id if cleaned version is empty
• Applied to all diagram types (not limited to flowcharts)

Sources: python/deepwiki-scraper.py:330-341 python/tests/test_mermaid_normalization.py:19-23

Step 6: Gantt Diagram Normalization

The normalize_gantt_diagram function assigns synthetic task identifiers to gantt chart tasks that are missing them, which is required by Mermaid 11.

Function : normalize_gantt_diagram(diagram_text: str) -> str

Pattern Matched : Task lines in format "Task Name" : start, end[, duration]

Transformation Applied :

• Insert synthetic task ID (task1, task2, etc.) after colon
• Only apply to tasks lacking valid identifiers
• Preserve tasks that already have IDs or use after dependencies

Implementation Details :

• Only processes diagrams starting with gantt keyword
• Task line regex: ^(\s*"[^"]+"\s*):\s*(.+)$
• Splits remainder on commas (max 3 parts)
• Checks if first token matches ^[A-Za-z_][\w-]*$ or starts with after
• Maintains counter (task_counter) for generating unique IDs
• Reconstructs line: "{task_name}" : {task_id}, {start}, {end}[, {duration}]

Sources: python/deepwiki-scraper.py:343-383

Step 7: Additional Preprocessing

Before the seven main normalization steps, diagrams undergo additional preprocessing in the extraction phase:

Quote Stripping : strip_wrapping_quotes(diagram_text: str) -> str

• Removes unnecessary quotation marks around edge labels: |"text"| → |text|
• Removes quotes in state transitions: : "label" → : label

Label Merging : merge_multiline_labels(diagram_text: str) -> str

• Collapses wrapped labels inside node shapes into \n sequences
• Handles multiple shape types: (), [], {}, (()), [[]], {{}}
• Skips lines containing structural tokens (arrows, keywords)
• Applied before unescaping, so works with both real and escaped newlines

Sources: python/deepwiki-scraper.py:907-1023

Main Orchestrator Function

The normalize_mermaid_diagram function orchestrates all normalization passes in the correct order.

Function Signature : normalize_mermaid_diagram(diagram_text: str) -> str

Implementation :

Key Characteristics :

• Each pass is idempotent and can be safely applied multiple times
• Passes are independent and order-dependent
• Edge label normalization must precede statement separator insertion
• Flowchart node normalization includes its own statement separator logic
• Empty label normalization should occur after other node transformations

Sources: python/deepwiki-scraper.py:385-393

graph TD
    Extract["extract_and_enhance_diagrams()"]
Loop["For each diagram match"]
Unescape["Unescape sequences\n(\\\n, \\ , \<, etc)"]
Preprocess["merge_multiline_labels()\nstrip_wrapping_quotes()"]
Normalize["normalize_mermaid_diagram()"]
Context["Extract context\n(heading, anchor text)"]
Pool["Add to diagram_contexts list"]
Extract --> Loop
 
   Loop --> Unescape
 
   Unescape --> Preprocess
 
   Preprocess --> Normalize
 
   Normalize --> Context
 
   Context --> Pool

Normalization Invocation Points

The normalization pipeline is invoked at a single location during diagram processing:

Invocation Context Diagram

Sources: python/deepwiki-scraper.py:880-1089 python/deepwiki-scraper.py:1058-1060

Testing and Validation

The normalization pipeline has dedicated unit tests covering each normalization function:

Test Coverage :

Example Test Case (Statement Separator Normalization):

End-to-End Test : The end-to-end test validates that multiple normalization steps work together correctly:

• Input: graph TD\n Stage1[""] --> Stage2["Stage 2"]\n Stage2 --> Stage3 Stage3 --> Stage4
• Validates empty label replacement: Stage1["Stage1"]
• Validates statement separation: Stage2 --> Stage3 and Stage3 --> Stage4 on separate lines

Sources: python/tests/test_mermaid_normalization.py:1-42

Common Edge Cases

The normalization pipeline handles several edge cases that commonly occur in DeepWiki diagrams:

Empty Diagram Handling :

• All normalizers check for empty/whitespace-only input
• Return original text unchanged if stripped content is empty

Diagram Type Detection :

• Each normalizer checks diagram type via first line keyword
• normalize_mermaid_edge_labels: only processes graph or flowchart
• normalize_mermaid_state_descriptions: only processes statediagram
• normalize_gantt_diagram: only processes gantt
• Other normalizers apply to all diagram types

Indentation Preservation :

• Statement separator normalization preserves original indentation level
• Converts tabs to 4 spaces for consistent formatting
• Inserts newlines with matching indentation

Backtick Escaping in Fence Blocks : When injecting normalized diagrams into markdown, the injection logic dynamically calculates fence length to avoid conflicts with backticks inside diagram code:

• Scans diagram for longest backtick run
• Uses max(3, max_backticks + 1) as fence length

Sources: python/deepwiki-scraper.py:1249-1255

Dismiss

Refresh this wiki

Enter email to refresh

Fuzzy Matching Algorithm

Loading…

Fuzzy Matching Algorithm

Relevant source files

• python/deepwiki-scraper.py

Purpose and Scope

The fuzzy matching algorithm places Mermaid diagrams extracted from DeepWiki’s JavaScript payload into the correct locations within Markdown files. It matches diagram context (as it appears in the JavaScript) to content locations in html2text-converted Markdown, accounting for formatting differences between the two representations.

This algorithm is implemented in extract_and_enhance_diagrams() function and processes all files after the initial markdown extraction phase completes.

Sources: python/deepwiki-scraper.py:880-1275

The Matching Problem

The fuzzy matching algorithm addresses a fundamental mismatch: diagrams are embedded in DeepWiki’s JavaScript payload alongside their surrounding context text, but this context text differs significantly from the final Markdown output produced by html2text. The algorithm must find where each diagram belongs despite these differences.

Format Differences Between Sources

Sources: python/deepwiki-scraper.py:898-903

Context Extraction Strategy

The algorithm extracts two types of context for each diagram to enable matching:

1. Last Heading Before Diagram

Extractinglast_heading from Context

The algorithm scans backwards through context_lines to find the most recent line starting with #, which provides a coarse-grained location hint.

Sources: python/deepwiki-scraper.py:1066-1071

2. Anchor Text (Last 2-3 Paragraphs)

Extractinganchor_text from Context

The anchor_text consists of the last 2-3 substantial non-heading lines before the diagram, truncated to 300 characters. This provides fine-grained matching capability.

Sources: python/deepwiki-scraper.py:1073-1081

Progressive Chunk Size Matching

The core of the fuzzy matching algorithm uses progressively smaller chunk sizes to find matches, prioritizing longer (more specific) matches over shorter ones.

Chunk Size Progression

The algorithm tests chunks in this order:

Matching Algorithm Flow

Progressive Chunk Matching in Code

Sources: python/deepwiki-scraper.py:1169-1239

Text Normalization

Both the diagram context and the target Markdown content undergo identical normalization to maximize matching success:

This process:

• Converts all text to lowercase
• Collapses all consecutive whitespace (spaces, tabs, newlines) into single spaces
• Removes leading/trailing whitespace

Sources: python/deepwiki-scraper.py:1166-1167 python/deepwiki-scraper.py:1185-1186

Fallback: Heading-Based Matching

If progressive chunk matching fails (best_match_line == -1 and heading exists), the algorithm falls back to heading-based matching:

Heading Fallback Implementation

Heading-based matches receive a fixed best_match_score of 50, lower than any chunk-based match (minimum 80), indicating lower confidence.

Sources: python/deepwiki-scraper.py:1203-1216

Insertion Point Calculation

Once a match is found with best_match_score >= 80, the algorithm calculates the precise insert_line for the diagram:

Insertion After Headings

Calculatinginsert_line After Heading

Sources: python/deepwiki-scraper.py:1222-1230

Insertion After Paragraphs

Calculatinginsert_line After Paragraph

Sources: python/deepwiki-scraper.py:1232-1236

Scoring and Deduplication

The algorithm tracks which diagrams have been used to prevent duplicates:

For each file, the algorithm:

1. Attempts to match all diagrams in diagram_contexts with file content
2. Stores successful matches with their scores in pending_insertions as tuples: (insert_line, diagram, best_match_score, idx)
3. Marks diagrams as used by adding their index to diagrams_used set
4. Sorts pending_insertions by line number (descending) to avoid index shifting
5. Inserts diagrams from bottom to top

Sources: python/deepwiki-scraper.py:1162-1163 python/deepwiki-scraper.py1238 python/deepwiki-scraper.py:1242-1243

Diagram Insertion Format

Diagrams are inserted with proper Markdown fencing and spacing, accounting for backticks in the diagram content:

Buildinglines_to_insert

This results in the following structure in the Markdown file:

Next paragraph text.

If the diagram contains triple backticks, the fence length is increased (e.g., to 4 or 5 backticks) to avoid conflicts.

**Sources:** <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/python/deepwiki-scraper.py#L1249-L1266" min=1249 max=1266 file-path="python/deepwiki-scraper.py">Hii</FileRef>

## Complete Matching Pipeline

**`extract_and_enhance_diagrams()` Function Flow**

```mermaid
flowchart TD
    Start["extract_and_enhance_diagrams(\nrepo, temp_dir,\nsession, diagram_source_url)"] --> Fetch["response = session.get(\ndiagram_source_url)\nhtml_text = response.text"]
    Fetch --> ExtractPattern["diagram_pattern =\nr'```mermaid(?:\\r\\n|\\n|\r?\n)\n(.*?)(?:\\r\\n|\\n|\r?\n)```'\ndiagram_matches =\nlist(re.finditer(\npattern, html_text, re.DOTALL))"]
    ExtractPattern --> ContextLoop["diagram_contexts = []\nfor match in diagram_matches:\ncontext_start =\nmax(0, match.start() - 2000)"]
    ContextLoop --> ParseContext["extract:\n- last_heading\n- anchor_text[-300:]\n- diagram (unescaped)"]
    ParseContext --> NormalizeDiag["diagram =\nmerge_multiline_labels(diagram)\ndiagram =\nstrip_wrapping_quotes(diagram)\ndiagram =\nnormalize_mermaid_diagram(diagram)"]
    NormalizeDiag --> AppendContext["diagram_contexts.append({\n'last_heading': last_heading,\n'anchor_text': anchor_text,\n'diagram': diagram\n})"]
    AppendContext --> FindFiles["md_files =\nlist(temp_dir.glob('**/*.md'))"]
    
    FindFiles --> FileLoop["for md_file in md_files:"]
    FileLoop --> ReadFile["content = f.read()"]
    ReadFile --> CheckExists["re.search(\nr'^\\s*`{3,}\\s*mermaid\\b',\ncontent, re.IGNORECASE\n| re.MULTILINE)?"]
    CheckExists -->|Yes| Skip["continue"]
    CheckExists -->|No| SplitLines["lines = content.split('\\n')"]
    SplitLines --> InitVars["diagrams_used = set()\npending_insertions = []\ncontent_normalized =\ncontent.lower()"]
    InitVars --> DiagLoop["for idx, item in\nenumerate(diagram_contexts):"]
    
    DiagLoop --> TryChunks["for chunk_size in\n[300, 200, 150, 100, 80]:\ntest_chunk =\nanchor_normalized[-chunk_size:]\npos = content_normalized\n.find(test_chunk)"]
    TryChunks -->|Found| CalcLine["convert pos to line_num\nbest_match_line = line_num\nbest_match_score = chunk_size"]
    TryChunks -->|Not found| TryHeading["heading fallback matching"]
    TryHeading -->|Found| SetScore["best_match_score = 50"]
    CalcLine --> CheckScore["best_match_score >= 80?"]
    SetScore --> CheckScore
    CheckScore -->|Yes| CalcInsert["calculate insert_line:\nenforce_content_start()\nadvance_past_lists()"]
    CalcInsert --> AppendPending["pending_insertions.append(\ninsert_line, diagram,\nbest_match_score, idx)\ndiagrams_used.add(idx)"]
    CheckScore -->|No| NextDiag["next diagram"]
    
    AppendPending --> NextDiag
    NextDiag --> DiagLoop
    DiagLoop -->|All done| SortPending["pending_insertions.sort(\nkey=lambda x: x[0],\nreverse=True)"]
    SortPending --> InsertLoop["for insert_line, diagram,\nscore, idx in\npending_insertions:\ncalculate fence_len\nlines.insert(insert_line,\nlines_to_insert)"]
    InsertLoop --> SaveFile["with open(md_file, 'w')\nas f:\nf.write('\\n'.join(lines))"]
    SaveFile --> FileLoop
    FileLoop -->|All files| PrintStats["print(f'Enhanced\n{enhanced_count} files')"]

Sources: python/deepwiki-scraper.py:880-1275

Key Functions and Variables

Performance Characteristics

The algorithm processes diagrams in a single pass per file with the following complexity:

For a typical file with 1000 lines and 48 diagram candidates with context, the algorithm completes in under 100ms per file.

Sources: python/deepwiki-scraper.py:880-1275

Match Quality Statistics

As reported in the console output, the algorithm typically achieves:

• Total diagrams in JavaScript : ~461 diagrams across all pages
• Diagrams with sufficient context : ~48 diagrams (500+ char context)
• Average match rate : 60-80% of diagrams with context are successfully placed
• Typical score distribution :
  • 300-char matches: 20-30% (highest confidence)
  • 200-char matches: 15-25%
  • 150-char matches: 15-20%
  • 100-char matches: 10-15%
  • 80-char matches: 5-10%
  • Heading fallback: 5-10% (lowest confidence)

Sources: README.md:132-136 tools/deepwiki-scraper.py674

Dismiss

Refresh this wiki

Enter email to refresh

Phase 3: mdBook Build

Loading…

Phase 3: mdBook Build

Relevant source files

• scripts/build-docs.sh

Purpose and Scope

Phase 3 is the final transformation stage that converts enhanced markdown files into a searchable HTML documentation website using mdBook. This phase creates the book structure, generates the table of contents, injects templates, installs mermaid rendering support, and executes the mdBook build process.

This page covers the overall Phase 3 workflow and its core components. For detailed information about specific sub-processes, see:

• SUMMARY.md generation algorithm: 8.1
• Template injection mechanics: 8.2
• Configuration system: 3
• Template system details: 11

Phase 3 begins after Phase 2 completes diagram enhancement (see 7) and ends with the production of deployable HTML artifacts.

Sources: scripts/build-docs.sh:95-310

Phase 3 Process Flow

Phase 3 executes six distinct steps, each transforming the workspace toward the final HTML output. The process is orchestrated by build-docs.sh and coordinates multiple tools.

graph TB
    Input["Enhanced Markdown Files\n/workspace/wiki/"]
Step2["Step 2: Initialize mdBook Structure\nCreate /workspace/book/\nGenerate book.toml"]
Step3["Step 3: Generate SUMMARY.md\nDiscover file structure\nSort pages numerically"]
Step4["Step 4: Process Markdown Files\nInject header/footer templates\nCopy to book/src/"]
Step5["Step 5: Install Mermaid Assets\nmdbook-mermaid install"]
Step6["Step 6: Build Book\nmdbook build"]
Step7["Step 7: Copy Outputs\nbook/ → /output/book/\nbook.toml → /output/"]
Output["Final Outputs\n/output/book/ (HTML)\n/output/markdown/\n/output/book.toml"]
Input --> Step2
 
   Step2 --> Step3
 
   Step3 --> Step4
 
   Step4 --> Step5
 
   Step5 --> Step6
 
   Step6 --> Step7
 
   Step7 --> Output

High-Level Phase 3 Pipeline

Sources: scripts/build-docs.sh:95-310

mdBook Structure Initialization

The first step creates the mdBook workspace at /workspace/book/ and generates the configuration file book.toml. This establishes the foundation for all subsequent operations.

Directory Structure Creation

The script creates the following directory hierarchy:

/workspace/book/
├── book.toml          (generated configuration)
└── src/               (will contain markdown files)
    ├── SUMMARY.md     (generated in Step 3)
    ├── *.md           (copied in Step 4)
    └── section-*/     (copied in Step 4)

Sources: scripts/build-docs.sh:96-122

book.toml Configuration

The book.toml file is generated dynamically from environment variables. The configuration structure follows the mdBook specification:

The generated configuration:

The git-repository-url setting enables the GitHub icon in the rendered book’s header, providing navigation back to the source repository.

Sources: scripts/build-docs.sh:101-119

SUMMARY.md Generation Process

Step 3 generates src/SUMMARY.md, which defines the book’s table of contents and navigation structure. This is a critical file that mdBook requires to determine page ordering and hierarchy.

graph TB
    WikiDir["/workspace/wiki/\nAll markdown files"]
FindOverview["Find Overview File\ngrep -Ev '^[0-9]'\nFirst non-numbered file"]
FindMain["Find Main Pages\ngrep -E '^[0-9]'\nFiles matching ^[0-9]*.md"]
Sort["Sort Numerically\nsort -t- -k1 -n\nBy leading number"]
CheckSections{"Has Subsections?\nsection-N directory exists?"}
FindSubs["Find Subsections\nls section-N/*.md\nSort numerically"]
ExtractTitle["Extract Title\nhead -1 file.md\nsed 's/^# //'"]
BuildEntry["Build TOC Entry\n- [Title](filename)"]
BuildNested["Build Nested Entry\n- [Title](filename)\n - [Subtitle](section-N/file)"]
Summary["src/SUMMARY.md\nGenerated TOC"]
WikiDir --> FindOverview
 
   WikiDir --> FindMain
 
   FindOverview --> Summary
 
   FindMain --> Sort
 
   Sort --> CheckSections
    
 
   CheckSections -->|No| ExtractTitle
 
   CheckSections -->|Yes| FindSubs
    
 
   ExtractTitle --> BuildEntry
 
   FindSubs --> ExtractTitle
 
   ExtractTitle --> BuildNested
    
 
   BuildEntry --> Summary
 
   BuildNested --> Summary

File Discovery and Sorting

Numeric Sorting Algorithm

The sorting mechanism uses shell built-ins to extract numeric prefixes and sort appropriately:

1. List all .md files in /workspace/wiki/: ls "$WIKI_DIR"/*.md
2. Filter by numeric prefix: grep -E '^[0-9]'
3. Sort using field delimiter - and numeric comparison: sort -t- -k1 -n
4. For each main page, check for subsection directory: section-$section_num
5. If subsections exist, repeat sort for subsection files

This ensures pages appear in the correct order: 1-overview.md, 2-architecture.md, 2.1-subsection.md, etc.

Sources: scripts/build-docs.sh:124-188

For detailed documentation of the SUMMARY.md generation algorithm, see 8.1.

Markdown File Processing

Step 4 copies markdown files from /workspace/wiki/ to /workspace/book/src/ and injects HTML templates into each file. This step bridges the gap between raw markdown and mdBook-ready content.

graph TB
    HeaderTemplate["templates/header.html\nRaw template with variables"]
FooterTemplate["templates/footer.html\nRaw template with variables"]
ProcessTemplate["process-template.py\nVariable substitution"]
EnvVars["Environment Variables\nREPO, BOOK_TITLE, DEEPWIKI_URL\nGITHUB_BADGE_URL, etc."]
HeaderHTML["HEADER_HTML\nProcessed HTML string"]
FooterHTML["FOOTER_HTML\nProcessed HTML string"]
MarkdownFiles["Markdown Files\nsrc/*.md, src/*/*.md"]
Inject["Injection Loop\nfor mdfile in src/*.md"]
TempFile["$mdfile.tmp\nHeader + Content + Footer"]
Replace["mv $mdfile.tmp $mdfile\nReplace original"]
FinalFiles["Final Book Source\nsrc/*.md with templates"]
HeaderTemplate --> ProcessTemplate
 
   FooterTemplate --> ProcessTemplate
 
   EnvVars --> ProcessTemplate
    
 
   ProcessTemplate --> HeaderHTML
 
   ProcessTemplate --> FooterHTML
    
 
   HeaderHTML --> Inject
 
   FooterHTML --> Inject
 
   MarkdownFiles --> Inject
    
 
   Inject --> TempFile
 
   TempFile --> Replace
 
   Replace --> FinalFiles

Template Processing Workflow

Template Variable Substitution

The process-template.py script is invoked twice, once for the header and once for the footer:

The processed HTML strings are then injected into every markdown file through shell redirection.

Sources: scripts/build-docs.sh:190-261

For detailed documentation of template mechanics and customization, see 8.2 and 11.

Mermaid Asset Installation

Step 5 installs the mdbook-mermaid preprocessor assets into the book directory. This step configures the JavaScript libraries and stylesheets required to render Mermaid diagrams in the final HTML output.

Installation Command

The installation is performed by the mdbook-mermaid binary:

This command:

1. Creates theme assets in book/theme/
2. Installs mermaid-init.js for diagram initialization
3. Configures mermaid.js library version
4. Sets up diagram rendering hooks for mdBook’s preprocessor chain

The preprocessor was configured in book.toml during Step 2:

This configuration tells mdBook to run the mermaid preprocessor before generating HTML, which converts mermaid code blocks into rendered diagrams.

Sources: scripts/build-docs.sh:263-266

graph LR
    BookToml["book.toml\nConfiguration"]
SrcDir["src/\nSUMMARY.md\n*.md files\nsection-*/ dirs"]
MdBookBuild["mdbook build\nMain build command"]
Preprocessor["mermaid preprocessor\nConvert mermaid blocks"]
Renderer["HTML renderer\nGenerate pages"]
Assets["Copy static assets\ntheme/, images/"]
Search["Build search index\nsearchindex.js"]
BookOutput["book/\nCompiled HTML"]
BookToml --> MdBookBuild
 
   SrcDir --> MdBookBuild
    
 
   MdBookBuild --> Preprocessor
 
   Preprocessor --> Renderer
 
   Renderer --> Assets
 
   Renderer --> Search
    
 
   Assets --> BookOutput
 
   Search --> BookOutput

Book Build Execution

Step 6 executes the core mdBook build process. This step transforms the prepared markdown files and configuration into a complete HTML documentation website.

mdBook Build Pipeline

Build Command

The build is invoked with no arguments, using the current directory’s configuration:

mdBook automatically:

• Reads book.toml for configuration
• Processes src/SUMMARY.md to determine page structure
• Runs configured preprocessors (mermaid)
• Generates HTML with search index
• Applies the configured theme (rust)
• Creates navigation elements with git repository link

The resulting output is written to book/, relative to the current working directory (/workspace/book/).

Sources: scripts/build-docs.sh:268-271

graph TB
    BookBuild["book/\nBuilt HTML website"]
WikiDir["wiki/\nEnhanced markdown"]
RawDir["raw_markdown/\nPre-enhancement snapshots"]
BookConfig["book.toml\nBuild configuration"]
OutputBook["/output/book/\nDeployable HTML"]
OutputMD["/output/markdown/\nFinal markdown source"]
OutputRaw["/output/raw_markdown/\nDebug snapshots"]
OutputConfig["/output/book.toml\nReference config"]
BookBuild -->|cp -r| OutputBook
 
   WikiDir -->|cp -r| OutputMD
 
   RawDir -->|cp -r if exists| OutputRaw
 
   BookConfig -->|cp| OutputConfig

Output Collection

Step 7 consolidates all build artifacts into the /output/ directory, which is typically mounted as a volume for access from the host system.

Output Artifacts and Layout

Copy Operations

The script performs four copy operations:

The /output/book/ directory is immediately servable as a static website:

Sources: scripts/build-docs.sh:273-309

graph TB
    Phase1["Phase 1:\nMarkdown Extraction"]
Phase2["Phase 2:\nDiagram Enhancement"]
CheckMode{"MARKDOWN_ONLY\n== 'true'?"}
CopyMD["Copy wiki/ to\n/output/markdown/"]
CopyRaw["Copy raw_markdown/ to\n/output/raw_markdown/"]
Exit["Exit with success\nSkip Phase 3"]
Phase3["Phase 3:\nmdBook Build"]
Phase1 --> Phase2
 
   Phase2 --> CheckMode
    
 
   CheckMode -->|Yes| CopyMD
 
   CopyMD --> CopyRaw
 
   CopyRaw --> Exit
    
 
   CheckMode -->|No| Phase3

Markdown-Only Mode

The MARKDOWN_ONLY environment variable provides an escape hatch that bypasses Phase 3 entirely. When set to "true", the script exits after Phase 2 (diagram enhancement) without building the HTML book.

Markdown-Only Workflow

This mode is useful for:

• Debugging diagram placement without HTML build overhead
• Consuming markdown in alternative build systems
• Inspecting intermediate transformation results
• CI/CD pipelines that only need markdown output

Sources: scripts/build-docs.sh:67-93

For more details on markdown-only mode, see 12.1.

graph TB
    subgraph "Shell Script Orchestration"
        BuildScript["build-docs.sh\nMain orchestrator"]
Step2Func["Lines 95-122\nmkdir, cat > book.toml"]
Step3Func["Lines 124-188\nSUMMARY.md generation"]
Step4Func["Lines 190-261\ncp, template injection"]
Step5Func["Lines 263-266\nmdbook-mermaid install"]
Step6Func["Lines 268-271\nmdbook build"]
Step7Func["Lines 273-295\ncp outputs"]
end
    
    subgraph "External Tools"
        MdBook["mdbook\nRust binary\n/usr/local/bin/mdbook"]
MdBookMermaid["mdbook-mermaid\nPreprocessor binary\n/usr/local/bin/mdbook-mermaid"]
ProcessTemplate["process-template.py\nPython script\n/usr/local/bin/process-template.py"]
end
    
    subgraph "Key Files"
        BookToml["book.toml\nGenerated at line 102"]
SummaryMd["src/SUMMARY.md\nGenerated at line 130-186"]
HeaderHtml["templates/header.html\nInput template"]
FooterHtml["templates/footer.html\nInput template"]
end
    
    subgraph "Directory Structures"
        BookDir["/workspace/book/\nBuild workspace"]
SrcDir["/workspace/book/src/\nMarkdown sources"]
OutputDir["/output/\nVolume mount point"]
end
    
 
   BuildScript --> Step2Func
 
   Step2Func --> Step3Func
 
   Step3Func --> Step4Func
 
   Step4Func --> Step5Func
 
   Step5Func --> Step6Func
 
   Step6Func --> Step7Func
    
 
   Step2Func --> BookToml
 
   Step2Func --> BookDir
 
   Step3Func --> SummaryMd
 
   Step4Func --> ProcessTemplate
 
   Step4Func --> SrcDir
 
   Step5Func --> MdBookMermaid
 
   Step6Func --> MdBook
 
   Step7Func --> OutputDir
    
 
   HeaderHtml --> ProcessTemplate
 
   FooterHtml --> ProcessTemplate

Phase 3 Component Map

This diagram maps Phase 3 concepts to their concrete implementations in the codebase:

Sources: scripts/build-docs.sh:95-310

Phase 3 Error Handling

Phase 3 operates under set -e mode, causing immediate script termination on any command failure. Common failure scenarios:

All errors are reported to stderr and result in a non-zero exit code due to set -e.

Sources: scripts/build-docs.sh2 scripts/build-docs.sh:95-310

Summary

Phase 3 transforms enhanced markdown into a searchable HTML documentation website through six orchestrated steps:

1. Structure Initialization : Creates mdBook workspace and book.toml configuration
2. SUMMARY.md Generation : Builds navigation structure with numeric sorting
3. Markdown Processing : Injects templates and copies files to book source
4. Mermaid Installation : Configures diagram rendering support
5. Book Build : Executes mdBook to generate HTML
6. Output Collection : Consolidates artifacts to /output/ volume

Phase 3 is entirely orchestrated by build-docs.sh and coordinates shell utilities, Python scripts, and Rust binaries to produce the final documentation website. The process is deterministic and idempotent, generating consistent output from the same input markdown.

Sources: scripts/build-docs.sh:95-310

Dismiss

Refresh this wiki

Enter email to refresh

SUMMARY.md Generation

Loading…

SUMMARY.md Generation

Relevant source files

• scripts/build-docs.sh

Purpose and Scope

This document explains how the SUMMARY.md file is dynamically generated from the scraped markdown content structure. The SUMMARY.md file serves as mdBook’s table of contents, defining the navigation structure and page hierarchy for the generated HTML documentation.

For information about how the markdown files are initially organized during scraping, see Wiki Structure Discovery. For details about the overall mdBook build configuration, see Configuration Generation.

SUMMARY.md in mdBook

The SUMMARY.md file is mdBook’s primary navigation document. It defines:

• The order of pages in the documentation
• The hierarchical structure (chapters and sub-chapters)
• The titles displayed in the navigation sidebar
• Which markdown files map to which sections

mdBook parses SUMMARY.md to construct the entire book structure. Pages not listed in SUMMARY.md will not be included in the generated documentation.

Sources: build-docs.sh:108-161

Generation Process Overview

The SUMMARY.md generation occurs in Step 3 of the build pipeline build-docs.sh:124-188 after markdown extraction is complete but before the mdBook build begins. The generation algorithm automatically discovers the file structure, applies numeric sorting to section files, and constructs a hierarchical table of contents.

SUMMARY.md Generation Pipeline

flowchart TD
    Start["Start Step 3:\nbuild-docs.sh:126"]
Init["Write Header:\necho '# Summary'\nLines 129-131"]
ListFiles["List all files:\nmain_pages_list=$(ls $WIKI_DIR/*.md)"]
FindOverview["Find overview_file:\ngrep -Ev '^[0-9]' /head -1 Lines 136-138"]
HasOverview{"overview_file exists?"}
ExtractOvTitle["Extract title: head -1/ sed 's/^# //'\nLine 140"]
WriteOv["Write: [{title}]($overview_file)\nLines 141-143"]
RemoveOv["Filter overview_file from list\nLine 143"]
NumericSort["Numeric Sort:\ngrep -E '^[0-9]' /sort -t- -k1 -n Lines 147-155"]
IterateMain["for file in main_pages Line 158"]
ExtractTitle["title=$ head -1/ sed 's/^# //')"]
ExtractSecNum["section_num=$(grep -oE '^[0-9]+')"]
CheckSecDir{"[ -d section-$section_num ]"}
WriteSec["echo '- [$title]($filename)'\nLine 171"]
IterateSub["ls section-$section_num/*.md /sort -t- -k1 -n Lines 174-180"]
WriteSub["echo ' - [$subtitle] section-$section_num/$subfilename '"]
WriteStandalone["echo '- [$title] $filename ' Line 183"]
Complete["Redirect to: src/SUMMARY.md Line 186"]
LogCount["Log entry count: grep -c '\\[' src/SUMMARY.md Line 188"]
End["End Step 3"]
Start --> Init
 Init --> ListFiles
 ListFiles --> FindOverview
 FindOverview --> HasOverview
 HasOverview -->|Yes|ExtractOvTitle
 ExtractOvTitle --> WriteOv
 WriteOv --> RemoveOv
 RemoveOv --> NumericSort
 HasOverview -->|No|NumericSort
 NumericSort --> IterateMain
 IterateMain --> ExtractTitle
 ExtractTitle --> ExtractSecNum
 ExtractSecNum --> CheckSecDir
 CheckSecDir -->|Yes: Has subsections|WriteSec
 WriteSec --> IterateSub
 IterateSub --> WriteSub
 WriteSub --> IterateMain
 CheckSecDir -->|No: Standalone|WriteStandalone
 WriteStandalone --> IterateMain
 IterateMain -->|Done| Complete
 
   Complete --> LogCount
 
   LogCount --> End

Sources: build-docs.sh:124-188

The algorithm executes three key phases:

Sources: build-docs.sh:124-188

Algorithm Components

Step 1: Overview File Selection

The algorithm identifies a special overview file by searching for files without numeric prefixes. This file becomes the introduction page, written before the numbered sections.

Overview File Detection Algorithm

flowchart TD
    Start["main_pages_list =\nls $WIKI_DIR/*.md"]
Filter["overview_file =\nawk -F/ '{print $NF}' /grep -Ev '^[0-9]'/\nhead -1"]
Check{"overview_file\nnot empty?"}
Verify{"File exists?\n[ -f $WIKI_DIR/$overview_file ]"}
Extract["title=$(head -1 $WIKI_DIR/$overview_file /sed 's/^# //'"]
Write["echo '[{title:-Overview}] $overview_file ' echo ''"]
Remove["main_pages_list = grep -v $overview_file"]
Continue["Proceed to numeric sorting"]
Start --> Filter
 Filter --> Check
 Check -->|Yes|Verify
 Check -->|No|Continue
 Verify -->|Yes|Extract
 Verify -->|No| Continue
 
   Extract --> Write
 
   Write --> Remove
 
   Remove --> Continue

Sources: build-docs.sh:133-145

Detection Logic:

The overview file is then excluded from main_pages_list before numeric sorting build-docs.sh143

Sources: build-docs.sh:133-145

Step 2: Numeric Sorting Pipeline

After overview extraction, remaining files are sorted numerically by their leading number prefix. This ensures pages appear in logical order (e.g., 2-start.md before 10-advanced.md).

Numeric Sorting Implementation

flowchart LR
    Input["main_pages_list\n(filtered from overview)"]
Basename["awk -F/ '{print $NF}'\nExtract filename"]
GrepNum["grep -E '^[0-9]'\nKeep only numbered files"]
NumSort["sort -t- -k1 -n\nNumeric sort on first field"]
Reconstruct["while read fname; do\n echo $WIKI_DIR/$fname\ndone"]
Output["main_pages\n(sorted file paths)"]
Input --> Basename
 
   Basename --> GrepNum
 
   GrepNum --> NumSort
 
   NumSort --> Reconstruct
 
   Reconstruct --> Output

Sources: build-docs.sh:147-155

Sort Command Breakdown:

Example Sorting:

Sources: build-docs.sh:147-155

Step 3: Subsection Detection and Iteration

For each main page, the algorithm extracts the numeric prefix (section_num) and checks for a corresponding section-N/ directory. If found, all subsection files are written with 2-space indentation.

Subsection Detection and Writing Flow

flowchart TD
    LoopStart["for file in main_pages\nLine 158"]
FileCheck["[ -f $file ] // continue"]
GetBasename["filename=$(basename $file)"]
ExtractTitle["title=$(head -1 $file /sed 's/^# //' Line 163"]
ExtractNum["section_num=$ echo $filename/\ngrep -oE '^[0-9]+' // true)\nLine 166"]
BuildPath["section_dir=$WIKI_DIR/section-$section_num\nLine 167"]
CheckBoth{"[ -n $section_num ] &&\n[ -d $section_dir ]"}
WriteMain["echo '- [$title]($filename)'\nLine 171"]
ListSubs["ls $section_dir/*.md 2>/dev/null /awk -F/ '{print $NF}'/\nsort -t- -k1 -n\nLines 174-175"]
SubLoop["while read subname\nLine 174"]
SubCheck["[ -f $section_dir/$subname ] // continue"]
SubBasename["subfilename=$(basename $subfile)"]
SubTitle["subtitle=$(head -1 $subfile /sed 's/^# //' Line 178"]
SubWrite["echo ' - [$subtitle] section-$section_num/$subfilename ' Line 179"]
WriteStandalone["echo '- [$title] $filename ' Line 183"]
NextFile["Continue loop"]
LoopStart --> FileCheck
 FileCheck --> GetBasename
 GetBasename --> ExtractTitle
 ExtractTitle --> ExtractNum
 ExtractNum --> BuildPath
 BuildPath --> CheckBoth
 CheckBoth -->|Yes: Has subsections|WriteMain
 WriteMain --> ListSubs
 ListSubs --> SubLoop
 SubLoop --> SubCheck
 SubCheck --> SubBasename
 SubBasename --> SubTitle
 SubTitle --> SubWrite
 SubWrite --> SubLoop
 SubLoop -->|Done|NextFile
 CheckBoth -->|No: Standalone| WriteStandalone
 
   WriteStandalone --> NextFile
    
 
   NextFile --> LoopStart

Sources: build-docs.sh:158-185

Variable Mapping:

Sources: build-docs.sh:158-185

Step 4: Subsection Numeric Sorting

Subsection files within section-N/ directories undergo the same numeric sorting as main pages, ensuring proper ordering (e.g., 5.2 before 5.10).

Subsection Sorting Pipeline

flowchart LR
    Dir["section_dir/\nsection-5/"]
List["ls $section_dir/*.md 2>/dev/null"]
Awk["awk -F/ '{print $NF}'"]
Sort["sort -t- -k1 -n"]
Loop["while read subname"]
Verify["[ -f $subfile ] // continue"]
Extract["subtitle=$(head -1 / sed 's/^# //')"]
Write["echo ' - [$subtitle](section-N/$subfilename)'"]
Dir --> List
 
   List --> Awk
 
   Awk --> Sort
 
   Sort --> Loop
 
   Loop --> Verify
 
   Verify --> Extract
 
   Extract --> Write
 
   Write --> Loop

Sources: build-docs.sh:174-180

Key Implementation Details:

Sorting Example:

Input files:        After sort:         SUMMARY.md output:
section-5/          section-5/          - <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/Component Reference" undefined  file-path="Component Reference">Hii</FileRef>
├── 5.10-tools.md   ├── 5.1-build.md      - <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/build-docs.sh" undefined  file-path="build-docs.sh">Hii</FileRef>
├── 5.2-scraper.md  ├── 5.2-scraper.md    - <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/Scraper" undefined  file-path="Scraper">Hii</FileRef>
└── 5.1-build.md    └── 5.10-tools.md     - <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/Tools" undefined  file-path="Tools">Hii</FileRef>

Sources: build-docs.sh:174-180

File Structure Conventions

The generation algorithm depends on the file structure created during markdown extraction (see Wiki Structure Discovery):

Diagram: File Structure Conventions for SUMMARY.md Generation

Sources: build-docs.sh:126-158

Title Extraction Method

All page titles are extracted using a consistent pattern:

This assumes that every markdown file begins with a level-1 heading (# Title). The sed command removes the # prefix, leaving only the title text.

Extraction Pipeline:

Sources: build-docs.sh120 build-docs.sh134 build-docs.sh150

Output Format

The generated SUMMARY.md follows mdBook’s syntax:

Format Rules:

Sources: build-docs.sh:113-159

flowchart TD
    Step3["Step 3 Comment\nLine 124-126"]
Header["Write Header Block\nLines 129-131\n{\n echo '# Summary'\n echo ''\n}"]
Overview["Overview Detection\nLines 133-145\nmain_pages_list=$(ls)\noverview_file=$(grep -Ev '^[0-9]')\nif [ -n $overview_file ]; then\n title=$(head -1)\n echo '[$title]($overview_file)'\n main_pages_list=$(grep -v $overview_file)\nfi"]
NumSort["Numeric Sort Block\nLines 147-155\nmain_pages=$(\n printf '%s' $main_pages_list\n /awk -F/ '{print $NF}'/ grep -E '^[0-9]'\n /sort -t- -k1 -n/ while read fname; do\n echo $WIKI_DIR/$fname\n done\n)"]
MainLoop["Main Page Loop\nLines 158-185\necho $main_pages /while read file do filename=$ basename $file title=$ head -1/ sed 's/^# //')\n section_num=$(grep -oE '^[0-9]+')\n section_dir=$WIKI_DIR/section-$section_num\n if [ -n $section_num ] && [ -d $section_dir ]; then\n ...\n fi\ndone"]
Subsection["Subsection Block\nLines 174-180\nls $section_dir/*.md\n /awk -F/ '{print $NF}'/ sort -t- -k1 -n\n / while read subname; do\n subtitle=$(head -1)\n echo ' - [$subtitle](...)'\n done"]
Redirect["Output Redirection\nLine 186\n} > src/SUMMARY.md"]
Log["Entry Count Log\nLine 188\necho 'Generated SUMMARY.md with\n$(grep -c '\\[' src/SUMMARY.md)
entries'"]
Step3 --> Header
 
   Header --> Overview
 
   Overview --> NumSort
 
   NumSort --> MainLoop
 
   MainLoop --> Subsection
 
   Subsection --> MainLoop
 
   MainLoop --> Redirect
 
   Redirect --> Log

Implementation Code Mapping

The SUMMARY.md generation is implemented in a single contiguous block within build-docs.sh. The following diagram maps algorithm phases to specific line ranges:

Code Structure and Execution Flow

Sources: build-docs.sh:124-188

Shell Variable Reference:

Sources: build-docs.sh28 build-docs.sh:124-188

Generation Statistics and Output

After generation completes, the script logs statistical information about the generated SUMMARY.md file:

Entry Counting Logic

Sources: build-docs.sh188

The grep -c command counts lines containing the <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/ character, which appears in every markdown link. This count includes#LNaN-LNaN“ NaN file-path=“ character, which appears in every markdown link. This count includes">Hii</FileRef> | +1 | | Main pages | - <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/Title" undefined file-path="Title">Hii</FileRef> | +1 per main page | | Subsections | - <FileRef file-url="https://github.com/jzombie/deepwiki-to-mdbook/blob/0378ae61/Title" undefined file-path="Title">Hii</FileRef> | +1 per subsection |

Example Output:

Generated SUMMARY.md with 23 entries

This indicates the book contains 23 total navigation links (overview + main pages + all subsections combined).

Sources: build-docs.sh188

Integration with mdBook Build

The generated src/SUMMARY.md file is used directly by mdBook during the build process (Step 6):

1. mdBook reads src/SUMMARY.md to determine book structure
2. For each entry, mdBook looks up the corresponding markdown file in src/
3. Files are processed in the order they appear in SUMMARY.md
4. The navigation sidebar reflects the hierarchy defined in SUMMARY.md

The generation happens in build-docs.sh:108-161 markdown files are copied to src/ in build-docs.sh166 and the mdBook build executes in build-docs.sh176

Sources: build-docs.sh:108-176

Dismiss

Refresh this wiki

Enter email to refresh

Template Injection

Loading…

Template Injection