📊 Python Document Intelligence Framework CPU Benchmarks

🎯 Executive Summary

Framework Performance Rankings

Success rates calculated on files actually tested by each framework. "—" indicates categories not included in this benchmark run. License details available in the Framework Details section below.

Memory Usage by Category

Memory usage shown as peak RSS (Resident Set Size) in MB during extraction

📊 Performance Analysis

1️⃣ Extraction Speed Rankings

🏆 Speed Champions (files/sec):
Multi-format frameworks showing consistent performance across all supported file types. Rankings based on current benchmark data.

📊 Performance Chart Not Available

Performance comparison data is available in the reports below.

Speed Analysis: Kreuzberg leads with 15+ files/sec, while Docling shows timeout issues on complex documents

2️⃣ Data Throughput Analysis

📊 Throughput Performance (MB/sec):
Measures actual data processing speed accounting for file sizes. Higher values indicate better scaling with document complexity.

📊 Throughput Chart Not Available

Throughput analysis data is available in the reports below.

Throughput Insights: Multi-format frameworks show consistent performance across diverse document types

3️⃣ Success Rate Reliability

✅ Reliability Rankings (% successful):
Framework reliability varies by document type and format support. See charts for detailed comparisons.
*Success rates calculated on supported formats only

📊 Success Rate Chart Not Available

Success rate data is available in the reports below.

Reliability Notes: Success rates calculated only on files each framework attempts to process

💾 Resource Usage Analysis

1️⃣ Memory Usage Rankings by Framework

🏆 Memory Efficiency Ranking (Lower MB = Better):
Memory usage varies significantly by framework and document type. See detailed analysis below.

📊 Memory Usage Chart Not Available

Memory usage data is available in the reports below.

Interpretation: Shows average peak memory consumption across all file types. Lower bars indicate more memory-efficient frameworks.

2️⃣ Detailed Memory Usage by File Type

📊 Format-Specific Memory Patterns:
• PDFs: Show highest memory variance (50MB - 2GB+)
• Images: Consistent high memory usage across frameworks
• Office Docs: Moderate memory requirements (200-800MB)
• Text/Markup: Lowest memory footprint (<100MB)

📊 Memory Analysis Available

Detailed memory profiling data is available in the interactive dashboard and detailed reports.

View Interactive Memory Analysis

Framework Behavior: Each framework shows distinct memory patterns per file type. Frameworks optimized for specific formats use significantly less memory on their target documents.

3️⃣ Performance by Document Size Categories

📏 Size Category Performance (Speed Ranking):
Tiny (<100KB): Fast extraction | Small (100KB-1MB): Consistent performance | Medium (1-10MB): Mixed results | Large (10-50MB): Framework timeouts common

📊 Category Analysis Chart Not Available

Category analysis data is available in the reports below.

Size Scaling: Performance patterns change dramatically with document size. Memory usage can increase exponentially for complex documents regardless of file size.

4️⃣ Installation Size Comparison

💿 Installation Footprint Ranking (Smaller = Better):
Framework installation sizes range from under 100MB to over 1GB depending on dependencies.

📦 Installation Size Analysis

Framework installation sizes range from 71MB to 1GB+ depending on dependencies:

Lightweight: Kreuzberg (71MB)

Moderate: Extractous (100MB), Unstructured (146MB)

Heavy: MarkItDown (251MB), Docling (1GB+)

Trade-offs: Larger installations often include ML models and extensive format support, while smaller frameworks focus on specific use cases.

📈 Key Memory Usage Insights

• 🔬 Measurement Method: RSS (Resident Set Size) tracked at 50ms intervals using psutil
• 📊 Framework Rankings: Memory efficiency varies by framework and use case - see charts for details
• 📄 Per-Format Variance: Memory usage patterns vary 10-50x between file types
• 📏 Size Correlation: Memory scales with document complexity, not just file size
• 🎯 Optimization Opportunities: Framework-format matching can reduce memory usage by 5-10x

📄 Format Support Analysis

📋 Metadata Extraction Analysis

1️⃣ Metadata Coverage by Framework

📊 Metadata Extraction Leaders:
Frameworks vary significantly in metadata extraction capabilities. Multi-format tools provide comprehensive coverage across diverse document types.

📊 Metadata Analysis Available

Comprehensive metadata extraction analysis covering 20+ fields per document type.

View Analysis

Coverage Analysis: Shows percentage of metadata fields successfully extracted by each framework across all document types.

2️⃣ Field Extraction Comparison

📋 Metadata Field Types:
• Document Properties: Title, author, creation/modification dates
• Content Metrics: Page count, word count, character count
• Technical Data: MIME type, encoding, compression info
• Quality Indicators: Language detection, format version

📋 Field Comparison Data

Detailed field-by-field comparison showing which frameworks extract specific metadata types.

Download CSV

Field Analysis: Compares specific metadata field extraction capabilities across frameworks, highlighting strengths and gaps.

🔍 Metadata Extraction Capabilities

• Document Properties: Title, author, creation/modification dates, language detection
• Content Metrics: Page count, word count, character count, document structure
• Technical Metadata: MIME type, file format version, encoding, compression
• Quality Assessment: Completeness scores, field coverage analysis, value examples
• Framework Comparison: Coverage percentage, unique fields per framework, extraction reliability

📈 Key Metadata Insights

• Coverage Variance: Frameworks extract different metadata fields with varying completeness
• Format Specialization: Different frameworks excel at different metadata types based on their design focus
• Quality Correlation: Better metadata extraction often indicates higher text extraction quality
• Language Detection: Multi-language document handling varies significantly across frameworks

✨ Quality Assessment Analysis

1️⃣ Quality Scores by Framework

🏆 Quality Rankings (Higher Score = Better):
Quality assessment provides ML-based scoring for extraction accuracy, coherence, and completeness across all tested frameworks and file types.

📊 Quality Assessment Available

Quality assessment requires enabling during benchmark execution:

uv run python -m src.cli benchmark --enable-quality-assessment

View Quality Data

Quality Metrics: Combines extraction completeness, text coherence, semantic similarity, and document-specific quality checks.

2️⃣ Readability Analysis

📖 Readability Metrics:
• Flesch Reading Ease: Higher scores = easier to read
• Gunning Fog Index: Lower scores = more accessible text
• Sentence Structure: Analysis of complexity and coherence

📖 Readability Analysis

Readability metrics computed using Flesch Reading Ease and Gunning Fog Index for extracted text quality assessment.

Enable quality assessment in benchmarks to generate readability charts

uv run python -m src.cli benchmark --enable-quality-assessment

Text Quality: Measures how well frameworks preserve readable, coherent text structure during extraction.

🔬 Quality Assessment Capabilities

• ML-Based Scoring: Sentence transformer models for semantic similarity and coherence analysis
• Readability Metrics: Flesch Reading Ease, Gunning Fog Index, average sentence/word length
• Content Quality: Extraction completeness, text coherence, noise ratio, gibberish detection
• Structural Analysis: Title detection, formatting preservation, table structure quality
• Document-Specific Checks: PDF page integrity, HTML tag removal quality, Word formatting preservation

📈 Quality Scoring Methodology

• Overall Quality Score (0-1): Weighted combination of multiple quality dimensions
• Extraction Completeness (25%): Estimated content coverage and missing information
• Text Coherence (20%): Sentence structure and logical flow preservation
• Semantic Similarity (20%): Meaning preservation compared to reference texts
• Readability (15%): Human readability and comprehension scores
• Structural Quality (20%): Format-specific quality checks and noise reduction

🎯 Key Quality Insights

• Framework Specialization: Quality scores vary by format - frameworks excel in their target document types
• Speed vs Quality Trade-off: Fastest frameworks may sacrifice some quality for performance
• OCR Quality Impact: Image-based documents show higher quality variance across frameworks
• Language Dependency: Quality scores affected by document language and OCR language configuration

📄 Format Support Analysis

Format Categories Tested

• Documents: PDF, DOCX, PPTX, XLSX, XLS, ODT
• Web/Markup: HTML, MD, RST, ORG
• Images: PNG, JPG, JPEG, BMP
• Email: EML, MSG
• Data: CSV, JSON, YAML
• Text: TXT

Framework Capabilities

• Multi-Format Frameworks:
• Kreuzberg 3.8.0: All formats except MSG (no open source support)
• Docling: PDF, DOCX, XLSX, PPTX, HTML, CSV, MD, AsciiDoc, Images (PNG, JPEG, TIFF, BMP, WEBP)
• MarkItDown: Comprehensive office and web formats
• Unstructured: 64+ file types including enterprise formats
• Extractous: Rust-based performance across common formats

🔧 Framework Details

📋 Detailed Reports & Data

🌐 HTML Report | 📝 Markdown Report | 📊 JSON Metrics | 📊 Summary Data

🔬 Advanced Analysis

Additional analysis modules are available in the detailed reports section above.

📊 Table Extraction Analysis

Specialized analysis of table detection and extraction capabilities across frameworks, focusing on structure preservation, cell accuracy, and formatting retention.

Table Detection Performance

📊 Table detection analysis available for documents with table content

View Analysis

Structure Preservation Quality

📋 Table structure analysis data available in JSON format

Browse Analysis

🔍 Table Extraction Capabilities

• Table Detection: Automatic identification of tabular content in documents
• Structure Preservation: Maintenance of row/column relationships and cell boundaries
• Content Accuracy: Correct extraction of cell content without OCR errors
• Format Support: Table extraction from PDF, DOCX, HTML, and spreadsheet formats
• Complex Layouts: Handling of merged cells, nested tables, and formatting

💾 Memory Profiling Data Available

• Peak Memory Tracking: psutil RSS measurements at 50ms intervals for every extraction
• Per-File-Type Memory: Memory usage breakdown by PDF, DOCX, HTML, images, etc.
• Size Category Analysis: Memory scaling from tiny (100KB) to huge (50MB+) documents
• Framework Memory Profiles: From Kreuzberg's 71MB to Docling's 1.7GB+ peak usage
• Memory Efficiency Metrics: MB/second throughput and memory-per-character ratios

📊 Per-File-Type Performance Analysis

Detailed per-file-type performance data is available in the benchmark reports above.

🔬 Performance Methodology by File Type

• PDF Documents: Tested with both text-based and image-based PDFs, including complex layouts and tables
• Office Documents: DOCX, PPTX, XLSX with varying complexity, embedded images, and formatting
• Web Content: HTML with CSS styling, JavaScript content, and embedded multimedia
• Images: OCR processing of screenshots, scanned documents, and rotated text
• Email Formats: EML and MSG with attachments, HTML content, and threading
• Data Formats: Structured CSV, JSON, YAML with varying sizes and nesting

📐 Performance Metrics Breakdown

• Extraction Time: Wall-clock time from file read to text output completion
• Memory Usage: Peak RSS memory consumption during extraction process
• Success Rate: Percentage of files successfully processed without errors or timeouts
• Throughput: Files per second and MB per second processing rates
• Text Quality: Character/word counts, readability scores, coherence metrics

🎯 Key Insights from File-Type Analysis

• Framework Specialization: Each framework has strengths in different file types and use cases
• Format-Specific Optimization: Frameworks show 10-100x performance differences in their specialty areas
• OCR Processing Costs: Image extraction consumes 10-50x more memory and time than text documents
• Scaling Behavior: Performance degrades differently by file size depending on document complexity
• Error Patterns: Framework failures cluster around specific file types and size thresholds

Links

• 📂 GitHub Repository
• ⚙️ GitHub Actions