---
title: parallel | CodeWeaver Docs
description: API reference for codeweaver.engine.chunker.parallel
url: "https://docs.knitli.com/api/engine/chunker/parallel"
type: static
generatedAt: "2026-04-17T17:21:08.428Z"
---

# parallel
       [Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F.%20I%20want%20to%20ask%20questions%20about%20it.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F.%20I%20want%20to%20ask%20questions%20about%20it.)[View in Markdown](/codeweaver/api/engine/chunker/parallel.md)       [Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F)[Share on X](https://x.com/intent/tweet?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F&text=parallel)[Share on Threads](https://threads.net/intent/post?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F&text=parallel)[Share on Bluesky](https://bsky.app/intent/compose?text=parallel%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F)[Share on Reddit](https://reddit.com/submit?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F&title=parallel)[Share on Hacker News](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F&t=parallel)[Share on Email](mailto:?subject=parallel&body=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F)[Share on WhatsApp](https://wa.me/?text=parallel%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F)[Share on Telegram](https://t.me/share/url?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fparallel%2F&text=parallel)
# `codeweaver.engine.chunker.parallel`
[Section titled “codeweaver.engine.chunker.parallel”](#codeweaverenginechunkerparallel)
Parallel chunking operations for efficient multi-file processing.

This module provides parallel chunking capabilities using ProcessPoolExecutor or ThreadPoolExecutor based on configuration settings. It enables efficient processing of large codebases by distributing file chunking across multiple workers while maintaining error isolation and memory efficiency.

Key Features:

 - Process-based or thread-based execution (configurable)
 - Independent file processing with error isolation
 - Memory-efficient iterator pattern
 - Graceful error handling with detailed logging
 - Automatic chunker selection per file

Architecture:

 - Uses ChunkerSelector for intelligent chunker routing
 - Creates fresh chunker instances in each worker
 - Yields results as (Path, list[CodeChunk]) tuples
 - Logs errors but continues processing remaining files

Performance Considerations:

 - Process-based: Better for CPU-bound parsing (default)
 - Thread-based: Better for I/O-bound operations
 - Memory usage scales with max_workers setting
 - Iterator pattern prevents loading all results in memory

## Function: `chunk_files_parallel`
[Section titled “Function: chunk_files_parallel”](#function-chunk_files_parallel)

**

```
chunk_files_parallel()
```

Chunk multiple files in parallel using process or thread pool.

Distributes file chunking across multiple workers for efficient processing of large codebases. Uses an iterator pattern to yield results as they complete, preventing memory exhaustion from loading all chunks at once.

Args: files: List of DiscoveredFile objects to chunk governor: ChunkGovernor providing resource limits and configuration max_workers: Maximum number of parallel workers. If None, uses settings from governor or defaults to CPU count. For process executor, limited to available CPUs. For thread executor, can exceed CPU count. executor_type: Type of executor to use - “process” or “thread” or None. If None, uses settings from governor or defaults to “process”. Process-based is better for CPU-bound parsing, thread-based for I/O. tokenizer: Optional tokenizer for accurate token counting

Yields: Tuples of (file_path, chunks) for successfully chunked files. Files that fail to chunk are logged but not yielded.

## Function: `chunk_files_parallel_dict`
[Section titled “Function: chunk_files_parallel_dict”](#function-chunk_files_parallel_dict)

**

```
chunk_files_parallel_dict()
```

Chunk multiple files in parallel and return as dictionary.