---
title: base | CodeWeaver Docs
description: API reference for codeweaver.engine.chunker.base
url: "https://docs.knitli.com/api/engine/chunker/base"
type: static
generatedAt: "2026-04-17T17:21:08.359Z"
---

# base
       [Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F.%20I%20want%20to%20ask%20questions%20about%20it.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F.%20I%20want%20to%20ask%20questions%20about%20it.)[View in Markdown](/codeweaver/api/engine/chunker/base.md)       [Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F)[Share on X](https://x.com/intent/tweet?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F&text=base)[Share on Threads](https://threads.net/intent/post?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F&text=base)[Share on Bluesky](https://bsky.app/intent/compose?text=base%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F)[Share on Reddit](https://reddit.com/submit?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F&title=base)[Share on Hacker News](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F&t=base)[Share on Email](mailto:?subject=base&body=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F)[Share on WhatsApp](https://wa.me/?text=base%20https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F)[Share on Telegram](https://t.me/share/url?url=https%3A%2F%2Fdocs.knitli.com%2Fcodeweaver%2Fapi%2Fengine%2Fchunker%2Fbase%2F&text=base)
# `codeweaver.engine.chunker.base`
[Section titled “codeweaver.engine.chunker.base”](#codeweaverenginechunkerbase)
Base chunker services and definitions.

CodeWeaver has a robust chunking system that allows it to extract meaningful information from any codebase. Chunks are created based on a graceful degradation strategy:

 1. **Semantic Chunking**: When we have a tree-sitter grammar for a language (there are currently 26 supported languages, see `codeweaver.language.SemanticSearchLanguage`), semantic chunking is the primary strategy.
 1. **Delimiter Chunking**: If semantic chunking isn’t available or fails (e.g., parse errors, oversized nodes without chunkable children), we fall back to delimiter-based chunking using language-specific patterns.
 1. **Generic Fallback**: If delimiter patterns don’t match, we use generic delimiters (braces, newlines, etc.) to ensure we can always produce chunks.

This multi-tiered approach ensures reliable chunking across 170+ languages while maintaining semantic quality for supported languages.

## Class: `AdaptiveChunkBehavior`
[Section titled “Class: AdaptiveChunkBehavior”](#class-adaptivechunkbehavior)
Actions for adaptive chunk sizing.

The adaptive chunking system classifies each chunk by size and determines the appropriate action:

 - KEEP: Chunk is within acceptable range, use as-is
 - MERGE: Chunk is too small, should combine with neighbors
 - TRY_CHILDREN: Chunk exceeds optimal, try semantic split first
 - FORCE_SPLIT: Chunk exceeds hard limit, must split mechanically

## Class: `BaseChunker`
[Section titled “Class: BaseChunker”](#class-basechunker)
Base class for chunkers.

### Method: `chunk`
[Section titled “Method: chunk”](#method-chunk)

**

```
chunk()
```

Chunk the given content into code chunks using `self._governor` settings.

Args: content: The text content to chunk. file: The DiscoveredFile object containing file metadata and source_id. context: Additional context for chunking.

Returns: List of CodeChunk objects with source_id from the DiscoveredFile.

## Class: `ChunkGovernor`
[Section titled “Class: ChunkGovernor”](#class-chunkgovernor)
Configuration for chunking behavior.

### Method: `classify_chunk_size`
[Section titled “Method: classify_chunk_size”](#method-classify_chunk_size)

**

```
classify_chunk_size()
```

Determine what action to take for a chunk of given size.

Args: tokens: Estimated token count of the chunk

Returns: AdaptiveChunkBehavior indicating the recommended action:

 - MERGE: tokens < floor_tokens (too small, combine with neighbors)
 - KEEP: floor_tokens <= tokens <= acceptable_max_tokens (good size)
 - TRY_CHILDREN: acceptable_max_tokens < tokens <= chunk_limit (try semantic split)
 - FORCE_SPLIT: tokens > chunk_limit (must split mechanically)

### Method: `from_settings`
[Section titled “Method: from_settings”](#method-from_settings)

**

```
from_settings()
```

Create a ChunkGovernor from ChunkerSettings.

Args: settings: The ChunkerSettings to create the governor from.

Returns: A ChunkGovernor instance.