Extract any document.
Chunk it for AI.
Transformations.Text extracts clean text from 14 document formats and splits it into overlapping chunks sized for vector-store ingestion — with format-specific metadata, async streaming, and DI registration included.
.pdf → PdfPigRoute by extension. No switch statement.
Pass a filename and a stream. The extractor resolves the right parser automatically, normalizes whitespace, handles BOM and encoding detection, and returns a typed success/failure result.
Source code extraction covers 25 languages: C#, Python, JavaScript, TypeScript, C/C++, Java, PowerShell, Bash, Go, Rust, SQL, Ruby, PHP, Swift, Kotlin, Scala, F#, VB, Lua, R, MATLAB, Dart, and more. Plain text handles any remaining extension as fallback.
The right split for every document type.
Every strategy returns IReadOnlyList<TextChunk> with index, text, start offset, and end offset — ready to zip into your vector-store batch.
ChunkByCharacters
Fixed-size windows with configurable overlap. The safest choice when token budget is the primary constraint and document structure does not matter.
ChunkBySentences
Accumulates sentences until maxLength would be exceeded, then starts a new chunk. Preserves sentence boundaries so no sentence is split mid-thought.
ChunkByParagraphs
Splits at blank-line paragraph boundaries, accumulating paragraphs until maxLength is reached. Natural for prose documents with clear paragraph structure.
ChunkBySections
One chunk per Markdown heading section. Sections longer than maxLength are further split by paragraphs. Keeps heading context with each chunk.
“How do I get chunks with their source offsets?”TextChunk.StartOffset / EndOffset
“Which strategy preserves sentence meaning?”ChunkBySentences
“I need one chunk per doc section.”ChunkBySections + ExtractSections
“How do I control overlap?”overlap parameter on all strategies
The awkward parts of AI data work, handled.
Extraction and chunking are step one. These utilities cover the friction points that follow: normalizing AI-emitted JSON, walking document structure, and profiling tabular data before it goes into a pipeline.
JSON schema validator
Validate AI-emitted JSON against a schema, list all errors, strip code fences (```json) before parsing, and compare two JSON payloads for changes.
Markdown structure
Build a heading map, extract labelled sections, normalize Markdown for consistent downstream parsing, and detect content changes between versions.
CSV data profiler
Infer column types, build value distributions, and detect data issues — duplicate headers, missing values, row-width mismatches — before passing tabular data to a model.
One call to register. One interface to inject.
Register all 14 extractors with a single DI call, then inject IDocumentTextExtractor wherever you process documents. Add or swap extractors for custom formats without touching the rest of the pipeline.
services.AddTextExtractor()— one-line setup- Customise with the callback overload to add, remove, or reorder extractors
TextExtractor.CreateDefaultExtractors()for manual compositionGetTextWithMetadataAsyncadds format-specific metadata keys to every result
What's coming next.
These features are queued for the next releases, ordered by effort-to-impact ratio. Contributions and priority feedback are welcome.
Install once.
Extract everything.
Practical questions.
For teams building RAG pipelines, document-processing services, and AI ingestion jobs.
Can I add support for a custom format?
Yes. Implement ITextExtractor with a CanHandle(extension) method and an ExtractText(byte[]) (and optionally ExtractText(Stream)) method, then register it via the DI callback or pass it directly to the TextExtractor constructor. You can also remove or replace built-in extractors using the same callback.
What does the metadata result add over plain text extraction?
Every result adds file.name, file.extension, file.bytes, text.length, text.lineCount, and text.wordCount. Format-specific additions include heading counts and section lists for Markdown, root kind and top-level property names for JSON, and subject, sender, date, and attachment names for EML. These keys are designed to enrich vector-store document records without a separate metadata pass.
Is there a streaming API — do I have to load the file into memory first?
No. GetTextAsync(fileName, Stream, ct) accepts a stream directly. The current implementation buffers internally to support extractors that require random access (PDF, DOCX), but stream overloads avoid forcing you to produce a byte array at the call site.
Which chunking strategy should I start with?
ChunkByParagraphs with maxLength: 1000, overlap: 100 is a solid starting point for most prose documents. Switch to ChunkBySentences if retrieval quality is sensitive to split boundaries, or ChunkBySections for Markdown-heavy knowledge bases where heading context matters.
Does it handle corrupt or unreadable files?
Yes. Every extractor wraps its work in a try/catch. A failed extraction returns ExtractionResult.Failure(errorMessage) rather than throwing, so a single bad file does not break a batch pipeline. Binary files with no extractable text return IsSuccess: true, Text: "" rather than an error.
Is this related to Transformations.Mapping?
Same family, different job. Transformations.Mapping is a Roslyn source generator for compile-time object mapping. Transformations.Text is a runtime document-processing library. They share the same package naming and .NET version targets but have no dependency on each other — install only what your project needs.