Haystack

PythonNLPNLTK PunktRAGDocument Preprocessing
Haystack preview

overview.

Haystack provides modular document splitters, retrievers, rankers, and agent pipelines for LLM applications. In PR #13646 (fixing issue #13645), I fixed sentence-mode splitting for Hindi, Marathi, Bengali, and other Indic scripts in DocumentSplitter and RecursiveDocumentSplitter. NLTK's default Punkt language variables only treat '.', '?', and '!' as sentence terminators, so Indic documents using Purnaviram danda (U+0964) and double danda (U+0965) were returned as a single unsplit chunk. Adding DandaPunktLanguageVars fixed Indic sentence boundaries while keeping English tokenization unchanged.

key features.

  • PR #13646: added DandaPunktLanguageVars with U+0964 and U+0965 sentence terminators for DocumentSplitter and RecursiveDocumentSplitter
  • Preserved existing English Punkt tokenization across both keep_white_spaces modes (533 preprocessor tests passing)