Product Updates
Literature Retrieval in Parallax: Set Algebra, Full-Corpus Recall, and Jev Screening
General-purpose agents search the literature through web search APIs, which return only the top-k results and a lot of noise, so they can’t be exhaustive. Parallax runs full-corpus recall over about 1.58 million arXiv papers across all disciplines since 2020, and models retrieval as composable operations on sets of papers. Keyword search is sped up with an adjacent-word fragment index. Semantic search uses halfvec ANN pruned by year and discipline, with a provably lossless exact batching path as a fallback. Screening and classification use Jev’s calibrated probabilities, over 10x faster than our LLM-based pipeline at less than a fifth of the cost. This post covers the system’s design and the performance work needed to run it in production.