Dataset: MS MARCO 8.8M passages | Date: 2026-03-11 | Commit: 39c60d4 | System X: v0.21.6

Current Status

Index Size & Build Time

Query Latency (p50)

Median latency in milliseconds. Lower is better.

Query Latency (p95)

95th percentile latency in milliseconds. Lower is better.

Throughput

Total time to execute 800 test queries sequentially.

Analysis

Query latency: pg_textsearch faster across all token counts

pg_textsearch is faster on all 8 token buckets at both p50 and p95, ranging from 26x faster on single-token queries to 2.3x faster on 8+ token queries at p50. SIMD-accelerated bitpack decoding (PR #250), stack-allocated decode buffers (PR #253), and BMW cache optimizations (PR #274) drove improvements across the board.

Overall throughput: pg_textsearch 3.9x faster

pg_textsearch completes 800 queries in 6.5s vs 25.2s for System X, a 3.9x throughput advantage. This is up from 3.2x on March 3, driven by continued scoring path optimizations.

Index build: System X 1.6x faster

System X builds its index in 143s vs 234s for pg_textsearch (1.6x faster). The arena allocator rewrite (PR #231) and leader-only merge (PR #244) previously cut build time from 270s to 234s.

Methodology

Both extensions benchmarked on identical GitHub Actions runners with the same Postgres configuration. See full methodology for details.

MS-MARCO v2 — 138M Passages

Large-Scale Benchmark

Dataset: MS MARCO v2 — 138,364,158 passages | Date: 2026-03-10 | pg_textsearch: v1.0.0-dev (main @ fb3b3b1) | System X: v0.21.6

Environment

Current Status (138M)

Index Build (138M)

Single-Client Query Latency (138M)

Top-10 results (LIMIT 10), BMW optimization enabled. 691 queries sampled across 8 token-count buckets.

Median Latency (p50)

95th Percentile Latency (p95)

Weighted-Average Latency

Weighted by observed query-length distribution from 1,010,916 MS-MARCO v1 Bing queries after English stopword removal and stemming (mean 3.7 lexemes, mode 3).

MS-MARCO Query Lexeme Count Distribution (1,010,916 queries)
  Lexemes = distinct stems after English stopword removal

  lexemes   queries      %  distribution
  ───────  ────────  ─────  ──────────────────────────────────────────────────
        0        11   0.0%  ▏
        1    35,638   3.5%  ▓▓▓▓▓
        2   165,033  16.3%  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
        3   304,887  30.2%  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
        4   264,177  26.1%  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
        5   143,765  14.2%  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
        6    59,558   5.9%  ▓▓▓▓▓▓▓▓▓
        7    22,595   2.2%  ▓▓▓
        8     8,627   0.9%  ▓
        9     3,395   0.3%  ▏
       10     1,555   0.2%  ▏
       11       721   0.1%  ▏
       12       402   0.0%  ▏
       13       235   0.0%  ▏
       14       123   0.0%  ▏
       15+      193   0.0%  ▏

    Total: 1,010,916 queries
     Mean: 3.7 lexemes
     Mode: 3 lexemes (30.2%)

  72.6% of queries have 2-4 lexemes.
  96.2% of queries have 1-6 lexemes.

Benchmark buckets 1–7 contain 100 queries each; bucket 8+ contains 38 queries covering all lengths ≥8. Weights applied to each bucket match the distribution above.

Throughput (138M)

Single-Client Sequential

691 queries run 3 times; median iteration reported.

Concurrent (pgbench, 16 clients, 60 s)

Analysis (138M)

Query latency: pg_textsearch faster across all token counts

pg_textsearch is faster on all 8 token buckets at p50, ranging from 11.7x faster on single-token queries to 1.1x on 8+ token queries. Cached skip entries and reusable decompression buffers (PR #274) reduced per-block overhead in the WAND inner loop by 20–25%, closing the gap on high-token queries. The weighted p50 advantage is 2.3x.

Tail latency: improved but mixed at p95

pg_textsearch has tighter tail latency on 1–5 token queries at p95. On 6–8+ token queries, System X still has tighter tails. The p95 gap narrowed significantly with the cache optimizations (e.g., 5-token p95 went from 200ms to 167ms, now faster than System X's 190ms). Further tail latency optimization on long queries remains an active area of work.

Concurrent throughput: pg_textsearch 4.7x higher TPS

Under 16-client concurrent load, pg_textsearch achieves 91.4 TPS vs 19.4 TPS for System X — a 4.7x advantage. This is significantly wider than the 1.5x single-client gap, indicating that pg_textsearch scales much better under concurrency. pg_textsearch uses native Postgres buffer management and shared memory, avoiding the external process coordination overhead present in System X's architecture.

Index build: System X 1.9x faster

System X builds its index in 8 min 55 s vs 17 min 37 s for pg_textsearch (1.9x faster). pg_textsearch's parallel build uses 15 workers for the scan phase, but the subsequent merge phase is single-threaded and I/O-bound, accounting for the majority of the build time. Despite the slower build, pg_textsearch produces a 26% smaller index (17 GB vs 23 GB).

Methodology (138M)

Both extensions benchmarked on the same dedicated EC2 instance (c6i.4xlarge), same Postgres 17.7 installation, same dataset. The table was loaded once; each extension built its index from scratch with page cache dropped before each build. Query benchmarks include warmup passes. The pgbench power test uses -M prepared mode with random query selection from 691 benchmark queries.