Vector search benchmark reveals a vast build-time gap
A vector search benchmark across six database engines found that turning the same 990,000 embeddings into a queryable index took anywhere from 1.9 minutes to 6.7 hours. Percona ran the tests on one machine with the same query set and container shape, measuring build time, storage, memory, accuracy, throughput, concurrent clients, filtered searches and the effect of rewriting 10% of the data.
The corpus was dbpedia-openai-1000k-angular, containing 990,000 OpenAI embeddings, each represented by 1,536 float32 numbers. An embedding is a numerical representation used to compare meaning or similarity. Every engine used HNSW, or Hierarchical Navigable Small World, a graph structure that accelerates searches for nearby vectors, with M set to 16 and the result count k set to 10. The host had an Intel Xeon Gold 6230 processor, 80 logical CPUs, AVX-512 and 187.5 GiB of RAM. The database container received CPUs 0 through 63 and a 120.8 GB memory limit; a separate test container received 16 CPUs and 8 GB. Separating them prevented the query set held by NumPy from being counted as database memory.
The tested systems were the open-source relational database 11.8.8 with MHNSW, the same open-source relational database 12.3.2 with MHNSW, AliSQL 8.0.44-2 with VIDX, pgvector 0.8.6 on the relational database used by many web applications, Valkey 9.1.1 with valkey-search, and Percona Search for MongoDB 1.70.3-1 on PSMDB 8.3.8-2. Percona Search was a Technical Preview whose documentation said it was not for production. Its tests used a scalar-quantized index, which stores less precise numerical values to reduce processing and storage requirements.
Build performance varied dramatically. Percona Search became queryable in 1.9 minutes, Valkey in 4.3 minutes and the relational database with pgvector in 40.9 minutes. The relational database spent 26.9 minutes loading through COPY and 14 minutes building the index with eight workers. The open-source database 11.8 needed 3.6 hours, the open-source database 12.3 needed 5.4 hours and AliSQL needed 6.7 hours. MHNSW and VIDX update their graph during every INSERT, so loading and index construction were one operation. Their rates ranged from 41 to 76 rows per second, compared with 613 for pgvector and 30,170 for Valkey.
The slowest builders produced the smallest indexes. The open-source database and AliSQL each used 3.9 GiB because their graphs kept 16-bit quantized vector copies, while pgvector and Valkey indexed full float32 values. The raw vectors alone occupied 5.7 GiB. Index sizes were 7.6 GiB for pgvector, 11.3 GiB for Valkey and 15.1 GiB for Percona Search. Including tables, AliSQL used 14.4 GiB, pgvector 15.2 GiB, the open-source database 11.8 and Valkey 18.2 GiB each, and Percona Search 20.9 GiB. Percona Search's index later measured 7.2 GiB at the start of the churn test, and Percona could not determine which figure represented steady state.
Peak resident memory ranged from 14.3 GiB for Valkey to 48.8 GiB for Percona Search. Cache allocations were engine-specific rather than identical: the InnoDB systems received a 42.3 GB buffer pool and 30.2 GB graph cache, pgvector received 72.5 GB of shared_buffers plus 12.1 GB of maintenance_work_mem, and Valkey had 108.7 GB of maxmemory. Percona Search used MongoDB's default WiredTiger cache of 57 to 64 GB. Index builds were limited to eight threads because higher counts caused contention and could stall writes into populated indexes.
Accuracy was measured as recall, meaning how many correct nearest neighbours the search returned. With one client and recall of at least 0.99, the open-source database 12.3 delivered 277 queries per second (QPS), Valkey 264, pgvector 210, AliSQL 126 and the open-source database 11.8 only 46. Percona Search's quantized index peaked at 0.892 recall, so it did not qualify; without quantization it recorded 88 QPS at the 0.99 floor. The 13 QPS difference between the open-source database 12.3 and Valkey was too small for one run to distinguish from noise. The open-source database 12.3 was also compiled with -march=native, unlike the x86-64-v3 builds used for the open-source database 11.8, AliSQL and pgvector.
AliSQL produced an unexplained anomaly. At M of 16, throughput increased from 2.6 to 4.4 QPS when ef_search, the amount of query-time search effort, rose from 400 to 800, although more work should not make the same index faster. Median latency also jumped from 6.6 milliseconds at ef_search 80 to 50.8 milliseconds at 120, then 191 to 384 milliseconds above that. Percona marked this range unreliable rather than treating it as a valid result.
The concurrency results require equally careful reading. At 32 clients and ef_search 100, Valkey reached 5,683 QPS with 10.2 millisecond p99 latency, meaning 99% of queries completed within that time. Pgvector reached 4,798 QPS at 10.8 milliseconds, while Percona Search reached 1,844 QPS at 43.5 milliseconds. Pgvector produced 21.7 times its single-client throughput. However, the harness had only 16 CPUs, so Valkey and Percona Search flattening between 16 and 32 clients may reflect client-side CPU limits.
No multi-client figures were reported for the open-source database 11.8, the open-source database 12.3 or AliSQL. Their threaded Python clients, all using the connector for the open-source database, remained near one query in flight as client counts rose. Percona had not identified the exact mechanism and did not profile the servers, so it made no claim about their server-side scaling. The harness has since been changed to put each client in a separate operating-system process, but that revision had only been tested with synthetic drivers and had not yet re-measured these engines.
Filtering exposed major behavioural differences. At 1% selectivity, meaning only 1% of rows passed the condition, Valkey returned 0.9996 recall at 8.2 QPS. The open-source database 12.3, the open-source database 11.8 and AliSQL kept searching after filtering, preserving recall between 0.9724 and 0.9884 but falling below one QPS, with p99 latency of 10 to 13 seconds. Percona Search filtered before vector search and accelerated from 71.9 QPS at 10% selectivity to 117.5 at 1%, although quantization limited recall to 0.8666.
Pgvector's default behaviour searched first, removed rows that failed the condition and then stopped. At 1% selectivity, all 500 queries returned fewer than the requested ten rows, producing recall of 0.1034 at 237.1 QPS without an error or warning. Pgvector 0.8 includes iterative scan to address this case, but the benchmark did not test it. The practical lesson is that fast completion is not useful if an application silently receives incomplete results.
After 99,000 rows were deleted and reinserted, pgvector held steady at 245 QPS, up 0.4%, while Valkey fell 4% to 460 QPS. Percona Search dropped 38.9% to 113 QPS, although recall barely changed from 0.8432 to 0.8416.