TL;DR
This paper introduces arcadedb-embedded, a Python library that allows developers to run a full-featured, multi-model database (ArcadeDB) directly within a Python process. By bundling a Java Runtime Environment (JRE) into a standard Python wheel, it provides a "one-stop-shop" for documents, property graphs, and vector search, eliminating the need to manage a complex stack of specialized, disconnected databases.
What Problem Does This Paper Address?
Modern AI and data science workflows often require three distinct types of data handling: transactional document storage (SQL), complex relationship mapping (Graphs), and semantic similarity search (Vectors). Currently, Python developers typically solve this by stitching together a "stack of specialists"—for example, using SQLite for metadata, NetworkX for graphs, and Chroma or FAISS for vectors.
This approach creates a "specialist stack tax." Data must be duplicated across systems, application code must manually keep these copies consistent, and queries spanning different models require passing identifiers back and forth between libraries. The authors aim to replace this fragmented architecture with a single, ACID-compliant engine that lives entirely inside the host Python process.
Methods
The core of the solution is ArcadeDB, an open-source Java database engine. To make it accessible to Python without requiring a separate server or even a manual Java installation, the authors developed arcadedb-embedded.
Key technical features include:
- In-Process JVM: It uses JPype to start a Java Virtual Machine (JVM) inside the Python process. This allows Python and Java to share the same memory address space, enabling efficient data transfer without network calls.
- Bundled JRE: The library uses
jlinkto create a trimmed-down JRE that is bundled directly into the Python wheel. A simplepip installprovides everything needed to run the database. - Multi-Model API: A single Python object handles SQL (for documents), OpenCypher (for graphs), and vector search (via HNSW-style indexes).
- Efficient Interop: The library supports zero-copy-style data exchange, allowing query results to be returned directly as NumPy arrays, pandas DataFrames, or PyArrow tables.
Datasets
The authors evaluated the system using a real-world dataset: the Cross Validated (Stack Exchange) public data dump. This corpus consists of:
- 426k posts (treated as documents).
- 346k users and their relationships (who answered what), forming a property graph.
- 1.2M text embeddings (384-dimensional vectors) generated from post titles and bodies.
They benchmarked ArcadeDB against "specialist" embedded databases: SQLite and DuckDB for documents, LadybugDB for graphs, and Chroma for vectors.
Key Insights and Ideas
- The Hybrid Workflow Advantage: The most powerful feature is the ability to chain different models in a single pipeline. A single query can perform a vector search to find similar questions, use SQL to filter those by score, and then use Cypher to traverse the graph to find the reputations of the authors—all without the data ever leaving the database engine.
- Generalist vs. Specialist: While specialized engines are often faster at their specific task (e.g., DuckDB is significantly faster for analytical SQL), ArcadeDB provides a competitive "middle ground" that simplifies the overall architecture for applications needing all three models.
- Memory Efficiency in Vectors: Unlike Chroma, which keeps its vector index in RAM, ArcadeDB's vector index is disk-backed. This resulted in 14% lower peak memory usage during benchmarks while maintaining slightly higher recall.
Key Results
- Capability: ArcadeDB was the only engine tested that natively supported all three models (Documents, Graph, Vectors) in a single engine.
- Document Performance: SQLite remains the king of point operations (87k ops/s vs. ArcadeDB's 6.4k), and DuckDB dominates analytical queries (9.3ms vs. ArcadeDB's 1,400ms).
- Graph Performance: ArcadeDB outperformed LadybugDB on transactional graph throughput (3,212 ops/s vs. 525 ops/s) but was slower on large-scale analytical traversals.
- Vector Performance: ArcadeDB achieved a recall@10 of 0.980, slightly beating Chroma's 0.972. However, Chroma's query latency was roughly 3.5x faster (1.17ms vs. 4.15ms).
- Data Transport: For large results (100k rows), using
to_arrow()was 18x faster than iterating through dictionaries, highlighting the importance of columnar transport formats in Python-Java bridges.
Limitations and Future Work
- Performance Gaps: As a generalist, ArcadeDB is currently slower than specialists like DuckDB for OLAP or SQLite for OLTP. The authors note that ArcadeDB is under active development and performance is expected to improve.
- Ingestion Speed: ArcadeDB lacks native Parquet or Arrow readers, meaning data must be loaded through Python, which is significantly slower than the native loading capabilities of DuckDB or SQLite.
- JVM Overhead: Running a JVM inside Python adds a memory baseline (approx. 800 MiB for document workloads) that smaller, C-based engines like SQLite avoid.
- SQL Limitations: ArcadeDB’s SQL dialect currently lacks support for
JOINoperations and window functions (OVER/PARTITION BY).
Summary updated