Getting started¶

Install¶

Contributors (editable checkout):

git clone https://github.com/harp-lab/srdatalog-python.git
cd srdatalog-python
uv sync --group dev
uv run python scripts/populate_vendor.py   # fetch boost / highway / RMM / spdlog (~3 min, one-time)
uv pip install -e .

End users (PyPI wheel, once published — vendor is already bundled):

uv pip install srdatalog

Toolchain requirements at JIT compile time¶

The library emits C++23 + CUDA code, so the backend compiler constraints are tighter than a vanilla C++ project:

Requirement	Why
clang++ ≥ 20 (mandatory)	`gcc` can’t compile CUDA kernels (no `-x cuda`); `nvcc` doesn’t support C++23; clang-18/19 reject some of our `constexpr` / literal-type usage. clang-20 is the first version that compiles the runtime cleanly.
CUDA 12.x toolkit	CUDA 13 dropped `thrust/optional.h`, which our pinned RMM still needs. The auto-detector in `srdatalog.runtime.find_cuda_root()` prefers a 12.x install when both are present.
`libboost_container` on the link path	The runtime’s PMR allocators reference `boost::container::pmr::get_default_resource` — a definition, not header-only, and not shipped in the vendored boost subset. `apt install libboost-container-dev` or `brew install boost`.
`ccache` (optional)	Auto-detected; near-instant warm rebuilds. Opt out with `SRDATALOG_JIT_NO_CCACHE=1`.
`ninja` (optional)	Auto-installed via the `ninja` PyPI wheel, so no system install needed.

The library auto-detects clang++ via $CXX or $PATH, and CUDA via $CUDA_HOME or the usual install paths.

The pipeline¶

Every program flows through five phases:

Python DSL (dsl.py)
  → HIR                     (srdatalog.hir)
  → MIR                     (srdatalog.mir)
  → C++/CUDA .cpp tree      (srdatalog.codegen.jit.*)
  → shared library via ninja + clang++   (srdatalog.codegen.jit.compiler_ninja)
  → dlopen + extern "C" shim             (user-side ctypes)

srdatalog.build.build_project() covers phases 1 – 3 in one call, and srdatalog.compile_jit_project() covers phase 4. Phase 5 is a stock ctypes.CDLL.

See Architecture for a deeper tour of what each phase does.

End-to-end example¶

The canonical “run any benchmark” script:

# Synthetic triangle count (no input CSVs needed)
python examples/run_benchmark.py triangle

# Transitive closure on a CSV directory
python examples/run_benchmark.py tc --data /path/to/edges

# Doop points-to on batik (defaults baked into doop_run.py)
python examples/doop_run.py

# Cap iterations for a quick sanity check
python examples/run_benchmark.py galen --data /path/to/data --max-iter 3

run_benchmark.py prints one line per phase (DSL build, emit, ninja compile, load, run, size readback) with timings, so you can see where time is going on your box.

Writing your own benchmark¶

from srdatalog import Var, Relation, Program

x, y, z = Var("x"), Var("y"), Var("z")
Edge = Relation("Edge", 2, input_file="Edge.csv")
TC   = Relation("TC", 2, print_size=True)

prog = Program(
  rules=[
    (TC(x, y) <= Edge(x, y)).named("TC_Base"),
    (TC(x, z) <= TC(x, y) & Edge(y, z)).named("TC_Rec"),
  ],
)

Relation pragmas you’ll actually use:

input_file="X.csv" — included in the auto-generated load_all shim.
print_size=True — runner emits a size readback after the fixpoint.
index_type="SRDatalog::GPU::Device2LevelIndex" — override the default LSM index (needed for very fat relations like VarPointsTo).

Rule builders: atom <= body1 & body2 & ~neg & Filter((v,), "..."). See the srdatalog.dsl API reference for every operator.

Opt-in exact bitmap join plans¶

with_plan(dedup_bitmap=True) selects an exact bitmap implementation for a binary set-valued join projection. It changes the physical plan, not the rule’s logical tuple set, and is not enabled automatically:

from srdatalog import Program, Relation, Var

join, value, destination = Var("join"), Var("value"), Var("destination")
Assign = Relation("Assign", 2, input_file="Assign.csv")
Points = Relation("Points", 2, input_file="Points.csv")
Output = Relation("Output", 2)
prog = Program([
    (Output(value, destination) <= Assign(join, destination) & Points(join, value))
    .with_plan(dedup_bitmap=True)
])

For recursive rules, select the intended semi-naive variants explicitly with with_plan(delta=..., dedup_bitmap=True). The compiler maintains the required dictionary indexes across producer strata and recursive iterations.

This specialization supports unconstrained binary NoProvenance projections: the shared join variable is eliminated and one distinct variable from each source is retained. Filters, negation, constants, incompatible execution strategies, count-only rules, unsupported index types, and plans matching no recursive variant are rejected rather than silently approximated. Runtime shared-memory capacity limits also fail explicitly.

Bitmap plans require a complete runner (build_project or compile_runner), not standalone compile_pipeline kernel emission. Keep the ordinary plan as a correctness/performance control; enabling a bitmap plan is not by itself evidence of an end-to-end speedup.

Translating from Nim¶

The upstream Nim reference has a few dozen benchmark programs under integration_tests/examples/. tools/nim_to_dsl.py auto-translates them:

python tools/nim_to_dsl.py path/to/upstream.nim --out examples/upstream.py

It handles schema blocks, rules blocks, dataset_consts, all the negation and filter shorthands, plan pragmas, and the split marker. See the Benchmarks page for the full list.