Pool GPU, CPU, VRAM, and RAM across networked devices into a unified compute layer

nmonit is a distributed compute fabric that aggregates hardware resources across multiple machines on a LAN and presents them as a single unified pool. Applications request what they need — memory, compute, GPU — and the fabric decides where to place it. The application doesn’t need to know the cluster topology.

Example: Run an LLM that needs 48GB VRAM on a cluster where no single GPU has more than 24GB. nmonit shards the model across two GPUs on different nodes transparently.

Architecture

A control plane (Go) handles scheduling, resource management, and the API. Node agents (Rust) run on each machine, reporting resources and executing tasks. Communication is via gRPC over RDMA (preferred) or TCP.

Key Features

• VRAM pooling — Aggregate GPU memory across all nodes. Run models larger than any single GPU’s VRAM.
• Distributed shared memory — Page-level remote memory access across nodes with configurable consistency (strict, relaxed, release).
• Smart scheduling — Topology-aware placement, data gravity, gang scheduling for distributed jobs, preemption with checkpoint.
• RDMA support — Zero-copy inter-node transfers via RoCE v2 or InfiniBand. Falls back to TCP automatically.
• Multi-tenancy — Resource quotas, network isolation, RBAC, mTLS everywhere.
• Fault tolerance — Node failure detection, automatic work redistribution, checkpoint/restart.
• ML-first design — Model sharding (data/tensor/pipeline parallelism), continuous batching, KV-cache offloading.
• Container-native — Any Docker/OCI container can tap into the fabric via runtime hooks.
• SDK — Rust, Python (numpy-compatible distributed arrays), Go, C bindings. MPI compatibility layer.
• CLI & REST API — Submit jobs, check cluster status, manage resources from the command line or programmatically.

Technology Stack

Component	Language	Role
Control Plane	Go	Scheduler, REST/gRPC API, resource manager, Raft consensus
Node Agent	Rust	Resource reporting, task execution, memory management, GPU access
Network	UCX (Rust)	RDMA (RoCE v2) with TCP fallback
Wire Protocol	Protobuf + gRPC	Code gen for Go, Rust, Python
GPU	CUDA (NVIDIA first)	VRAM pooling, model sharding
Metrics	Prometheus	Industry standard monitoring

Quick Start

# Clone and build
git clone https://github.com/ParadoxFuzzle/nmonit.git
cd nmonit
make

# Start the control plane (first node)
./bin/control-plane --bootstrap

# Start node agents on each machine
./bin/compute-agent --control-plane 192.168.1.10:9000

# Submit a job
./bin/compute submit --gpus 2 --vram 24G my-job.py

# Check cluster status
./bin/compute status

Requirements

• All nodes: Linux kernel 5.15+, systemd
• GPU nodes: NVIDIA drivers 535+, NVML
• RDMA (optional): RDMA-capable NICs (Mellanox ConnectX-4+, RoCE v2)
• Build tools: Rust 1.78+, Go 1.22+, Protocol Buffers (buf)

Project Status

Pre-alpha. Phase 1 implementation is underway. The architecture, protocol definitions, and core scaffolding are in place. See the architecture doc and PRD on GitHub for the full roadmap.

Download

Source code, build instructions, and documentation. Pre-built binaries coming soon.

License

MIT OR Apache-2.0