5
Days to Demo-Ready
4
API Endpoints Delivered
100%
GPU Isolation Enforced
Full
Audit Trail per Job
The Challenge
The client needed a GPU job management system that could be demonstrated end-to-end, serve as a prototype foundation for production scaling, and enforce hard GPU isolation between concurrent workloads. Existing solutions were either too heavy (Kubernetes-based orchestration with significant infrastructure overhead) or too lightweight (shell scripts with no scheduling intelligence or audit capability).
The requirements were specific: VRAM-aware scheduling, per-job GPU isolation using NVIDIA_VISIBLE_DEVICES, containerized execution, structured audit logs, and a REST API interface - all running on a single Rocky Linux 9.7 server with an RTX 3090.
Architecture Delivered
Component 1: FastAPI REST API Server
A production-grade FastAPI application providing three core endpoints:
- POST /jobs - Accepts job submissions with image, command, gpu_count, min_vram_mb, env, and volumes parameters. Returns job_id and initial status immediately.
- GET /jobs/{job_id} - Returns full job state: status, assigned GPUs, exit code, all timestamps.
- GET /jobs/{job_id}/logs - Returns captured stdout/stderr from the container run.
The API server runs as a systemd service (uvicorn), starts on boot, and restarts automatically on failure.
Component 2: VRAM-Aware Scheduler
A polling scheduler loop (separate systemd service) implementing:
- Queries nvidia-smi for real-time VRAM totals and current usage per GPU
- Calculates available VRAM as total − used for each GPU
- Selects GPUs where available VRAM ≥ job's min_vram_mb requirement
- Reserves selected GPUs before container launch - preventing double-allocation
- If no GPUs meet constraints, sleeps 2–5 seconds and retries - jobs queue gracefully
Component 3: Docker Container Runner
- GPU isolation via --gpus "device=N" flag
- Containers launched with --rm for automatic cleanup
- stdout/stderr captured to logs/{job_id}.log
- Exit code recorded to SQLite on completion
- Configurable job timeout (default 30–60 minutes)
Component 4: SQLite Persistence & Audit Layer
| Field | Description |
|---|---|
| job_id | UUID - globally unique job identifier |
| status | queued / running / succeeded / failed |
| image, command | Exact Docker image and command submitted |
| gpu_count, min_vram_mb | Resource requirements as submitted |
| assigned_gpu_indices | Actual GPU indices assigned at runtime |
| exit_code | Container process exit code |
| created_at, started_at, finished_at | Full timestamp chain for auditability |
| log_path | Path to job's stdout/stderr log file on disk |
Delivery Timeline
- Day 1: Server access validated · NVIDIA drivers + CUDA verified · Docker + NVIDIA Container Toolkit installed · GPU container smoke test passing
- Day 2: FastAPI service running · SQLite schema created · POST /jobs storing queued jobs · GET endpoints operational
- Day 3: Docker runner implemented · Log capture working · Exit codes recorded · Container cleanup on completion
- Day 4: Scheduler loop running · VRAM-aware GPU selection · GPU reservation preventing double-allocation · Full job lifecycle tested
- Day 5: systemd services configured · README with demo steps · End-to-end demo working reliably · Audit trail verified
Technologies Used
Outcome & Extensibility – System Value
The delivered framework provides a fully functional, demo-ready GPU job management system that serves as the foundation for production GPU orchestration. Its architecture cleanly separates API, scheduling, execution, and storage concerns - making it straightforward to extend to multi-GPU servers, multiple nodes, or cloud deployments. All GPU isolation guarantees are enforced at the container runtime level, providing hardware-level security without additional tooling.