Machine Learning Engineering Open Book

Run, manage, and scale AI workloads on any AI infrastructure. Use one system to access & manage all AI compute (Kubernetes, Slurm, 20+ clouds, on-prem).

Slurm: A Highly Scalable Workload Manager

A DSL for data-driven computational pipelines

Vendor-agnostic orchestration for training, inference and agentic workloads across NVIDIA, AMD, TPU, and Tenstorrent on clouds, Kubernetes, and bare metal.

Python 3.8+ toolbox for submitting jobs to Slurm

A WDL, CWL and Python API supporting easy-to-use workflow engine. It is scalable, efficient and cross-platform (Linux/macOS).

Best practices & guides on how to write distributed pytorch training code

Open source web interface for Slurm HPC & AI clusters

Python Interface to Slurm

A Slurm cluster using docker-compose

TUI for the Slurm Workload Manager

Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪

Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

A scheduler for GPU/CPU tasks

Run Slurm in Kubernetes

Create clusters of VMs on the cloud and configure them with Ansible.

Run Slurm on Kubernetes. A Slinky project.

An open-source toolkit for deploying and managing high performance clusters for HPC, AI, and data analytics workloads.