Terradev

Provision GPU instances for optimal parallel efficiency

Generate full preflight validation report with pass/warn/fail per check. Covers GPU drivers, CUDA, NCCL, RDMA, network, disk, and Docker.

GPU-specific preflight validation: NVIDIA drivers, CUDA version, GPU count, NCCL, NVLink topology, NCU stall-signature profiling, and adversarial config verification (V1-V3).

Network-specific preflight validation: RDMA availability, InfiniBand status, inter-node bandwidth, latency matrix, firewall rules.

Scan local machine and network for available GPU devices. Returns total VRAM pool for local-first provisioning.

View all instances and costs with state optimization

Manage GPU instances (stop/start/terminate)

Get cost analytics

Find cheaper alternatives for running instances

Get setup instructions for a provider

Configure provider credentials

Launch distributed training on provisioned GPU nodes. Supports torchrun, deepspeed, accelerate, and megatron. Use from_provision='latest' to auto-resolve node IPs from your last provision command.

Pre-training validation: GPU availability, NCCL, RDMA, drivers across all nodes.

Compress, chunk, checksum, and position datasets near compute. Supports local paths, S3/GCS URIs, HTTP URLs, and HuggingFace dataset names. Returns staging plan with agent recommendations.

CLI-native provisioning with manifest cache and drift detection. Use --fix-drift to detect and auto-fix drifted infrastructure.

Explicit versioned rollback. Format: job@version (e.g., llama3@v3).

List cached manifests and versions for jobs.

Run a declarative YAML workflow that chains multiple Terradev commands (provision → preflight → train → monitor → checkpoint). Returns step-by-step execution status with cost estimates and confirmation gates for expensive operations.

Get current Terradev state: running training jobs, active instances, spend-to-date, alerts. Call this on session start to resume context from previous sessions.

Request user consent for data movement across cloud regions. GDPR/SOC2 compliant consent tracking with audit trail.

Record a consent response (granted or denied) for a pending consent request.

Evaluate OPA (Open Policy Agent) policies for data access. Checks region restrictions, classification rules, and compliance requirements.

Move data with full governance audit trail. Requires prior consent and OPA policy approval. Tracks integrity, encryption, and compliance.

Get data movement audit log. Filter by user, dataset, or time range.

Generate comprehensive compliance report: consent stats, policy evaluations, data movements, violations. For GDPR/SOC2/HIPAA audits.

Deploy model to InferX serverless platform

Check InferX endpoint status

List deployed InferX models

Get cost analysis for inference endpoints

Configure InferX serverless platform credentials.

Delete an InferX model deployment.

Get InferX account usage statistics: requests, cost, GPU hours, latency.

Get InferX pricing quotes for a GPU type.

Semantic-aware inference routing. Analyzes query content across 6 signal dimensions (modality, complexity, domain, language, safety, keywords), applies NUMA-aware endpoint scoring, and selects the optimal inference endpoint. Uses DAG parallel execution for signal extraction.

Disaggregated Prefill/Decode routing (DistServe architecture). Splits LLM inference into compute-bound prefill phase (routed to FLOPS-optimized GPUs like H100 SXM) and memory-bound decode phase (routed to bandwidth-optimized GPUs like MI300X). Tracks KV cache handoffs between endpoint pairs.

Run health checks and auto-failover for inference endpoints. If a primary endpoint is unhealthy and has a backup configured, traffic automatically shifts to the backup provider.

GPU NUMA topology report with intra-GPU XCD (Accelerated Compute Die) awareness. Models MI300X (8 XCDs, 192GB HBM3), MI300A (6 XCDs, 128GB), H200 (unified 141GB HBM3e), H100 (80GB). Reports PCIe locality (PIX/PXB/PHB/SYS), GPU-NIC pairing, SR-IOV VF status, and generates XCD-aware NCCL/AITER environment variables.

Start a vLLM inference server on a remote instance via SSH/systemd. Supports Multi-LoRA, Sleep Mode, KV Offloading, Speculative Decoding.

Stop a vLLM server on a remote instance.

Test inference against a running vLLM endpoint (completions or chat).

Get vLLM server info: loaded models, config, and health status.

Put a vLLM server to sleep. Level 1: offload to CPU (fast wake). Level 2: discard weights (minimal RAM).

Wake a sleeping vLLM server. For Level 2 sleep, also reloads weights and resets prefix cache.

Complete SGLang optimization stack with workload-specific auto-tuning for 7 workload types: agentic chat, batch inference, low latency, MoE models, PD disaggregated, structured output, and RAG.

[DEPRECATED] Use sglang action='start' instead. Start an SGLang inference server on a remote instance.

[DEPRECATED] Use sglang action='stop' instead. Stop an SGLang server on a remote instance.

[DEPRECATED] Use sglang action='inference' instead. Test inference against a running SGLang endpoint.

List models available on an Ollama server.

Pull a model to an Ollama server on a remote instance.

Generate text using an Ollama model (non-chat completions).

Chat with an Ollama model using the chat/completions API.

Get detailed information about an Ollama model (parameters, template, license).

Automatically optimize vLLM configuration by analyzing workload patterns. Selects optimal settings for the 6 critical knobs based on live endpoint metrics or sample request files.

Analyze a running vLLM server's workload and return specific optimization recommendations with before/after comparisons.

Benchmark a vLLM endpoint with concurrent requests. Returns throughput (req/s), success rate, and total latency.

List currently running Ollama models.

Configure Langfuse credentials (public key, secret key, host URL).

Test Langfuse connectivity and list accessible projects.

List recent LLM traces from Langfuse.

Get a single Langfuse trace with all observations/spans.

List evaluation scores from Langfuse, optionally filtered by trace or score name.

Create an evaluation score for a Langfuse trace (e.g. quality, accuracy, relevance).

List Langfuse datasets for evaluation and fine-tuning.

Export Langfuse traces as instruction/response pairs for LoRA fine-tuning. Filters by quality score.

Get aggregated quality metrics from Langfuse scores for drift detection.

Print OTEL environment variables for instrumenting LLM apps to send traces to Langfuse.

Generate Kubernetes deployment manifest for self-hosted Langfuse.

Create Kubernetes cluster with GPU nodes for optimal multi-cloud deployment

List Kubernetes clusters

Get information about a specific cluster

Destroy a Kubernetes cluster

Generate Helm charts from workload specifications.

Generate a GPU-aware KServe InferenceService YAML manifest with NUMA pinning, resource limits derived from model size and VRAM, and topology hints.

List KServe InferenceServices in a Kubernetes namespace.

Get detailed status of a KServe InferenceService including readiness, traffic split, and URL.

Install NVIDIA GPU Operator on a Kubernetes cluster. Configures driver containers, device plugin, DCGM exporter, and GPU Feature Discovery.

Configure Kubernetes GPU device plugin settings: time-slicing, MIG strategy, and resource naming.

Configure Multi-Instance GPU (MIG) partitioning on A100/H100 GPUs. Splits a single GPU into isolated instances for multi-tenant workloads.

Configure GPU time-slicing for Kubernetes. Allows multiple pods to share a single GPU with configurable oversubscription.

Deploy model to HuggingFace Spaces

Get HuggingFace Space deployment status.

Get Ray cluster status including node count, resources, memory, and running jobs.

Start a Ray cluster (head node or worker). For distributed ML training and inference.

Stop the Ray cluster on the current node.

Submit a job script to the Ray cluster for distributed execution.

List all running Ray jobs and tasks.

Generate a Ray Serve LLM Wide-EP (Expert Parallel) deployment for MoE models. Returns Python script and config for distributed MoE serving with EPLB and DeepEP.

Generate a Ray Serve LLM disaggregated Prefill/Decode deployment. Splits inference into compute-bound prefill and memory-bound decode phases with KV cache transfer via NIXL.

Compute optimal TP/DP/EP parallelism strategy for a given MoE model and GPU count. Returns recommended configuration with rationale.

List all Weights & Biases projects for the configured entity.

List runs in a W&B project with status, metrics summary, and config.

Get detailed info, metrics, and artifacts for a specific W&B run.

List MLflow experiments on the configured tracking server.

Log a Terradev training run to MLflow with auto-injected GPU type, provider, cost/hr, and duration as params.

Register a trained model in the MLflow model registry with Terradev provenance tags.

Get DVC repository status: tracked files, remotes, and changes since last commit.

Show DVC diff between two revisions (e.g. training checkpoints). Shows added, modified, deleted files.

Atomic checkpoint staging: DVC add + push + git commit in one operation. Promotes a training checkpoint to versioned storage.

Push DVC-tracked data to the configured remote storage.

Search and browse HuggingFace Hub models. Filter by author, task, library. Returns model ID, downloads, likes, and tags.

Search and browse HuggingFace Hub datasets. Filter by author and search query.

Get detailed model info: architecture, size, downloads, license, tags, pipeline_tag, and model card.

Create a HuggingFace Inference Endpoint (paid GPU endpoint). Supports custom GPU types, regions, and scaling.

List all active HuggingFace Inference Endpoints with status, URL, and cost.

Get detailed info about a specific HuggingFace Inference Endpoint: status, URL, scaling config, cost.

Delete a HuggingFace Inference Endpoint.

Run inference on a HuggingFace Inference Endpoint. Supports text generation, embeddings, and custom inputs.

Auto-generate an optimized deployment template for any HuggingFace model. Analyzes model size, architecture, and quantization to select optimal hardware and generate ready-to-deploy configs.

Get hardware recommendation with cost breakdown for any HuggingFace model. Returns optimal GPU type, estimated cost, and performance score.

Compare all hardware options for a HuggingFace model. Returns side-by-side cost, performance, and compatibility analysis.

Create a LangChain workflow.

Create an SGLang model-serving pipeline via LangChain. Connects LangChain agents to SGLang inference endpoints.

Create a LangGraph stateful workflow with monitoring. Supports agent graphs, tool calling, and state persistence.

Create an orchestrator-worker pattern workflow in LangGraph. The orchestrator delegates tasks to specialized worker agents.

Create an evaluator-optimizer workflow in LangGraph. Generates outputs, evaluates quality, and iteratively improves.

Get the status and metrics of a LangGraph workflow execution.

Create a custom W&B dashboard with GPU metrics, training loss, and cost panels.

Auto-create a Terradev-specific W&B dashboard with GPU utilization, cost tracking, training metrics, and infrastructure panels.

Create a W&B report with custom sections, charts, and narrative text.

Auto-generate a Terradev infrastructure report: GPU costs, provider comparison, training efficiency, and recommendations.

Set up custom W&B alerts for GPU metrics: cost thresholds, utilization drops, training anomalies.

Auto-create standard Terradev alerts: GPU cost > budget, utilization < 50%, training loss spike, straggler detection.

Get comprehensive W&B monitoring overview: dashboards, reports, alerts, active runs.

Test connection to Arize Phoenix server. Returns collector endpoint and project count.

List Phoenix projects (trace namespaces).

List recent spans for a Phoenix project. Supports SpanQuery DSL filters like "span_kind == 'RETRIEVER'" or "status_code == 'ERROR'".

View full execution tree for a specific trace ID. Shows span hierarchy, latencies, and token counts.

Generate OpenTelemetry environment variables for instrumenting serving pods with Phoenix tracing.

Generate Python instrumentation snippet for adding Phoenix tracing to LLM applications.

Generate Kubernetes deployment manifest for self-hosted Arize Phoenix server.

Test connection to NeMo Guardrails server.

Send a message through NeMo Guardrails and return the safety-filtered response. Applies topical, jailbreak, PII, and factcheck rails.

Generate default Colang 2.x guardrails configuration files (topical, jailbreak, PII, factcheck rails).

Generate Kubernetes deployment manifest for NeMo Guardrails server (standalone or sidecar mode).

Test connection to Qdrant vector database. Returns cluster info and collection count.

List all Qdrant vector collections with their point counts and configurations.

Create a Qdrant vector collection. Auto-configures vector dimensions from embedding model name.

Get detailed info and stats for a Qdrant collection.

Count points (vectors) in a Qdrant collection.

Generate Kubernetes StatefulSet manifest for self-hosted Qdrant vector database.

Run a DeepEval test suite from a Python test file.

List available DeepEval metrics for LLM evaluation.

Evaluate a single LLM output with a DeepEval metric (AnswerRelevancy, Faithfulness, Hallucination, etc.).

Generate a starter DeepEval test file.

Find the cheapest egress route between cloud providers/regions for model weights or dataset transfer. Supports multi-hop routing.

Optimize dataset or model staging across regions by finding the cheapest transfer plan. Integrates with the dataset stager for parallel uploads.

Start the model orchestrator for multi-model GPU sharing with eviction policies.

Register a model with the orchestrator.

Load a model into GPU memory.

Evict a model from GPU memory.

Get orchestrator and model status including GPU memory utilization.

Test inference with a model via the orchestrator.

Start the warm pool manager for intelligent model pre-warming. 5 strategies: traffic_based, time_based, priority_based, cost_optimized, latency_optimized.

Get warm pool status: hit rate, cold starts, memory saved, cost saved.

GPU price intelligence with quantitative analytics. Computes delta (rate of change), gamma (acceleration), and annualized realized volatility on GPU spot/on-demand prices across 21+ providers. Identifies cheapest time windows and provider arbitrage opportunities.

Deep cost analysis of current GPU infrastructure: per-provider breakdown, utilization efficiency, waste identification, and optimization potential.

Generate actionable cost optimization recommendations: spot migration, GPU right-sizing, provider arbitrage, idle shutdown, and density packing.

Simulate cost optimization scenarios with ROI projections. Compare current vs optimized infrastructure costs.

Get GPU price trend analysis with delta (rate of change), gamma (acceleration), and annualized volatility. Identifies cheapest time windows.

Spot instance risk assessment per provider. Returns interruption probability, mean time to interruption, and recommended mitigation.

Deploy Mixture-of-Experts models with production-ready cluster templates. Auto-applies vLLM cost optimizations (KV cache offloading for up to 9x throughput, MTP speculative decoding for up to 2.8x speed, sleep mode for 18-200x faster restarts). Supports GLM-5, Qwen 3.5, Mistral Large 3, DeepSeek V4, Llama 5. Configures NVLink topology, tensor parallelism, FP8 quantization, vLLM/SGLang backends, and GPU-aware HPA autoscaling.

List all training jobs and their state (created, running, completed, failed).

Real-time GPU monitoring for training jobs. Shows utilization, memory, temperature, power, and cost.

List all checkpoints for a training job.

Manually trigger a checkpoint save for a running training job.

Stop a running training job. Kills training processes on all nodes in parallel.

Resume a training job from its latest checkpoint. Rebuilds config with topology revalidation.

Restore a specific checkpoint for a training job.

Promote a checkpoint to a final model path for serving.

Delete a checkpoint.

List LoRA adapters loaded on a running vLLM endpoint. Shows base models and hot-loaded fine-tuned adapters.

Hot-load a LoRA adapter onto a running vLLM endpoint. The adapter becomes immediately available as a model name for inference requests. Uses vLLM's fused_moe_lora kernel for 454% higher output tokens/sec on MoE models.

Hot-unload a LoRA adapter from a running vLLM endpoint. Frees GPU memory for other adapters.

Generate a complete training configuration from a declarative spec. Auto-detects framework, sets optimal parallelism, and configures distributed training.

Full distributed training launch with framework auto-detection, topology validation, and monitoring. Combines preflight + train + monitor in one operation.

Get complete training monitoring snapshot: GPU metrics (utilization, memory, temp, power), training metrics (loss, grad_norm, lr, throughput), straggler detection, and cost estimate.

Detect straggler nodes in distributed training. Identifies GPUs with significantly lower utilization that slow the whole job.

Configure agentic inference serving settings.

Print Helm values for agentic inference deployment.

Print K8s deployment manifests for agentic inference.

Print engine launch arguments for copy-paste.

Print LMCache environment variables.

Show current agentic serving configuration.

Show real-time cost breakdown for a fleet by tier.

Provision a heterogeneous agent fleet across all tiers simultaneously.

Create a LangGraph workflow.

Create an SGLang pipeline.

Create a LangChain workflow.

Test connection to LangChain service.

Create a LangGraph workflow.

Deploy a workflow.

Get workflow status.

Test connection to LangGraph service.

Send a message to a Letta agent.

Create a new stateful Letta agent.

Delete a Letta agent.

List Letta agents.

Teach a Letta agent a durable fact.

Show the state of a Letta agent.

Store a memory in Mem0 for an agent or user.