RunInfra

What is RunInfra

RunInfra is an automated open-source AI model optimization and deployment API platform. It benchmarks GPUs, optimizes kernels, and deploys a production API with an exportable stack your team can inspect and own.

How to use RunInfra

1. Describe the inference goal: Start with the open-source model, workload shape, latency target, cost limit, quality bar, and deployment preference.
2. Benchmark GPU and runtime options: RunInfra checks compatible serving engines and GPU targets, then benchmarks latency, throughput, VRAM, and cost.
3. Apply compatible optimizations: RunInfra applies supported runtime settings, batching, quantization, KV cache, kernel, and routing optimizations only where the selected model and backend support them.
4. Review the evidence: The result is a benchmark receipt with measured before and after performance, cost, GPU fit, and reproduction notes.
5. Deploy or export the stack: Deploy the measured configuration on RunInfra Cloud or export the runnable code, Docker, Kubernetes, and runbook artifacts.

Features of RunInfra

• Chat-native AI application builder: Describe what you want to run, and RunInfra builds and optimizes the pipeline.
• Real GPU profiling and benchmarking: Benchmarks latency, throughput, VRAM, and cost across GPUs.
• Optimization across supported quantization, serving, and KV cache paths: Tries quantization, KV cache, serving, and kernel tweaks.
• Evidence-gated deployment with scale-to-zero: Deploy as REST endpoints in one click; scale-to-zero idle cost.
• OpenAI-compatible API endpoints: Deployed pipelines serve as REST endpoints.
• Managed and self-hosted deployment paths: Deploy on RunInfra Cloud or export the stack to your own cloud.
• Smart multi-model routing: Route requests across models.
• Pipeline versioning and comparison: Version and compare pipeline configurations.
• Speculative decoding and KV cache tuning: Optimize decoding and cache.
• Credit-based usage billing: 1 credit = $1, with scale-to-zero idle cost.

Use Cases of RunInfra

• Compare serving engines: Find the best serving engine for a model like Qwen 2.5 7B.
• Tune latency: Optimize a model for low latency.
• Ship speech: Deploy Whisper Large V3 Turbo with p95 and cost checks.
• Scale retrieval: Build BGE-M3 embeddings with batch throughput metrics.
• Support copilot: Build a support copilot with Whisper and Qwen, tuned for latency.

Pricing

RunInfra offers self-serve plans and enterprise deployment. Pricing is credit-based (1 credit = $1) with scale-to-zero idle cost. Low price: $50, high price: $1000 per month (USD).

FAQ

What is RunInfra? Describe what you want to run. RunInfra picks compatible open models, benchmarks GPUs, tunes the runtime, and gives you a deploy-ready stack.

How do I build my first pipeline? Type what you want, like 'a support copilot with Whisper and Qwen, tuned for latency.' RunInfra builds and optimizes the pipeline. Chat to refine it, then deploy.

Which AI models are supported? Vetted Hugging Face models. LLM serving is fully supported end to end. Speech, embedding, vision, and image generation models are supported for optimization and benchmarking, with managed serving in staged rollout.

How does GPU kernel optimization work? RunInfra profiles your model across GPUs, tries quantization, KV cache, serving, and kernel tweaks, and benchmarks the best tradeoff of speed, memory, and cost.

Can I deploy pipelines as APIs? Yes. Supported pipelines deploy as REST endpoints in one click. If something isn't deployable yet, RunInfra tells you why.

How is this different from using closed-source APIs? Closed APIs hide the model and the infrastructure. With RunInfra you see both, and you benchmark open models against your own latency, throughput, and cost targets. Export the stack or run it in your own cloud to own it outright.

Is my data secure? Encrypted in transit and at rest, on isolated infrastructure. Your inference data never trains anything and never leaves your deployment. RunInfra is SOC 2 Type II compliant.