Runink: Rethinking data pipelines with Go, Raft, and Linux primitives

Combine zero-copy streaming, functional pipelines, and auditable governance into one secure, open source data platform.

What is Runink?

TL;DR: Runink is a streamlined, Go-native platform for managing secure and efficient data pipelines without Kubernetes. It uses Linux primitives (namespaces, cgroups) for isolation, Raft consensus for strong consistency and governance, and complies with open standards such as OpenLineage, Open Data Contracts, and OpenTelemetry—simplifying orchestration, enhancing performance, and ensuring transparent data governance.

---

Modern data platforms often rely on complex stacks involving Kubernetes, Spark, Airflow, and DBT—each tool addressing part of the data pipeline lifecycle. Runink presents a radically simplified, vertically integrated alternative: A Go-native distributed pipeline execution and governance platform built upon Linux primitives and Raft consensus. In this article, we’ll explore the core components of Runink—Runink CLI/API, Runi Agent/Slices, the Barn Domain Model Control Plane, and Herd Domain Boundary Controls—and demonstrate how they redefine pipeline orchestration.

Runink CLI/API: Developer experience first

Runink emphasizes developer ergonomics with its powerful CLI, runi, designed to streamline pipeline creation and management. Instead of YAML sprawl and disparate tools, Runink provides a unified interface for contract management, scenario compilation, and golden testing. Developers declaratively define pipelines using the intuitive Runink DSL, facilitating rapid, test-driven development and CI/CD automation.

Runi agent and runi slices: Lightweight, stream-oriented isolation

At the execution layer, Runink deploys Runi Agents on each node to orchestrate isolated Runi Slices—ephemeral Go processes launched via os/exec, scoped with chroot, and confined using Linux cgroups and namespaces. Instead of spinning up containers, slices stream data through io.Reader/Writer pipelines, backed by buffered I/O and os.Pipe()—enabling zero-copy, backpressure-aware processing without materializing entire datasets. Each slice is evaluated in isolation using eval, subject to strict CPU and memory limits, making execution not only fast and memory-stable, but also auditable and secure by default.

Read more: How I use AI agents to automate my workflow and save hours

Barn domain model control plane: Raft-backed governance

The core orchestration is managed by Barn, Runink’s domain model control plane built upon Raft consensus. Barn provides strong consistency across cluster state, secrets management, and scheduling, eliminating common distributed system pitfalls like split-brain conditions. The Raft-backed store ensures deterministic task placement and robust fault tolerance, critical for secure and auditable data pipelines.

Herd domain boundary controls: Multi-tenancy and security

Runink introduces Herds as the primary isolation boundary—composed of one or more Runi Agents and their associated Runi Slices. Each Herd defines a domain-specific execution environment with its own RBAC policies, resource quotas, and namespace-scoped metadata. By grouping Runi slices under a shared Herd, Runink ensures strict tenant separation, secure data handling, and controlled resource usage—all enforced natively via cgroups and Linux namespaces. This built-in, slice-aware governance model eliminates the complexity of external orchestrators while providing robust multi-tenancy and compliance guarantees.

Read more: Your 2025 guide to AI, no-code, and developer-led software

Built-in data governance and lineage

Runink captures detailed lineage information natively, embedding run metadata directly into pipeline execution. This lineage includes schema contracts, transformation versions, input/output mappings, and runtime metrics, enabling precise audit trails and regulatory compliance. Governance becomes integral to pipeline execution rather than an external process, ensuring data responsibility at every step.

Zero-copy, functional pipelines with declarative scheduling

Runink emphasizes zero-copy, streaming pipelines using Go’s efficient I/O primitives. Data flows through pipelines without unnecessary buffering, significantly enhancing performance and memory efficiency. Its functional programming approach simplifies pipeline testing, guaranteeing correctness through golden tests and schema contracts.

Declarative scheduling completes the picture, allowing engineers to specify resource requirements and constraints explicitly, leaving the scheduler to determine optimal execution placement using Raft-consistent state.

---

By leveraging Go, Linux primitives, and Raft, Runink offers a unique, high-performance, secure-by-default alternative to conventional pipeline stacks. Its integrated design simplifies operations, increases performance, and provides comprehensive data governance capabilities—redefining how platform teams and data engineers build and manage their data infrastructure.

Get started: runink.org

More from We Love Open Source

• What is OpenTelemetry?
• 6 limitations of AI code assistants
• Your 2025 guide to AI, no-code, and developer-led software
• How I use AI agents to automate my workflow and save hours
• Why Kubernetes is essential for AI and open source projects in 2025