A high-performance packet analytics pipeline. From the ground up.

Wirelake is built on a zero-copy AF_XDP capture core, protocol-native decoders, and Apache Arrow/Parquet output. Every component is designed for sustained 100Gbps throughput with no packet loss and no compromise on data fidelity.

Capture Layer

AF_XDP with eBPF/XDP for kernel-bypass, zero-copy packet ingestion. Traffic is distributed across RSS queues, with a dedicated capture thread per queue. This eliminates kernel overhead and ensures sustained line-rate throughput even on dense, high-PPS traffic.

Key properties

  • → AF_XDP zero-copy — no kernel/userspace memory copies
  • → eBPF/XDP for early packet classification and steering
  • → Per-RSS-queue threading — no contention, linear scaling
  • → Compatible with Intel X710, E810, and other XDP-native NICs

Decode Layer

Protocol decoders run in dedicated threads, pulling from per-protocol queues. Decoders are implemented in C for maximum performance. Each decoder produces structured, typed records — not raw bytes — ready for columnar serialisation.

Supported protocols (v1)

  • → DNS (UDP/TCP, all record types)
  • → GTP-U / GTP-C (mobile carrier core network)
  • → Multicast UDP (financial market data feeds)
  • → More protocols added per customer requirements

Storage Layer

Decoded records are written to Apache Parquet via the Arrow C GLib and Parquet GLib libraries. Files are partitioned by hour using Hive-style directory layout, enabling efficient predicate pushdown in DuckDB, Spark, and Trino.

/mnt/store1/<protocol>_parquet/ts_hour=<epoch_hours>/

Schema conventions

  • → IP addresses stored as binary(16) (IPv4-mapped IPv6)
  • → Timestamps in nanoseconds (ts_ns)
  • → Flow IDs via Toeplitz hashing
  • → Per-minute summary Parquet files for fast dashboard queries

Query Layer

No proprietary query engine. Wirelake output is standard Parquet — queryable with any tool that speaks Apache Arrow.

DuckDB Apache Spark / Trino Grafana (DuckDB datasource plugin) Python / pandas / Polars

Multi-node deployment.

For deployments spanning multiple capture nodes, Wirelake includes an Arrow Flight SQL gateway for cross-node query federation. Nodes register with Consul KV using semantic tags, enabling routing to specific node subsets by location, protocol, or role — without enumerating IP addresses. An optional async warm tier via MinIO allows historical data to be offloaded from local NVMe. The local write path never depends on network availability.

On-premises. Always.

Wirelake is deployed on your hardware, in your facility. Packet data never traverses a network you don't control. This is a hard design requirement, not a configuration option.

Requirements

  • → Linux (Ubuntu 22.04 LTS or later)
  • → Intel E810-C or X710-DA2 NIC (XDP native mode required)
  • → Local NVMe storage
  • → No cloud dependencies on the capture or write path

Management tools

Consul KV Prometheus + Grafana Grafana Loki Ansible

See Wirelake on your infrastructure.

We work with teams to run a proof of concept on your hardware, with your traffic. No commitment, no obligation.

Get a Demo →