LSM-tree storage engine with Gorilla compression, PromQL queries, async API, and a Prometheus-compatible server. Embed a full time-series engine directly in your application — no external server required.
use std::time::Duration;
use tsink::{DataPoint, Label, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new()
.with_data_path("./tsink-data")
.with_retention(Duration::from_secs(7 * 24 * 3600))
.with_memory_limit(512 * 1024 * 1024)
.build()?;
storage.insert_rows(&[
Row::with_labels(
"cpu_usage",
vec![Label::new("host", "server-01")],
DataPoint::new(1_700_000_000, 45.5),
),
])?;
// PromQL query engine
let engine = tsink::promql::Engine::new(storage.clone());
let result = engine.instant_query("cpu_usage", 1_700_000_000)?;
storage.close()?;
Ok(())
}
0.8.0 is a complete storage engine rewrite with a new LSM-tree architecture, PromQL engine, and much more.
Complete rewrite with L0/L1/L2 tiered compaction, background merges, and dual-lane encoding for numeric and blob data.
Full lexer, parser, and evaluator with 23 built-in functions, 15 binary operators, and 7 aggregation operators.
f64, i64, u64, bool, bytes, and string — with per-type codecs and automatic From conversions.
Runtime-agnostic AsyncStorage with bounded queues, plus a tokio-based HTTP server with Prometheus remote read/write.
Memory budget enforcement, series cardinality cap, and WAL size limits with graduated pressure relief.
Checksummed WAL segments with fsync, idempotent replay recovery, and configurable size limits.
Everything you need for time-series data, nothing you don't.
Adaptive codec selection: Gorilla XOR for floats, delta-of-delta bitpacking, zigzag encoding, and constant RLE. Achieves ~0.68 bytes per data point — a 23x compression ratio.
Tiered L0 → L1 → L2 segment compaction runs in the background. Reduces read amplification while maintaining write throughput with dual numeric/blob lanes.
Instant and range queries with 23 built-in functions including rate, irate, avg_over_time, and more. Full by/without aggregation support.
64 internal shards eliminate write contention. Lock-free reads with concurrent writes via configurable worker pools. Full Send + Sync safety.
Six value types: f64, i64, u64, bool, bytes, and string. Per-type codecs, automatic From conversions, and custom bytes aggregation via Codec/Aggregator traits.
Segmented WAL with CRC32 checksums and idempotent replay. Choose Periodic sync for throughput or PerAppend for strongest durability.
Runtime-agnostic AsyncStorage with bounded queues and dedicated worker threads. Works with tokio, async-std, or any executor.
Prometheus-compatible HTTP server with remote read/write, PromQL HTTP API, TLS, bearer auth, text format ingestion, and graceful shutdown.
Matcher-based series filtering with =, !=, =~, !~ operators. Discover series dynamically with select_series and regex label matching.
12 built-in functions: Sum, Min, Max, Avg, First, Last, Count, Median, Range, Variance, StdDev, and None. Downsampling with pagination built in.
Memory budget with admission backpressure, series cardinality cap, and WAL size limit. Prevents OOM crashes and unbounded growth.
Automatic cgroup v1/v2 detection for CPU and memory quotas. Optimal thread pool sizing in Docker, Kubernetes, and other containerized environments.
Ballpark figures from cargo bench in quick mode. Run on your hardware for precise numbers.
Add tsink to your Cargo.toml and start storing time-series data.
# Core library
tsink = "0.8.0"
# With PromQL support
tsink = { version = "0.8.0", features = ["promql"] }
# With async storage
tsink = { version = "0.8.0", features = ["async-storage"] }
use tsink::{DataPoint, Label, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
storage.insert_rows(&[
Row::new("cpu_usage", DataPoint::new(1_700_000_000, 42.5)),
Row::new("cpu_usage", DataPoint::new(1_700_000_010, 43.1)),
Row::with_labels(
"http_requests",
vec![Label::new("method", "GET"), Label::new("status", "200")],
DataPoint::new(1_700_000_000, 120u64),
),
])?;
// Time range is [start, end) (end-exclusive)
let cpu = storage.select("cpu_usage", &[], 1_700_000_000, 1_700_000_100)?;
assert_eq!(cpu.len(), 2);
// Label order does not matter for series identity
let get_200 = storage.select(
"http_requests",
&[Label::new("status", "200"), Label::new("method", "GET")],
1_700_000_000, 1_700_000_100,
)?;
assert_eq!(get_200.len(), 1);
storage.close()?;
Ok(())
}
use std::time::Duration;
use tsink::{StorageBuilder, WalSyncMode, TimestampPrecision};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new()
.with_data_path("./tsink-data")
.with_timestamp_precision(TimestampPrecision::Milliseconds)
.with_retention(Duration::from_secs(30 * 24 * 3600)) // 30 days
.with_partition_duration(Duration::from_secs(6 * 3600)) // 6-hour partitions
.with_chunk_points(4096)
.with_max_writers(16)
.with_write_timeout(Duration::from_secs(60))
.with_memory_limit(1024 * 1024 * 1024) // 1 GB
.with_cardinality_limit(500_000)
.with_wal_sync_mode(WalSyncMode::Periodic(Duration::from_secs(1)))
.with_wal_buffer_size(16384)
.build()?;
// Data survives restarts. WAL ensures durability on crash.
// Retention auto-cleans data older than 30 days.
// Memory budget prevents OOM with admission backpressure.
storage.close()?;
Ok(())
}
use tsink::{DataPoint, Label, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
// Insert with multi-dimensional labels
storage.insert_rows(&[
Row::with_labels(
"http_requests",
vec![
Label::new("method", "GET"),
Label::new("status", "200"),
Label::new("endpoint", "/api/users"),
],
DataPoint::new(1_700_000_000, 150.0),
),
Row::with_labels(
"http_requests",
vec![
Label::new("method", "POST"),
Label::new("status", "201"),
],
DataPoint::new(1_700_000_000, 25.0),
),
])?;
// Query all label combinations for a metric
let all = storage.select_all("http_requests", 1_700_000_000, 1_700_000_100)?;
for (labels, points) in &all {
println!("Labels: {:?}, Points: {}", labels, points.len());
}
storage.close()?;
Ok(())
}
use tsink::{Aggregation, DataPoint, QueryOptions, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
storage.insert_rows(&[
Row::new("cpu", DataPoint::new(1_000, 1.0)),
Row::new("cpu", DataPoint::new(2_000, 2.0)),
Row::new("cpu", DataPoint::new(3_000, 3.0)),
Row::new("cpu", DataPoint::new(4_500, 1.5)),
])?;
// Downsample: 2-second buckets with average aggregation
let opts = QueryOptions::new(1_000, 5_000)
.with_downsample(2_000, Aggregation::Avg)
.with_pagination(0, Some(10));
let buckets = storage.select_with_options("cpu", opts)?;
assert_eq!(buckets.len(), 2);
// 12 aggregation functions available:
// Sum, Min, Max, Avg, First, Last, Count,
// Median, Range, Variance, StdDev, None
storage.close()?;
Ok(())
}
// Enable with: tsink = { version = "0.8.0", features = ["promql"] }
use std::sync::Arc;
use tsink::{StorageBuilder, DataPoint, Row};
use tsink::promql::Engine;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
storage.insert_rows(&[
Row::new("http_requests_total", DataPoint::new(1_000, 10.0)),
Row::new("http_requests_total", DataPoint::new(2_000, 25.0)),
Row::new("http_requests_total", DataPoint::new(3_000, 50.0)),
])?;
let engine = Engine::new(storage.clone());
// Instant query — evaluates at a single point in time
let result = engine.instant_query("http_requests_total", 3_000)?;
// Range query — evaluates at each step across a time window
let result = engine.range_query(
"rate(http_requests_total[1m])",
1_000, 3_000, 1_000,
)?;
// Supports: rate, irate, increase, avg_over_time, sum_over_time,
// abs, ceil, floor, round, clamp, label_replace, label_join,
// sort, sort_desc, scalar, vector, time, timestamp, and more
storage.close()?;
Ok(())
}
// Enable with: tsink = { version = "0.8.0", features = ["async-storage"] }
use tsink::{AsyncStorageBuilder, DataPoint, Row};
async fn run() -> tsink::Result<()> {
let storage = AsyncStorageBuilder::new()
.with_queue_capacity(1024)
.with_read_workers(4)
.build()?;
// Async insert
storage.insert_rows(vec![
Row::new("cpu", DataPoint::new(1, 42.0)),
]).await?;
// Async query
let points = storage.select("cpu", vec![], 0, 10).await?;
assert_eq!(points.len(), 1);
// Synchronous introspection available too:
// storage.memory_used(), storage.memory_budget(),
// storage.inner(), storage.into_inner()
storage.close().await?;
Ok(())
}
use tsink::{SeriesSelection, SeriesMatcher, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
// ... insert data ...
// Matcher-based series discovery
let selection = SeriesSelection::new()
.with_metric("http_requests")
.with_matcher(SeriesMatcher::equal("method", "GET"))
.with_matcher(SeriesMatcher::regex_match("status", "2.."))
.with_time_range(1_700_000_000, 1_700_100_000);
let series = storage.select_series(&selection)?;
// Matcher operators: = (equal), != (not_equal),
// =~ (regex_match), !~ (regex_no_match)
// Also: list_metrics() and list_metrics_with_wal()
let all_series = storage.list_metrics()?;
for s in &all_series {
println!("Metric: {}, Labels: {:?}", s.name, s.labels);
}
storage.close()?;
Ok(())
}
Enable with the promql feature flag. Lexer, parser, and evaluator built from scratch.
rate
irate
increase
avg_over_time
sum_over_time
min_over_time
max_over_time
count_over_time
abs
ceil
floor
round
clamp
clamp_min
clamp_max
scalar
vector
time
timestamp
sort
sort_desc
label_replace
label_join
sum
avg
min
max
count
topk
bottomk
+ - * / % ^
== != < > <= >=
and or unless
With on/ignoring vector matching, by/without grouping, and bool modifier.
Run tsink as a standalone HTTP server with remote read/write, TLS, and bearer auth.
# Start the server
$ cargo run -p tsink-server -- server \
--listen 127.0.0.1:9201 \
--data-path ./tsink-data \
--memory-limit 1G \
--retention 14d
# PromQL instant query
$ curl 'http://localhost:9201/api/v1/query?query=up&time=1700000000'
# PromQL range query
$ curl 'http://localhost:9201/api/v1/query_range?query=up&start=1700000000&end=1700000060&step=15s'
# Ingest Prometheus text format
$ curl -X POST http://localhost:9201/api/v1/import/prometheus \
-H 'Content-Type: text/plain' \
--data-binary @metrics.txt
/healthz
Health check
/ready
Readiness probe
/metrics
Self-monitoring
/api/v1/query
PromQL instant
/api/v1/query_range
PromQL range
/api/v1/write
Remote write
/api/v1/read
Remote read
/api/v1/series
Series metadata
/api/v1/labels
Label names
/api/v1/import/prometheus
Text ingestion
/api/v1/status/tsdb
TSDB stats
remote_write:
- url: http://127.0.0.1:9201/api/v1/write
remote_read:
- url: http://127.0.0.1:9201/api/v1/read
LSM-tree architecture with sharded concurrency and dual encoding lanes.
StorageBuilderStorageAsyncStoragePromQL EngineThe encoder tries all applicable candidates and picks the most compact representation.
Fixed-step RLE
Run-length encoding for fixed-interval timestamps
Delta-of-delta bitpack
Primary strategy with bit-packing
Delta varint
Varint-encoded deltas for irregular intervals
Gorilla XOR
XOR of IEEE 754 floats (f64)
Zigzag delta bitpack
Zigzag + delta + bit-packing (i64)
Delta bitpack
Delta encoding + bit-packing (u64)
Constant RLE
Run-length for constant values (any numeric)
Bool bitpack
1 bit per value (bool)
Delta block
Blob lane compression (bytes, string)
Every type, trait, and method you need to build with tsink.
The main interface for all time-series operations. Thread-safe (Send + Sync).
insert_rows(&self, rows: &[Row]) -> Result<()>
Write one or more data points in a single batch
select(&self, metric, labels, start, end) -> Result<Vec<DataPoint>>
Query points by metric name, label filters, and time range
select_into(&self, metric, labels, start, end, buf) -> Result<()>
Write into a caller-provided buffer for allocation reuse
select_all(&self, metric, start, end) -> Result<Vec<(Labels, Points)>>
Query all label combinations for a metric
select_with_options(&self, metric, opts) -> Result<Vec<DataPoint>>
Downsampling, aggregation, custom bytes aggregation, pagination
select_series(&self, selection: &SeriesSelection) -> Result<...>
Matcher-based series discovery (=, !=, =~, !~)
list_metrics(&self) -> Result<Vec<MetricSeries>>
Discover all metric series in the database
list_metrics_with_wal(&self) -> Result<Vec<MetricSeries>>
Like list_metrics, including WAL-only series
close(&self) -> Result<()>
Gracefully shut down, flushing all buffers and WAL
Fluent builder for creating configured Storage instances.
StorageBuilder::new() -> Self
Create builder with sensible defaults
.with_data_path(path) -> Self
Enable persistent disk storage at the given path
.with_retention(duration) -> Self
Set automatic data cleanup policy (default: 14 days)
.with_memory_limit(bytes) -> Self
Hard in-memory budget with admission backpressure
.with_cardinality_limit(n) -> Self
Cap total unique metric+label series
.with_wal_size_limit(bytes) -> Self
Hard cap on total WAL bytes on disk
.with_chunk_points(n) -> Self
Target points per chunk before flushing (default: 2048)
.build() -> Result<Arc<dyn Storage>>
Construct the storage engine
The fundamental data structures for time-series storage.
DataPoint { timestamp: i64, value: Value }
Timestamp + typed value (f64, i64, u64, bool, bytes, string)
Value::F64 | I64 | U64 | Bool | Bytes | String
Six typed variants with auto From conversions
Row::new(metric, data_point)
A metric name paired with a single data point
Row::with_labels(metric, labels, data_point)
A data point with multi-dimensional label metadata
Label::new(name, value)
Key-value pair for dimensional tagging
Codec / Aggregator traits
Custom bytes encoding and aggregation logic
Advanced query customization with downsampling, aggregation, and series discovery.
QueryOptions::new(start, end)
Create a time-range query
.with_downsample(interval, agg) -> Self
Bucket data into time intervals with aggregation
.with_custom_bytes_aggregation(codec, agg)
Custom aggregation for non-numeric data
Aggregation::{ Sum, Min, Max, Avg, First, Last, Count, Median, Range, Variance, StdDev }
12 built-in aggregation functions
SeriesSelection / SeriesMatcher
Matcher-based series discovery with regex support
Enums for precision, durability, and comprehensive error handling.
TimestampPrecision::{ Nanoseconds, Microseconds, Milliseconds, Seconds }
Unit for interpreting timestamp values
WalSyncMode::{ PerAppend, Periodic(Duration) }
WAL fsync strategy for durability vs throughput
TsinkError::{ InvalidTimeRange, WriteTimeout, MemoryBudgetExceeded, CardinalityLimitExceeded, WalSizeLimitExceeded, ValueTypeMismatch, DataCorruption, ... }
Comprehensive error types for all failure modes
Optional engines enabled via Cargo feature flags.
promql::Engine::new(storage) -> Engine
Create PromQL engine over a storage handle
engine.instant_query(query, time) -> Result<...>
Evaluate PromQL at a single timestamp
engine.range_query(query, start, end, step) -> Result<...>
Evaluate PromQL across a time window
AsyncStorageBuilder::new().build() -> Result<AsyncStorage>
Runtime-agnostic async facade with bounded queues
async_storage.insert_rows(rows).await
Async insert via dedicated worker threads
async_storage.select(metric, labels, start, end).await
Async query with configurable read workers
Tune every aspect of the storage engine via the builder API.
| Option | Description | Default |
|---|---|---|
with_data_path(path) |
Directory for persistent segment files and WAL | None (in-memory) |
with_chunk_points(n) |
Target data points per chunk before flushing | 2048 |
with_retention(dur) |
How long to keep data before auto-cleanup | 14 days |
with_retention_enforced(bool) |
Enforce retention window (false keeps data forever) | true |
with_timestamp_precision(p) |
Nanoseconds, Microseconds, Milliseconds, or Seconds | Nanoseconds |
with_max_writers(n) |
Maximum concurrent write workers (cgroup-aware) | CPU count |
with_write_timeout(dur) |
Timeout for write operations | 30 seconds |
with_partition_duration(dur) |
Time range covered by each partition | 1 hour |
with_memory_limit(bytes) |
Hard in-memory budget with admission backpressure | Unlimited |
with_cardinality_limit(n) |
Max unique metric + label-set series | Unlimited |
with_wal_enabled(bool) |
Enable write-ahead logging for durability | true |
with_wal_sync_mode(mode) |
WAL fsync strategy: Periodic(Duration) or PerAppend |
Periodic(1s) |
with_wal_buffer_size(n) |
WAL buffer size in bytes | 4096 |
with_wal_size_limit(bytes) |
Hard cap on total WAL bytes across all segments | Unlimited |
Add tsink to your Rust project. Store millions of data points with sub-byte compression, query with PromQL, or run it as a Prometheus-compatible server.