LSM-based storage with disk-backed roaring postings, optional PromQL, atomic snapshot/restore, and structured observability counters. Embed a full time-series engine directly in your application — no external server required.
use std::time::Duration;
use tsink::{DataPoint, Label, Row, StorageBuilder, WalReplayMode, WalSyncMode};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new()
.with_data_path("./tsink-data")
.with_wal_sync_mode(WalSyncMode::Periodic(Duration::from_secs(1)))
.with_wal_replay_mode(WalReplayMode::Salvage)
.with_memory_limit(512 * 1024 * 1024)
.build()?;
storage.insert_rows(&[
Row::with_labels(
"cpu_usage",
vec![Label::new("host", "server-01")],
DataPoint::new(1_700_000_000, 45.5),
),
])?;
let obs = storage.observability_snapshot();
println!("wal_bytes={}", obs.wal.size_bytes);
storage.snapshot(std::path::Path::new("./snapshots/tsink-001"))?;
storage.close()?;
Ok(())
}
0.9.0 introduces storage format v2, atomic snapshot/restore, deep observability, and tighter safety defaults. Note: persistent data from 0.8.x is not wire-compatible with 0.9.0.
Persistent series dictionaries moved to series_index.bin with roaring-bitmap postings for faster matcher candidate resolution on reopen and query.
Metricless selector semantics, one-to-one vector matching for binary ops, and metric-name handling in vector binary results were aligned in 0.9.0.
Create segment-consistent, WAL-aware snapshots from live storage and restore atomically with staged publish semantics.
observability_snapshot() now exposes structured WAL, flush, compaction, query, and health internals for diagnostics and monitoring.
Choose Salvage or Strict replay mode for corruption handling, and prevent multi-process write collisions with a per-path .tsink.lock.
tsink-server deepens /metrics and /api/v1/status/tsdb, and adds gated admin snapshot/restore endpoints with optional path-prefix restrictions.
Everything you need for embedded time-series workloads, with production-grade durability and introspection.
Adaptive codec selection across Gorilla XOR, delta/bitpack, and RLE; chunk payloads are also zstd-compressed when beneficial. Typical numeric workloads reach ~0.4 bytes per point (~40x).
Tiered L0 → L1 → L2 segment compaction runs in the background. Reduces read amplification while maintaining write throughput with dual numeric/blob lanes.
Instant and range queries with 20+ built-in functions, full by/without aggregation support, and 0.9.0 semantic alignment for metricless selectors and vector matching.
64 internal shards eliminate write contention. Lock-free reads with concurrent writes via configurable worker pools. Full Send + Sync safety.
Six value types: f64, i64, u64, bool, bytes, and string. Per-type codecs, automatic From conversions, and custom bytes aggregation via Codec/Aggregator traits.
Segmented WAL with CRC32 checksums and idempotent replay. Choose Periodic or PerAppend sync, and pick replay policy: Salvage (default) or Strict.
Inspect structured WAL, flush, compaction, query, and health internals using observability_snapshot() for dashboards, alerting, and troubleshooting.
Runtime-agnostic AsyncStorage with bounded queues and dedicated workers. Forwards core builder options including WAL replay mode and background fail-fast.
Create live, segment-consistent backups with snapshot() and restore atomically with StorageBuilder::restore_from_snapshot.
Prometheus-compatible HTTP server with remote read/write, PromQL API, TLS, bearer auth, admin snapshot/restore endpoints (gated), and deep self-observability.
Matcher-based series filtering with =, !=, =~, !~. 0.9.0 persists dictionaries and roaring postings to accelerate matcher candidate resolution.
12 built-in aggregations: Sum, Min, Max, Avg, First, Last, Count, Median, Range, Variance, StdDev, and None. Supports downsampling, pagination, and custom bytes aggregation.
Memory budget with admission backpressure, series cardinality cap, and WAL size limit. 0.9.0 adds incremental shard memory accounting and optional background fail-fast mode.
Automatic cgroup v1/v2 CPU quota detection sets sensible worker defaults in containers. Override explicitly with TSINK_MAX_CPUS when needed.
Native Python bindings via UniFFI. Full access to the storage engine from Python with pip install tsink_uniffi — no performance compromise.
Ballpark figures from cargo bench in quick mode. Run on your hardware for precise numbers.
Add tsink to your Cargo.toml and start storing time-series data.
# Core library
tsink = "0.9"
# With PromQL support
tsink = { version = "0.9", features = ["promql"] }
# With async storage
tsink = { version = "0.9", features = ["async-storage"] }
use tsink::{DataPoint, Label, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
storage.insert_rows(&[
Row::new("cpu_usage", DataPoint::new(1_700_000_000, 42.5)),
Row::new("cpu_usage", DataPoint::new(1_700_000_010, 43.1)),
Row::with_labels(
"http_requests",
vec![Label::new("method", "GET"), Label::new("status", "200")],
DataPoint::new(1_700_000_000, 120u64),
),
])?;
// Time range is [start, end) (end-exclusive)
let cpu = storage.select("cpu_usage", &[], 1_700_000_000, 1_700_000_100)?;
assert_eq!(cpu.len(), 2);
// Label order does not matter for series identity
let get_200 = storage.select(
"http_requests",
&[Label::new("status", "200"), Label::new("method", "GET")],
1_700_000_000, 1_700_000_100,
)?;
assert_eq!(get_200.len(), 1);
storage.close()?;
Ok(())
}
use std::time::Duration;
use tsink::{StorageBuilder, TimestampPrecision, WalReplayMode, WalSyncMode};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new()
.with_data_path("./tsink-data")
.with_timestamp_precision(TimestampPrecision::Milliseconds)
.with_retention(Duration::from_secs(30 * 24 * 3600)) // 30 days
.with_partition_duration(Duration::from_secs(6 * 3600)) // 6-hour partitions
.with_chunk_points(4096)
.with_max_writers(16)
.with_write_timeout(Duration::from_secs(60))
.with_memory_limit(1024 * 1024 * 1024) // 1 GB
.with_cardinality_limit(500_000)
.with_wal_sync_mode(WalSyncMode::Periodic(Duration::from_secs(1)))
.with_wal_replay_mode(WalReplayMode::Salvage)
.with_background_fail_fast(true)
.with_wal_buffer_size(16384)
.build()?;
// Data survives restarts. WAL ensures durability on crash.
// Replay mode controls corruption handling at startup.
// Background fail-fast can stop new ops after worker failures.
storage.close()?;
Ok(())
}
use tsink::{DataPoint, Label, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
// Insert with multi-dimensional labels
storage.insert_rows(&[
Row::with_labels(
"http_requests",
vec![
Label::new("method", "GET"),
Label::new("status", "200"),
Label::new("endpoint", "/api/users"),
],
DataPoint::new(1_700_000_000, 150.0),
),
Row::with_labels(
"http_requests",
vec![
Label::new("method", "POST"),
Label::new("status", "201"),
],
DataPoint::new(1_700_000_000, 25.0),
),
])?;
// Query all label combinations for a metric
let all = storage.select_all("http_requests", 1_700_000_000, 1_700_000_100)?;
for (labels, points) in &all {
println!("Labels: {:?}, Points: {}", labels, points.len());
}
storage.close()?;
Ok(())
}
use tsink::{Aggregation, DataPoint, QueryOptions, Row, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
storage.insert_rows(&[
Row::new("cpu", DataPoint::new(1_000, 1.0)),
Row::new("cpu", DataPoint::new(2_000, 2.0)),
Row::new("cpu", DataPoint::new(3_000, 3.0)),
Row::new("cpu", DataPoint::new(4_500, 1.5)),
])?;
// Downsample: 2-second buckets with average aggregation
let opts = QueryOptions::new(1_000, 5_000)
.with_downsample(2_000, Aggregation::Avg)
.with_pagination(0, Some(10));
let buckets = storage.select_with_options("cpu", opts)?;
assert_eq!(buckets.len(), 2);
// 12 aggregation functions available:
// Sum, Min, Max, Avg, First, Last, Count,
// Median, Range, Variance, StdDev, None
storage.close()?;
Ok(())
}
// Enable with: tsink = { version = "0.9", features = ["promql"] }
use std::sync::Arc;
use tsink::{StorageBuilder, DataPoint, Row};
use tsink::promql::Engine;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
storage.insert_rows(&[
Row::new("http_requests_total", DataPoint::new(1_000, 10.0)),
Row::new("http_requests_total", DataPoint::new(2_000, 25.0)),
Row::new("http_requests_total", DataPoint::new(3_000, 50.0)),
])?;
let engine = Engine::new(storage.clone());
// Instant query — evaluates at a single point in time
let result = engine.instant_query("http_requests_total", 3_000)?;
// Range query — evaluates at each step across a time window
let result = engine.range_query(
"rate(http_requests_total[1m])",
1_000, 3_000, 1_000,
)?;
// Supports: rate, irate, increase, avg_over_time, sum_over_time,
// abs, ceil, floor, round, clamp, label_replace, label_join,
// sort, sort_desc, scalar, vector, time, timestamp, and more
storage.close()?;
Ok(())
}
// Enable with: tsink = { version = "0.9", features = ["async-storage"] }
use tsink::{AsyncStorageBuilder, DataPoint, Row};
async fn run() -> tsink::Result<()> {
let storage = AsyncStorageBuilder::new()
.with_queue_capacity(1024)
.with_read_workers(4)
.build()?;
// Async insert
storage.insert_rows(vec![
Row::new("cpu", DataPoint::new(1, 42.0)),
]).await?;
// Async query
let points = storage.select("cpu", vec![], 0, 10).await?;
assert_eq!(points.len(), 1);
// Synchronous introspection available too:
// storage.memory_used(), storage.memory_budget(),
// storage.inner(), storage.into_inner()
storage.close().await?;
Ok(())
}
use tsink::{SeriesSelection, SeriesMatcher, StorageBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let storage = StorageBuilder::new().build()?;
// ... insert data ...
// Matcher-based series discovery
let selection = SeriesSelection::new()
.with_metric("http_requests")
.with_matcher(SeriesMatcher::equal("method", "GET"))
.with_matcher(SeriesMatcher::regex_match("status", "2.."))
.with_time_range(1_700_000_000, 1_700_100_000);
let series = storage.select_series(&selection)?;
// Matcher operators: = (equal), != (not_equal),
// =~ (regex_match), !~ (regex_no_match)
// Also: list_metrics() and list_metrics_with_wal()
let all_series = storage.list_metrics()?;
for s in &all_series {
println!("Metric: {}, Labels: {:?}", s.name, s.labels);
}
storage.close()?;
Ok(())
}
Enable with the promql feature flag. Built-in lexer/parser/evaluator with 20+ functions and 0.9.0 semantic alignment updates.
rate
irate
increase
avg_over_time
sum_over_time
min_over_time
max_over_time
count_over_time
abs
ceil
floor
round
clamp
clamp_min
clamp_max
scalar
vector
time
timestamp
sort
sort_desc
label_replace
label_join
sum
avg
min
max
count
topk
bottomk
+ - * / % ^
== != < > <= >=
and or unless
With on/ignoring vector matching, by/without grouping, and bool. 0.9.0 enforces stricter one-to-one vector matching semantics.
Run tsink as a standalone HTTP service with remote read/write, PromQL API, TLS, bearer auth, and gated admin snapshot/restore endpoints.
# Start the server (admin API is opt-in)
$ cargo run -p tsink-server -- server \
--listen 127.0.0.1:9201 \
--data-path ./tsink-data \
--memory-limit 1G \
--retention 14d \
--auth-token secret-token \
--enable-admin-api \
--admin-path-prefix /srv/tsink-admin
# PromQL instant query
$ curl -H 'Authorization: Bearer secret-token' \
'http://localhost:9201/api/v1/query?query=up&time=1700000000'
# Create snapshot via admin API
$ curl -X POST http://localhost:9201/api/v1/admin/snapshot \
-H 'Authorization: Bearer secret-token' \
-H 'Content-Type: application/json' \
-d '{"path":"/srv/tsink-admin/snap-001"}'
/healthz
Health check
/ready
Readiness probe
/metrics
Self-monitoring
/api/v1/query
PromQL instant
/api/v1/query_range
PromQL range
/api/v1/write
Remote write
/api/v1/read
Remote read
/api/v1/series
Series metadata
/api/v1/labels
Label names
/api/v1/label/<name>/values
Values by label
/api/v1/import/prometheus
Text ingestion
/api/v1/status/tsdb
TSDB stats
/api/v1/admin/snapshot
Admin snapshot
/api/v1/admin/restore
Admin restore
/api/v1/admin/delete_series
Admin delete (stub 501)
remote_write:
- url: http://127.0.0.1:9201/api/v1/write
remote_read:
- url: http://127.0.0.1:9201/api/v1/read
LSM-style architecture with sharded concurrency, dual encoding lanes, and a disk-backed global series index.
StorageBuilderStorageAsyncStoragePromQL Engineseries_index.bin accelerate reopen and matcher-based selection
The encoder picks the most compact lane codec, then optionally wraps chunk payloads with zstd when it reduces size.
Fixed-step RLE
Run-length encoding for fixed-interval timestamps
Delta-of-delta bitpack
Primary strategy with bit-packing
Delta varint
Varint-encoded deltas for irregular intervals
Gorilla XOR
XOR of IEEE 754 floats (f64)
Zigzag delta bitpack
Zigzag + delta + bit-packing (i64)
Delta bitpack
Delta encoding + bit-packing (u64)
Constant RLE
Run-length for constant values (any numeric)
Bool bitpack
1 bit per value (bool)
Delta block
Blob lane compression (bytes, string)
Zstd wrapper (optional)
Chunk payload compressed at level 1 when smaller; decoded transparently on reads
Every type, trait, and method you need to build with tsink.
The main interface for all time-series operations. Thread-safe (Send + Sync).
insert_rows(&self, rows: &[Row]) -> Result<()>
Write one or more data points in a single batch
select(&self, metric, labels, start, end) -> Result<Vec<DataPoint>>
Query points by metric name, label filters, and time range
select_into(&self, metric, labels, start, end, buf) -> Result<()>
Write into a caller-provided buffer for allocation reuse
select_all(&self, metric, start, end) -> Result<Vec<(Labels, Points)>>
Query all label combinations for a metric
select_with_options(&self, metric, opts) -> Result<Vec<DataPoint>>
Downsampling, aggregation, custom bytes aggregation, pagination
select_series(&self, selection: &SeriesSelection) -> Result<Vec<MetricSeries>>
Matcher-based series discovery (=, !=, =~, !~)
list_metrics(&self) -> Result<Vec<MetricSeries>>
Discover all metric series in the database
list_metrics_with_wal(&self) -> Result<Vec<MetricSeries>>
Like list_metrics, including WAL-only series
memory_used(&self) / memory_budget(&self)
Inspect current/target memory footprint
observability_snapshot(&self) -> StorageObservabilitySnapshot
Structured WAL/flush/compaction/query/health internals
snapshot(&self, destination) -> Result<()>
Create an atomic on-disk snapshot from a live storage
close(&self) -> Result<()>
Gracefully shut down, flushing all buffers and WAL
Fluent builder for creating configured Storage instances.
StorageBuilder::new() -> Self
Create builder with sensible defaults
.with_data_path(path) -> Self
Enable persistent disk storage at the given path
.with_retention(duration) -> Self
Set automatic data cleanup policy (default: 14 days)
.with_memory_limit(bytes) -> Self
Hard in-memory budget with admission backpressure
.with_cardinality_limit(n) -> Self
Cap total unique metric+label series
.with_wal_size_limit(bytes) -> Self
Hard cap on total WAL bytes on disk
.with_chunk_points(n) -> Self
Target points per chunk before flushing (default: 2048)
.with_wal_replay_mode(mode) -> Self
Choose replay behavior on mid-log corruption (Salvage/Strict)
.with_background_fail_fast(bool) -> Self
Stop new operations after background worker failures
.build() -> Result<Arc<dyn Storage>>
Construct the storage engine
StorageBuilder::restore_from_snapshot(snapshot, data_path)
Atomically restore a data directory from snapshot contents
The fundamental data structures for time-series storage.
DataPoint { timestamp: i64, value: Value }
Timestamp + typed value (f64, i64, u64, bool, bytes, string)
Value::F64 | I64 | U64 | Bool | Bytes | String
Six typed variants with auto From conversions
Row::new(metric, data_point)
A metric name paired with a single data point
Row::with_labels(metric, labels, data_point)
A data point with multi-dimensional label metadata
Label::new(name, value)
Key-value pair for dimensional tagging
Codec / Aggregator traits
Custom bytes encoding and aggregation logic
StorageObservabilitySnapshot { wal, flush, compaction, query, health }
Structured runtime snapshot for diagnostics and monitoring
Advanced query customization with downsampling, aggregation, and series discovery.
QueryOptions::new(start, end)
Create a time-range query
.with_downsample(interval, agg) -> Self
Bucket data into time intervals with aggregation
.with_custom_bytes_aggregation(codec, agg)
Custom aggregation for non-numeric data
Aggregation::{ None, Sum, Min, Max, Avg, First, Last, Count, Median, Range, Variance, StdDev }
12 built-in aggregation functions + passthrough mode
SeriesSelection / SeriesMatcher
Matcher-based series discovery with regex support
Enums for precision, durability, and comprehensive error handling.
TimestampPrecision::{ Nanoseconds, Microseconds, Milliseconds, Seconds }
Unit for interpreting timestamp values
WalSyncMode::{ PerAppend, Periodic(Duration) }
WAL fsync strategy for durability vs throughput
WalReplayMode::{ Salvage, Strict }
Startup replay policy when WAL corruption is encountered
TsinkError::{ InvalidTimeRange, WriteTimeout, MemoryBudgetExceeded, CardinalityLimitExceeded, WalSizeLimitExceeded, ValueTypeMismatch, DataCorruption, ... }
Comprehensive error types for all failure modes
Full PromQL query engine enabled via the promql feature flag.
promql::Engine::new(storage) -> Engine
Create PromQL engine over a storage handle
promql::Engine::with_precision(storage, precision)
Construct engine for timestamps not expressed in nanoseconds
engine.instant_query(query, time) -> Result<...>
Evaluate PromQL at a single timestamp
engine.range_query(query, start, end, step) -> Result<...>
Evaluate PromQL across a time window
Runtime-agnostic async facade enabled via the async-storage feature flag.
AsyncStorageBuilder::new().build() -> Result<AsyncStorage>
Runtime-agnostic async facade with bounded queues
.with_wal_replay_mode(mode) / .with_background_fail_fast(bool)
Forward durability and fail-fast controls to inner storage builder
async_storage.insert_rows(rows).await
Async insert via dedicated worker threads
async_storage.select(metric, labels, start, end).await
Async query with configurable read workers
async_storage.snapshot(path).await
Create atomic snapshots via async write worker
Tune every aspect of the storage engine via the builder API.
| Option | Description | Default |
|---|---|---|
with_data_path(path) |
Directory for persistent segment files and WAL | None (in-memory) |
with_chunk_points(n) |
Target data points per chunk before flushing | 2048 |
with_retention(dur) |
How long to keep data before auto-cleanup | 14 days |
with_retention_enforced(bool) |
Enforce retention window (false keeps data forever) | true |
with_timestamp_precision(p) |
Nanoseconds, Microseconds, Milliseconds, or Seconds | Nanoseconds |
with_max_writers(n) |
Maximum concurrent write workers (cgroup-aware) | CPU count |
with_write_timeout(dur) |
Timeout for write operations | 30 seconds |
with_partition_duration(dur) |
Time range covered by each partition | 1 hour |
with_memory_limit(bytes) |
Hard in-memory budget with admission backpressure | Unlimited |
with_cardinality_limit(n) |
Max unique metric + label-set series | Unlimited |
with_wal_enabled(bool) |
Enable write-ahead logging for durability | true |
with_wal_sync_mode(mode) |
WAL fsync strategy: Periodic(Duration) or PerAppend |
Periodic(1s) |
with_wal_replay_mode(mode) |
Replay policy on WAL corruption: Salvage or Strict |
Salvage |
with_wal_buffer_size(n) |
WAL buffer size in bytes | 4096 |
with_wal_size_limit(bytes) |
Hard cap on total WAL bytes across all segments | Unlimited |
with_background_fail_fast(bool) |
Stop new operations after background flush/compaction worker failures | false |
Add tsink to your Rust project. Store millions of points with adaptive compression, snapshot and restore atomically, inspect runtime internals, and query with PromQL or server mode.