S-Query: Opening the Black Box of Internal Stream Processor State

The idea

Distributed streaming systems grew up as analytics engines, and their state management matured to serve fault tolerance: partitioned state, coordinated checkpoints, exactly-once processing. Meanwhile the workloads moved on — model serving, microservices, cloud applications built on stateful functions. For those, the operators' internal state is effectively a materialised view of the application's data, and it is invisible. S-Query argues that exposing it turns a black box into something you can query, audit and debug.

Contributions

• An approach and a reference architecture for querying the internal distributed state of a stream processor, with different isolation levels rather than one fixed guarantee.
• Two complementary modes: queries over the live state, which are fresh but carry no correctness guarantee, and queries over the snapshot state produced by periodic checkpoints, which are consistent but slightly behind.
• An implementation in an existing open-source stream processor, Hazelcast Jet, rather than a purpose-built prototype.

What it found

The snapshot configuration adds up to 8 ms of latency at the 99.99th percentile and a negligible increase from the 0th to the 90th — that is, the cost of making state queryable lands almost entirely in the tail, and is small even there.

Prior art could look up keys in live state (TSpoon, Apache Flink) but not join, filter or aggregate across it, which is what an external application actually wants.