InfluxData Boosts Telemetry Processing: 94% Storage Cut, Sub-10ms Queries
InfluxData has significantly enhanced its telemetry processing capabilities. Aron Koren, Senior Developer Advocate at InfluxData, led the improvements, which include reducing network round trips, boosting query speed, and slashing storage needs.
The team implemented batching, capturing six complete telemetry snapshots per database write. This cut down network round trips and improved efficiency.
Koren and his team built a telemetry pipeline handling 5,400 data points per second. They achieved sub-10 millisecond query responses, a remarkable feat in high-frequency data processing.
InfluxDB 3 Enterprise's aggressive compaction was configured, reducing storage from 500MB to a mere 30MB, a 94% reduction. This not only saves space but also speeds up data retrieval.
Related metrics were grouped into memory blocks, reducing API calls from 2,700 to 5,400 calls per second down to a mere 240 to 480 calls per second, a 90%+ reduction. This optimizes resource usage and prevents API call overload.
The team utilized InfluxDB 3 Enterprise's Last Value Cache, reducing query time from 30+ seconds to less than 10ms. This dramatic improvement ensures real-time data access and swift decision-making.
To further enhance efficiency, real-time queries were separated from historical ones, allowing for more efficient data handling.
Koren's team discovered techniques for processing flight simulator data at 60FPS. These techniques, applicable to high-frequency telemetry systems, have significantly boosted InfluxData's capabilities.
The enhancements have led to a 94% storage reduction, 90% fewer API calls, and a dramatic query time reduction from 30 seconds to sub-10ms. These improvements have not only boosted InfluxData's performance but also enhanced the user experience, providing smoother and more responsive dashboards.
