Perhaps you could use the Round Robin Database Tool? It certainly operate along the lines that you want.
RRDtool (round-robin database tool) aims to handle time series data such as network bandwidth, temperatures or CPU load. The data is stored in a circular buffer based database, thus the system storage footprint remains constant over time.
It also includes tools to extract round-robin data in a graphical format, for which it was originally intended. Bindings exist for several programming languages, e.g. Perl, Python, Ruby, Tcl, PHP and Lua. There is an independent full Java implementation called rrd4j
If you can’t find a port or build it yourself, then the same principles are demonstrated with MySql at Round Robin Database with MySQL.
It’s what we do on every project I have worked on as an embedded develop for the last few decades (although we roll our own):
- Some data are important/vital, and we do whatever we can to retain them all (not always possible, if there is along communications break)
- Some we need the last X entries for (and can dimension that to a fixed size at design time)
- Some we need the last X minutes’ worth of data (“round robin”, also size fixed at design time)
- Some are “nice to have”, but we can ditch them if the space is needed (not so common, though)
Plus, of course, there is the question of which to store in NVM. This is all really a question for your project’s software architect - unless that’s you, in which case it’s a question for us :-)
There are other things that can help, such as compressing the data, or Run Length Encoding (e.g don't save the values for 1,000 consecutive readings with the same value; just save the value once, with a repeat counter), etc