Decoupling data fetching from processing and publishing

Question

I am working on a project that involves raspberry pi and the accelerometer sensor ADXL355. The raspberry's responsibility is to fetch the data from the sensor, filter only useful data and publish it to a thingsboard.io instance on the cloud. All the different devices are powered by a battery, charged by a solar panel and monitored by the Raspberry.

As of now this is my setup:

This setup works fine but I would like to add more reliability on fetching data. It is vital to me that every single datapoint is fetched and processed. This set up doesn't allow me to operate safely on the linux os without possibly interfere with the process of fetching data. I would like to be able to restart the raspberry without losing data.

The sensor offers a fifo buffer functionality: it will save in an internal buffer up to 32 (xyz) datapoints, but considering that the sampling rate could go up to 1KHz, that leaves me with a 32 milliseconds buffer, which is nowhere close to enough to sustain a possible deploy of a new firmware or reboot of the OS of the Raspberry.

Sometimes (even though I applied the RT patch) and set the priority of the program that fetches data to the highest possible, I see intervals of up to 18ms before the program gets its context back (from some kernel interrupt routine I assume?), I'm worried that it could happen that the interval gets bigger than the 32 ms buffer that I have and potentially lose data.

---

What I wanted to know is what are the best practices regarding decoupling data fetch from the transmission and processing of said data?

What I thought about was to move the sensor in a dedicated Arduino board, which runs a bare metal firmware with as little responsibilities as possible fetching the data, assigning a timestamp, saving the data.

With this approach I wouldn't know to access the buffer without interfering with the program which is fetching the data. Is there any specific component that solves my problem? Like some sort of memory component that can be accessed simultaneously by different "users"? This buffer should be able to save at least up to 1GB worth of data to make up for a possible downtime window of 24 hours.

Answer 1

Perhaps you can have the arduino be in one of two modes.

• A default pass thru mode, where it is just sending out what it gets from the sensor

• An aggregate mode, where it is taking the accelerometer inputs and computing a net displacement (or whatever end result you need).

When the pi wants to reboot, it can put the arduino in aggregate mode for the duration of the reboot.

To solve the issue of missing occasional values, maybe implement a circular buffer in the arduino and tag the readings with a sequence number. Maybe a seq range of 0-127 would be enough ? you only have 8k RAM in the arduino ?

Answer 2

I suppose your current data fetching is in a user space process, but given you see scheduling problems you could consider moving the data fetching into the kernel space, by implementing a kernel driver for your sensor in C. This driver could implement a buffering larger than the sensor itself provides.

On top of that there can be a user space application that could do the processing which is not that time critical and it can do the data backup if the uplink connection is not available. Storing 1 GB of data could be done here as well, using the file system.

Regarding reboot and OS updates on the Raspberry Pi, it is a tough task to fetch data from the sensor in the meantime. In this case you would definitely need additional hardware with a microcontroller that could still read the sensor and store the data and act as a buffer. There is no need for a 3rd buffer module, the microcontroller should be the buffer. Depending on the reboot time you might have to add additional memory capacity to the microcontroller to be able to cover the whole duration.

On the other hand storing 1 GB of data here is a different level so this unit should be used for covering the reboot time of the Raspberry only, nothing more. If there is no uplink the Raspberry Pi should still store the data by itself, then it could be sent when connection comes back.

Personally I do not think that you should cover reboots and OS upgrades as these should not happen that often when the application gets finalized and deployed. (During development it should not matter if you loose some data I think.) Regular updates and reboots should be documented that these kind of operations are traffic affecting and should be executed as a maintenance task. During the system is not operational. But if this is the requirement then this is the requirement. I understand that, then it is extra hardware but you do not have to consider writing a kernel driver.

I also agree with @jstola that the easiest would be to get a WiFi enabled module and hook it up with the sensor eliminating the complexity and the bulkiness of an RPi. OTA (Over the air) programming are usually supported by the companies developing the platforms you can choose from. There is OTA programming support for ESP with Arduino as well, so you would have plenty of choices for CI/CD. Based on the information you have shared so far about your application you would not even require on OS I think.