15

What IoT services are available for storing/sending/publishing (and opposite operations) generic small amount of data in the cloud?

I am looking, for example, for a service where a device could store a value in the cloud. And some other entity (another device, a website with some JS code, a web server, a mobile app) could retrieve this value.

It could be some kind of asynchronous communication, for example for storing and retrieving something as small as a key&value pair, <255bytes, an integer, a string, at most a small JSON object. The service could offer a REST API (so it could be accessed by a large variety of languages) with some token to authenticate and the key & value to store.

A detailed example of a use case is:

There is a temperature sensor at home, and I want it to store the value in the cloud(somewhere out of home). In such way, I could access it, regardless of my home connection is down or not. Morevoer, this would avoid keeping and maintaining a dedicated server + portForwarding + DynamicDNS.

So far I have not been able to find something like this, but occasionally, I have found some examples of what I mean to describe:

What other similar (free/open) alternatives are there?

  • @BenceKaulics I think both use cases would apply. The professional solution could imply more standarization and reliability. The experimental solution could be a interesting open-source project to join. – urnenfeld Jan 4 '17 at 19:29
  • I hate to be a spoil sport, but I don't see this as an IoT request, more like web serives, for which we have other sites. Use whatever you like - your own site, free hosting, Amazon Web Services, etc. You haven't really told us what you are trying to achieve. For instance, do you want to code any of the server software? No details, just a warm, fuzzy feeling, so it will be very difficult to answer you. – Mawg Jan 5 '17 at 13:54
  • 2
    @Mawg It has the web-service tag(which I did not create). The use case is IoT focused, as the references given are. It asks for services available, while this could be achieved by a simple php+db, the intention is not reinventing the wheel and get to know which services the IoT community is aware of. But let me know in which community you think it fits better. – urnenfeld Jan 5 '17 at 17:02
  • It's a close call. I will admit that I sometimes post on Stack Overflow, where another site would be more correct. BUT, S.O has a bigger audience, so I have more chance of a reply. There is always a bit of overlap with other sites and we will have to work it out as we go along. +1 for not taking offence & now you are aware of the security site. The question *could still use a little more detail, though :-) – Mawg Jan 6 '17 at 8:26
  • 2
    @Mawg of course no offence :). I really spent some time thinking where to place the question... I did try to shape more the question before, Let's see now, I added the exact use case, and tentative solutions. – urnenfeld Jan 6 '17 at 15:27
10

Take a look at these services:

Both of these services will accept simple key/value data from a device. I believe they both have libraries ready to import for Particle devices since you mentioned that.

7

Many cloud providers like Amazon, Microsoft, Google, IBM etc., trying to attract IoT space by providing easier ways to send/store/analyze sensors data to their cloud. Even they are to acquire hardware vendors to increase their scope in IoT.

I didn't use any services other than AWS, so I can explain my experience with AWS and how we integrated this for production usage.


Scenario:

We have hundreds of sensors each one sending 184-428 bytes of data every minute to the local gateway which aggregates the data and stores the data locally and sends the same data to AWS cloud. We also have special-cased sensors which send data directly to the cloud.

Cloud Services

We are using AWS IoT, AWS S3, AWS DynamoDB, AWS Lambda, AWS API Gateway, AWS SNS, AWS Cloudwatch, AWS RedShift to build entire solution. Basically, these are not specific to IoT(Except AWS IoT) as we can use them for mobile, web.

Gateway uses AWS IoT SDK to connect, authenticate, and exchange messages with AWS IoT using the MQTT, HTTP, or WebSockets protocols(We are using node JS SDK connecting via MQTT). We are MQTT broker locally on the device gateway and bridging it to AWS IoT endpoint from there we are running instant checks on the data received (using rules engine, AWS Lambda functions) and storing them into DynamoDB archiving into S3,Glacier(Storing is done without writing single line of done just using AWS triggers to store data).

6

It is for experimental or test use only but maybe it will change in the future.

So my suggestion is to use MQTT, more precisely its Mosquitto implementation. They host a test broker to which you could connect your subscriber and publisher clients. (Here is a guide about the setup process on Windows 7.)

Note the following:

You are free to use it for any application, but please do not abuse or rely upon it for anything of importance. You should also build your client to cope with the broker restarting.

But basically you can publish temperature data to this broker.

On subscriber-client side I have recently used this Android application. It is a very basic application, still in development but for test purposes it is very great. Received messages are shown on a dashboard, nothing fancy just the bare values.

I have started to use both of these as a first stage of a my MQTT learning process and found both great for beginners.

  • 1
    Would not MQTT be a solution more fitting in a LAN context? For problems like privacy/security/multiuser? – urnenfeld Jan 4 '17 at 21:47
  • @urnenfeld MQTT can support your website, mobile app and "other device" requirement as you can subscribe with all of them. Also certainly you would like to send data from other devices in the future and not just temperature so multiuser capability is good IMO even if it is only a multi publisher single subscriber network. Also security and privacy should always be a problem. As for LAN, perhaps it fits that more but with if you take care of security you could make your broker available from everywhere, not an expert in this matter. – Bence Kaulics Jan 5 '17 at 9:14
  • 1
    Erm, this test broker is a test broker so it should not be used for a running application. But the idea is here : you can open the port of your MQTT broker from your firewall and connect from everywhere. – Goufalite Jan 5 '17 at 13:28
  • I'm sorry but OP wants to store data, MQTT is just a messenger but you can use it to send data to a dedicated server. To retrieve a value the client must send a query to MQTT to retrieve the value through a publication. MQTT is great but a webservice has a better place here... – Goufalite Jan 5 '17 at 15:44
  • @Goufalite Yes, storing data is more complicated. – Bence Kaulics Jan 5 '17 at 15:52
5

There are two components to this:

  1. How do you want your data stored? There is no real way to make a "generic" data service that will really fit all needs. What you want is called a "Time Series Databases", and there are hundreds of them because every detail of how you store data matters at scale. (If you are not at scale, just store it in any old database, it will work for a while.)

    Each time series database was written because the others didn't do exactly they way they wanted. For example, consider how Graphite stores it's data: Every metric (say temperature from one source) is stored in a fixed-size file. No matter how often you send metrics, or how long you send them, the file is a constant size.

    The downside is that older data is at a lower resolution, and after a set interval you define (such as 1 year) the data is thrown away. But the upside is that it's just as fast to graph a day as a year, and metrics don't grow in size over time.

    In other storage systems, generating a graph for a year might involve fetching millions of data points, and might require massive amounts of data storage.

    The big downside of Graphite is that every metric creates a new file, so if you have dynamic metrics (say cloud boxes coming and going), it might not be a good fit.

    Contrast that with Prometheus, where metrics are stored mostly by time. You can have lots of dynamic metrics, and it's fine. But don't try to store those metrics for the long-term, it will take forever to go back and read them.

    No one size will fit all.

    P.S. Graphana is an excellent way to visualize your data. It has plug-ins for most time-series databases.

  2. Who is going to store your data? There are thousands of places like the ones you mentioned. It's so easy to spin up a time-series DB in the cloud, but it's REALLY hard to make money at it.Most of these companies will go out of business after a while, or start price gouging. (Even keeping their prices steady is price gouging -- since the cost of computing is going down constantly.) Many times, they find they can't attract as many new customers as they though, so they try to raise prices (under the guise of changing their pricing model). It turns out it costs a LOT of money to store everybody's data...

    I recommend self-hosting, or use a reputable cloud provider such as an AWS CloudWatch. (Expensive if you have a lot of metrics, but free for under 50 metrics!)

  • The idea was to get a set of the best of those "thousand of places". I have already collected a collection(even done my own) which will post here sometime – urnenfeld Dec 21 '17 at 9:40
3

uBeac is a new freeware visualization tools which we have developed and it is Beta version. It is not opensource, but fully free to use.

You can define a gateway and you will get a unique URI. You can set the URI in you gateway or device to send HTTP/MQTT data to.

These are some of its features:

  • Team, building, floor plan definition
  • Real-time data visualization
  • Custom dashboard design using different widgets
  • Define sensor types, units, and prefixes Map visualization
  • ...

It supports generic Json data format and different predefined gateway as well. If you don't want to use predefined payload formats, they are open to develop your custom payload processing.

  • Hi @Amir, we've noticed your answers so far are duplicates of each other and are promoting your product. Answering where your product is relevant is fine, but please be sure to read the self-promotion guidelines in the help center. It is also a good idea to tailor your answers to the question at hand rather than posting identical text to ensure that you're actually answering the question fully rather than just promoting your product. Thanks. – Aurora0001 Apr 8 at 9:45
1

I'm surprised no one on here mentioned Dweet. This is a super simple, super fun way to get things communicating. You should for sure try it out, because hey, it's free!

  • 1
    It was indeed proposed, but moderation removed it. One of the cool things of this service is that you don't need even to register, and they show some plotting of your data. On the counterpart, your data is pretty public :) – urnenfeld Mar 14 at 7:59
0

flespi provides free and commercial cloud services:

Obligatory disclaimer: I’m working for the company that develops the flespi platform. While I did my best to stay objective, as always on the Internet, please double check all information in this answer to rule out bias that may affect your decision.

  • Again, please declare any connection to a service when recommending it (e.g. that you work for flespi) – hardillb Mar 14 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.