Thursday, January 20, 2011

[Data In] Email bulk data into Nimbits

At Nimbits, we're all about trying to give you as many ways to get data in and out of the data logging engine as possible. Now, you can send an email to Nimbits and have it's contents processed and recorded into your data points.  When the email is received the data is processed and all of your configured calculations, alerts, Instant MessagesFacebook posts etc will fire.

Since your Nimbits account is tied to your email address, the data is handled based on the email address the system receives the email from.
Besides being able to bulk load historical data, you can also use this method to feed data into Nimbits from Linux shell scripts, powershell or any other system that can send an email.  

To make use of this feature, just send an email to: data@nimbits1.appspotmail.com

If you're running your own Nimbits Server instance on appspot, just replace Nimbits1 with your app id. 

You do not have to include a subject. In the message body add the point name (case sensitive) and a value separated by commas and the different points separated by semi-colons like this:

Point1,3.14;Point2,5.6

and so on.  Values are recorded with the current time (UTC) when the value is processed. If you want to specify a time stamp you can do so by including the time you want in UNIX time format. You can do a bulk load of point data to a single point by providing values like this:  

Point1,45.4,1278448474000;Point1,89.3;127844000000;Point1,44.324,12784480000


Please note the time is in milliseconds, not seconds since the unix epoch. 

There are many ways to get the current time in UNIX time - Java developers can use the Java.util.date getTime() function. I posted a handy way C# developers can work with UNIX time here.




6 comments:

  1. I wasn't able to get the timestamp part to work -- all of my data was imported with the current time rather than the time I specified. The data I sent was formatted like this:

    Test1,61.16,1294637100;Test1,62.76,1294637400;Test1,59.48,1294637700;Test1,54.65,1294638000

    Any idea what I'm doing wrong? Aside from the timestamp, everything else was recorded quickly (and almost instantaneously - impressive!)

    Thanks,

    Dan.

    ReplyDelete
  2. Hi Dan, thanks for using Nimbits. I think the problem is my example was misleading. The unix time you send needs to be in milliseconds, not seconds. I updated the article and if you multiply your timestamp values by 1000 (or tack on 000) it should work.

    ReplyDelete
  3. Hi Ben,

    Thanks. Converting to milliseconds fixed the timestamp problem.

    However, I'm running into a different problem now. I'm finding that that only the first 2-3 points in each e-mail are recorded, with the remainder being discarded.

    For example, I sent the following string (a series of points 5 minutes apart), and only the 1st 2 points (237.98 and 236.82) are recorded

    Test1,237.98,1294615500000;Test1,236.82,1294615800000;Test1,227.63,1294616100000;Test1,233.24,1294616400000;Test1,239.75,1294616700000;

    I've only done a quick-and-dirty test sending from Gmail through the browser interface, but I wonder if the problem is something obvious.

    Is there is a limit on how many points can be sent per e-mail? Is sending the e-mail from Gmail's browser interface (as opposed to SMTP) likely to cause problems because it reformats the data?

    Thanks,

    Dan.

    ReplyDelete
  4. Hi Dan,

    Thanks for bringing that to my attention. I just put in a little fix and tested a large bulk email that seems to work. I think most users only sent in one or two at a time, i'd like to test large volumes a little more.


    I generally use Gmail browser interface for my testing, so we support that. Please let me know if you run into any more trouble.

    As far as limits, you should be able to send in large volumes. Please take a look at the Terms section on www.nimbits.com - essentially app.nimbits.com is open to the public to use, but if you need to process huge amounts of data, you should download and set up your own instance of nimbits server.

    ReplyDelete
  5. Hi Ben,

    I just tried e-mailing 168 points, with timestamps 5 mins apart, and it works great now.

    I've started using Nimbits as a repository for energy usage data sent from Adafruit's "Tweet-A-Watt" package. So far, it has worked without a hitch - sending data with the REST API from Python, every 5 minutes. I used the e-mail bulk interface to upload some older data.

    Nimbits seems ideal for this type of project, and it's so easy to get up-and-running.

    Thanks,

    Dan.

    ReplyDelete
  6. Homestead Bulk Email Restrictions Solution- Aweber. Homestead temporarily suspended my email account because of the amount of subscribers

    Email Blast

    ReplyDelete