Saturday, October 30, 2010

[Data In / Out] Pushing and Pulling Data with URL Fetch

A Nimbits Data Point represents a thing, such as a temperature probe, stock price, anything. That point can become part of a system of connected objects, it's value can be used in calculations, alarms, triggers or anything else.

Further, when you make a data point public. It can be found on the Nimbits Data Point Search Engine so other users can connect their points to yours. If you wish, any http capable data logging system you have can be easily connected to the Nimbits Cloud using this method.

A Data Point can be configured to either push or pull (HTTP GET or POST) to a specified url on on a schedule or as an event.   URL Fetch is an app engine service, take a look at Google's guide to this for more details.

To familiarize yourself with this, try logging into the public Nimbits server on and create a new data point.  Double Click the data point to bring up its data screen. Then, use the Point Settings Menu to bring up the properties of this data point.

There is a tab for selecting URL Fetch Settings:

With these options, every time a program requests the current value of a data point, it can be configured to attempt to pull a new value from the source URL.   If you are hosting your own Nimbits Server on App Engine, you can set yourself up to fetch a new value on a schedule using CRON

That source url can be anything, a server on your network (with the ports open on your firewall). It can be a call to a Nimbits web service requesting the current value of another point. You can use the Nimbits Data Point Search Engine to find public points on other Nimbits Servers and pull that data into your system.

The relay service will take whatever value is written to that point and do an HTTP Get to the specified URL with the current value included in the parameters.  This way, a newly recorded value can act as a trigger to call another URL, triggering further events. 

Please note the caveat that fetch events occur when a new value is actually recorded, which means the value falls out of any compression tolerance settings you configured for a point. Typically, Nimbits only records new values.  If the same number sent into the system over and over it is ignored and will not trigger an event. 

When a URL is fetched, Nimbits is expecting a raw number value in the result. HTTP headers are ignored, there shouldn't be any HTML or XML.  If you load the URL browser, you should only see the number value. You can also provide a full JSON representation of a value object, which can include text annotations, GPS Coordinates, and a timestamp:

{"lat":0.0,"lng":0.0,"d":104.0,"note="hello world","timestamp":"Oct 29, 2010 9:48:28 PM"}

If this is what Nimbits pulls from your URL in a fetch, it would record the value 104.0 to the point with this timestamp with "Hello World" in the note annotation.