Monday, October 10, 2011

Machine To Machine (M2M) Communication with Nimbits in the Cloud

You can use Nimbits to do much more than record sensor readings to the cloud. You can create a data point and store a history of data objects in them in any format that can be serialized into a string i.e JSON or XML.  A Nimbits data point can be used to relay data between software systems, or hardware devices such as arduino, using the cloud as a back end.

You can create streams of data objects, and store them in a data point series. The data can then be accessed from anywhere, and used to shape the behavior of connected devices and software.

In other words, you can serialize a class containing any sort of data, feed it into a data point on the cloud, and then read that data from anywhere to drive software systems, or devices connected to Nimbits data points.

This tutorial shows you how I use the Nimbits SDK to relay the tell a robot how to feel. Take, for example this class called Robot - it contains three emotional states, happy, sad and angry.

public class Robot {
    Emotion emotion;

    enum Emotion {
        happy,sad,angry
    }
    
    public Robot() { //don't forget a no arg constructor
    }
    
    public Emotion getEmotion() {
        return emotion;
    }

    public void setEmotion(Emotion emotion) {
        this.emotion = emotion;
    }  
}


I have a data point on Nimbits call MyRobot. I want to transmit an emotional state to the robot, as well as GPS Coordinates of where it is, a number value ranking the level of that state, and an instance of the class above:

public void sendData() throws Exception {
 Robot robot = new Robot();
 robot.setEmotion(Robot.Emotion.happy);
 Value value = ValueModelFactory
    .createValueModel(40.111, -75.146,10, 
    "I'm a happy robot", robot);

 client.recordValue(pointName, value);
}



Let's imagine somewhere else in the world, the robot is connected to the internet and is monitoring this data point on Nimbits. It can get the latest value being sent to it, recieving the robot class and the emotion being sent to it:

public void getData()  {
  
 Robot robot = (Robot) client.getCurrentDataObject(pointName, Robot.class);
 if (robot.getEmotion() == Robot.Emotion.angry) {
   destroyAllHumans();
 }
 else {
   eatHerringSandwiches();
 }


Our Robot downloads the current emotional state from the cloud and carries out it's instructions. In this case, the robot is happy so the human race is lucky.

What's happened here is this:

  • The data was recorded to the cloud
  • All of the things Nimbits does when it gets a new value in a Point is available, such as alerts, compression, calculations and so on. 
  • The stored value is instantly accessible from anywhere
  • The current value can be tied into other systems to trigger behavior.
Learn more at nimbits.com 




Wednesday, August 24, 2011

[Data In] Linux Command Line Data Logging

Nimbits provides you with an SDK (software development kit) that allows you to develop your own software using libraries we provided. These libraries simplify the use of the REST API which are web services you can also use to access Nimbits data.

Nimbits has a collection of REST web services you can use to create data points, categories, and read and write data to data points.  These services use Nimbits' built in authentication process using a secret key, or you can use Google account credentials. The SDK just saves you the trouble of figuring out google's authentication process and gives you easy to use functions to automate your use of Nimbits.

Debian and Ubuntu users can write scripts that use Nimbits by installing the SDK from our repository. This  simply installs the java nimbits library JAR on your system and puts a shell script called nimbits in your path so you can script against nimbits at the command line like this:


$nimbits action=recordvalue point=foo value=42.0 email=test@example.com password=secretpassword


Installing

Before installing Nimbits, you'll want to install Java and ensure it's on your class path. You can know this by opening a teminal and typing

$java -version

You can install java using a command like this:

$sudo apt-get install openjdk-6-jdk

You can install the Nimbits SDK using aptitude from our repository. This will keep you up to date with new versions of the SDK. Learn more about debian repositories here:

https://help.ubuntu.com/community/Repositories/Ubuntu

Simply edit your /etc/apt/sources.list file and add the following line to the bottom

deb http://deb.nimbits.net/packages/ ./
then run:


sudo apt-get update
sudo apt-get upgrade
You will be prompted to install nimbits. (you may also get a warning that you are unable to authenticate us, please enter Y To continue).


You can add /opt/nimbits/nimbits.jar to your java programs and use all of the methods in there to program against a Nimbits Server, or you can use the command line interface to write bash scripts. The SDK provides a tools for storing URL, password and account information in a hash file on your system.

Please visit the Nimbits SDK WIKI for more samples and information.

Example

Here is an example command the records a value to a data point using the hashed credential file mentioned above. The file contains the url of the Target Nimbits Server and credentials.

#!/bin/bash


KEY=[path to my key file]
nimbits -point=foo -action=recordvalue -value=1.0 -i=$KEY









Sunday, July 3, 2011

Process Diagrams with SVG



Whenever you walk in to manufacturing or process control centers, you see the giant overhead displays and monitors with graphics on them used to monitor systems.

These graphics were drawn by engineers using very expensive tools that then connect those graphics to data historians and process control systems. It was my desire to have this kind of interface for a home, small business, or hobbyist that first inspired me to develop Nimbits.


You can create a Process Diagram in Nimbits in SVG format to make a visual representation of your systems and display changing point values, color changing alerts and link to other diagrams and web pages.

I use InkScape to draw my diagrams. It is a free and open source project that runs on Windows and Linux. You can install it on Ubuntu right from the repository, or download it on windows, macs and Linux systems from http://inkscape.org/. Nimbits can display most SVG Diagrams, and the official doc for adding live point data to them is available here on the Nimbits Wiki.

I like to use my aquarium as an example for Nimbits, even though it can do much more than monitor fish. First, it's a simple, closed system with a lot of moving parts. Secondly, i'd like to see if I can make my fish tank a tax deduction.

Since Nimbits diagrams can be made public, so anyone can see them.  Feel free to take a look at my Fish Tanks current status right now here: Link to My Aquarium's Process Control Diagram.  You should see something like this (if your using IE, you may need to install Chrome Frame):



This graphic demonstrates the following features:


  • Try clicking on the F Temp to see a live chart of current values. 
  • Check out the Temp C value. This is the result of a real time calculation that converts my F reading to C. 
  • The two charts, one showing a water level and the other showing the Temp F are just Google Charts using the Nimbits Chart API. You can create any type of chart and embed them in your diagram. 
  • Check out the light, indicating the light is on or off. If you load this graphic at night on the east coast of the US, the light should be black.
  • Click on the Pump.  It will take you to a drill down graphic to see some data about the pump. 
  • Click on the links, they will take you to external web sites. 
  • The Data points in this diagram are shared as public points, you can mix private and public points, Nimbits will only show you what you're authorized to see.
  • Some of these values are entered manually. The Nitrate value expects a new value once a week, if I forget, the point goes Idle and turns red to remind me to do a test.

  • Log into Nimbits and create your Data Points.
  • Download InkScape and start a new drawing. 
  • Add a Text Box to your drawing
  • Click on the Text Box and Select Edit - Edit XML
  • Add a point=pointname value, and an action=[value,onoff,alert,idle]
  • Based on the action=value, Nimbits will do the following when the graphic is viewed:
    • value - show the current value in the box
    • onoff - if the value is zero, turn the shape black, else turn it yellow
    • alert - turn the shape red if the point is in an alert state.
    • idle - turn the shape red if the point is idle.
  • You can also draw a shape and add diagram=diagramName action=self to add a link to another diagram.

On the Nimbits Portal, select the "Upload Diagram" button above the Point tree to upload the diagram just like a Point, you can drag it into folders and organize it with your point.

You can then configure that graphic to share it publicly, or with your connected users.

When you are in the Nimbits Portal, you can click on the graphic to add it to the data display, enter values and trend by clicking on values.


Here is an XML snippet from my SVG diagram that shows the format of a Nimbits XML Node it is all generated by the drawing program, it's just a matter of adding the action and point properties.


<text action="idle,value" point="PH"  y="187.78877" x="-119.55978" id="tspan3034-1" id="tspan3198" /></text>





Please visit www.nimbits.com to learn more.


Wednesday, May 11, 2011

The Nimbits Annotated Timeline chart


The Nimbits Server UI uses the Google Visualization API to display data in the main portal. You can use the Annotated timeline to slice and dice your data by selecting time ranges and dragging other points into it. 

What's interesting about Nimbits and time ranges, is that it uses one logical algorithm to attempt to understand what time range you are requesting. This algorithm is used anywhere a timespan is being requested. So if you are pulling data from the chart API web service, downloading a series, or entering a time range into the chart, you can use the same date formats and nimbits will understand.


Some examples of timespan requests are a unix epoch time in milliseconds or seconds, a text date, or even an abstract date like "now" or "*" for the current time, and -1d for one day in the past from the end time. 

Nimbits interprets all date requests as a start and end time. Start times can be  relative to end times when using an abstract like -7d (the last 7 days) as above. 

When you double click on a data point in Nimbits, the portal opens with a point control window with the current value of the data point in it and the chart control.  In order to show useful information right away, Nimbits gets the date of the last recorded value for that point, and the last 100 recorded values from that date.  


Chart Features
  • Enter a start and end timestamp boxes integerate into Nimbit's date interpretation algorithm. Many combinations of date entiry is possible, including a Unix time, * or now in the end date for the current time, -1d -5h etc in the start time for a relative range to the start time. Read more here.
  • Annotations. Any recorded value with a text value in the note field will show as an annotation. Shown here as "Hello" 
  • Drag and drop. Drag your points or those from your connections into the chart to compare them on the same display.
  • Click the unpin button to move the chart, or maximize to full screen.
  • The zoom selection will zoom into the current data.  Note that you must select a time range and click the refresh button to first download the data you want, then use the zoom options to zoom into that data. The zoom options do not download data outside the selected time range. 
  • MinY and MaxY selections for changing the Y axis. 
  • Zoom slider at the bottom for zooming in.

Tuesday, April 19, 2011

Using Nimbits to log unit test performance metrics

Alternate title: Application, heal thy self.

This article will show you how to add a couple of lines of code to your Unit Tests to change them from simply telling you if your code passes or fails, to all singing all dancing integrations tests that log execution times, enforce SLAs, and even warn you if the test isn't running.

The reason for the alternate title above is that as I was writing the Nimbits source code, I realized that there was an incredible useful aspect to the Nimbits server: I could actually use the server to record how long my tests took to run, and see the history of this performance metric. Nimbits was actually using itself to improve performance. So I could see if changes I made to the code was improving performance or not.

Take, for example this test, which writes 10 random values to a data point called b1.


@Test
public void testLoadRandom() throws IOException {
Random r = new Random();
for (int i = 0; i < 10; i++) { Value v = nimbitsSDK.recordValue("b1", (r.nextDouble() * 100), new Date()); 

 Assert.assertNotNull(v); 
 Assert.assertTrue(v.getKey() > 0);
}
}


Good, this test will run once an hour on my Hudson server and fails if any of the 10 values do not get recorded into the b1 data point.   This of course, can be any test you are running on any project. In my example here i'm using Nimbits to test and tune Nimbits which I know is a little creepy.

You can add a couple of lines of code to this test to time how long it takes to write these 10 values - that is to make 10 round trips to Nimbits, waiting for each response.  We're then going to record how long the test took to run and save that time in another time series Data Point on a Nimbits Server. This way, we can better understand how our system is performing. What's more, there are many built in features we can take advantage of. Check this out.

Here is the same unit test with references to the Nimbits SDK, and some code to time the test. We then write the result to the data point



@Test
public void testLoadRandom() throws IOException {
Date start = new Date();
Random r = new Random();
for (int i = 0; i < 10; i++) { Value v = Common.client().recordValue("b1", (r.nextDouble() * 100), new Date()); Assert.assertNotNull(v); Assert.assertTrue(v.getKey() > 0);
}
Date end = new Date();
long time = (end.getTime() - start.getTime()) / 1000;
nimbitsSDK.recordValue("Time to record 10 values to b1", time, new Date());
}


I have a data point setup called "Time to record 10 values to b1" that holds the time each test took to run. Let's take a look at some of the value Nimbits brings to the table besides being able to see how our system performs.

Alerts:  A data point can be configured to send you an email, text message, twitter etc if a value exceeds a threshold. 


Why yes, I'd like to know if the time is over 20 seconds or below 0, both indicating something was wrong.


Idle Alarms: Nimbits can alert you if a data point is idle when it should have received a value.


Sure, the build can fail on an error, but what if the whole build server or Hudson is down? Nimbits can alert you if a point has not received a new value. Yes, i'd like to know if my hourly tests are not generating data.





See the impact of your code changes:


The test I wrote above has been humming away for several weeks. Here is a snapshot of the resulting times that spanned the upgrade of the Test Nimbits Server Instance to Nimbits 3.1, being released this week on www.nimbits.com.  Imagine going into your boss's office with a chart like this showing real improvement as a result of a code change. (Yes, Nimbits Server 3.1 is way faster)





Obvious improvement with the upgrade, one of the many uses of a Nimbits Server. You can use the live nimbits server, download your own, and get the Nimbits SDK for java and .net on www.nimbits.com

Sunday, March 6, 2011

Chat with your Nimbits Data Points over IM

You can chat with your Nimbits data over any XMPP compatible instant messenger client, And there are a lot of them! Chances are, the IM client on your phone, desktop or browser can talk to a Nimbits Server.

Using IM, you can create points, get current values, record values and list points just like a command prompt at a terminal.

To get started, log into your Nimbits Server - if you are using the public server, just log into app.nimbits.com. Click the "Request IM Invite" button on the top menu, and then check your IM client. You should get a request authorize the Nimbits Server to send you instant messages.


From there, you can send an IM to Nimbits with the following commands


  • ?  - Help - list these commands
  • c foo - create a new data point called foo
  • foo=42 - set the current value of foo to 42
  • foo=bar - set the current text value of foo to bar
  • ls - list all points and categories
  • foo?  get the current value of foo.


Your chat session may look like this:



After you authorise Nimbits to send you instant messages, you can also configure a data point to send you an alarm using IM, or just send you an IM any time a point gets a new value.

To set this up,  log into Nimbits and double click on a point, you can select the Point Properties menu to configure that data point.  You can then check off the box to receive an instant message from Nimbits any time that point records a new value or, under the alarms tab, get an IM if the value is above or below a limit you set.




Then, when a new value comes into that data point from anywhere, you'll get an IM from Nimbits.





Saturday, March 5, 2011

Connecting Spreadsheets To Nimbits

There are many ways to record historical values into Nimbits. You can either write software interfaces using the SDK, or record data manually to points using the Portal, Data Acquisition Studio, or the Android Data Logger.

The fun starts when you use all of the exiting ways to visualize your data as it changes.  The posting will help you get your data into popular spreadsheets.

The posting assumes you have logged into Nimbits http://app.nimbits.com and created a data point. For this tutorial, i'm using the point FProbe1 - the point i created in my blog posting showing how to feed a temperature into a data point using a Temp Probe.

You'll also want to set the point to public. Double click on the point to bring up it's properties and check off "Public."  This lets anyone see the current value of your point. There are ways to securly access your non-public points using a Secret key. But for this demo, let's keep it simple.  Since your points and data are connected to your google account, and spreadsheets like excel can't log into Google, we have to either skip the authentication process, or provide nimbits with a secret key so it knows who you are.

Making the point public really simplifies things for you, so for this demo let's do just that: (click to enlarge)



Now that I have a data point, and it's public. It's value can be accessed using a URL in this format:

http://app.nimbits.com/service/currentvalue?point=TempF&email=bsautner@gmail.com

You can go to that URL right now since my point is public. You'll get one little http response from the system with the current value of that point.

You'll want to replace the point name and your email address. You should be getting excited at this point realizing that if your data is available this way, there are a lot of fun ways to connect to Nimbits.

In Excel 2007 

1. Click on the cell you want your value to show up in
2. Select Data from the Ribbon and click the From Web icon in the Get External Data section
3. Enter your url like the one above and click and click Go.
4. There is no XML, the result of this service is just the value of the point, so the result page isn't pretty. It's ok, just click the yellow arrow next to your current value:



5. Click import and you're done!

6. Let's also setup excel so your values will update automatically. In my case, i have a temp probe feeding values into this point, so i want to see the changing values in excel

On the Data Ribbon, select Properties. Check off Enable Background refresh and to refresh the data every 5 minutes.



You can now use your data feed in your spreadsheet calculations and graphs. Imagine having a master spreadsheet at your office and seeing value coming in as your staff enter data on their Android based phones.



In Google Docs




1. Create a new spreadsheet in Google Docs.
2. Past this in a cell with your url:
=ImportData("http://app.nimbits.com/service/currentvalue?point=TempF&email=bsautner@gmail.com")

You should see your value load. Google Docs will automatically reload your data. Enjoy!

Thursday, January 20, 2011

[Data In] Email bulk data into Nimbits

At Nimbits, we're all about trying to give you as many ways to get data in and out of the data logging engine as possible. Now, you can send an email to Nimbits and have it's contents processed and recorded into your data points.  When the email is received the data is processed and all of your configured calculations, alerts, Instant MessagesFacebook posts etc will fire.

Since your Nimbits account is tied to your email address, the data is handled based on the email address the system receives the email from.
Besides being able to bulk load historical data, you can also use this method to feed data into Nimbits from Linux shell scripts, powershell or any other system that can send an email.  

To make use of this feature, just send an email to: data@nimbits1.appspotmail.com

If you're running your own Nimbits Server instance on appspot, just replace Nimbits1 with your app id. 

You do not have to include a subject. In the message body add the point name (case sensitive) and a value separated by commas and the different points separated by semi-colons like this:

Point1,3.14;Point2,5.6

and so on.  Values are recorded with the current time (UTC) when the value is processed. If you want to specify a time stamp you can do so by including the time you want in UNIX time format. You can do a bulk load of point data to a single point by providing values like this:  

Point1,45.4,1278448474000;Point1,89.3;127844000000;Point1,44.324,12784480000


Please note the time is in milliseconds, not seconds since the unix epoch. 

There are many ways to get the current time in UNIX time - Java developers can use the Java.util.date getTime() function. I posted a handy way C# developers can work with UNIX time here.




Wednesday, January 19, 2011

Nimbits Server 3.0.x Updates

Nimbits is a collection of software designed for managing time series data, such as reading from a temperature probe, on the cloud.  Nimbits Server is a data logging service that provides storage, compression, alarms, relay and calculation services with a REST inspired API for feeding data in and out of the system.

We've been releasing minor updated to the Nimbits Server 3.0 that includes improvements in the user interface and back end performance.  This summarizes what's new in version 3.0.5 - 3.0.9rc. 3.0.9rc is currently running on the public server, with 3.0.8 available for download.

  • More Mem Cache - commonly accessed data is held more often in memory rather than hitting the data store. This includes the most recent value for every data point, which gives us significant performance improvements.
  • Point Property Panel - A redesigned point property panel makes it easier to manage how a point behaves when it is written to.
  • Installation in the Chrome App Store - we enhanced Nimbits to run in the Chrome Web Store.
  • Trending Enhancements - we were hearing of some limitations users were experiencing when trying to chart large amounts of data. We addressed this by splitting data download requests into smaller chunks - which means you should be able to trend any amount of data that your computer can handle.
  • Added Motion Chart - watch a video of your data with bubbles changing over time with zoom, line and bar charts as well.




  • Improved calculation engine.  Performance improvements to the calc engine allows you to create a data point that will trigger a formula to be calculated each time the point is written to with a new value. The result of the calculation can then be stored in another point, further triggering alarms, other calculations etc.  Supported symbols include  *, +, -, *, /, ^, %, cos, sin, tan, acos, asin, atan, sqrt, sqr, log, min, max, ceil, floor, abs, neg, rndr.
Nimbits 3.1 is getting ready for a March release which will include:

  • A new Multi-tenant architecture that isolates point data into seperate slices on the cloud, which will give us a huge performance boost
  • A new feature where a point will go into alarm when it doesn't receive a new value in a set amount of time
  • Enhancements to the IM Chat feature. You can already interact with Nimbits using an IM Client. We're adding a bunch of new commands for getting your point data.