Saturday, March 17, 2012

Nimbits 3.2 - what's new

Nimbits 3.2 represents one of the biggest changes in the structure of the app since 6 years ago when I re-wrote the entire thing from C# to java.  We are hosting a BETA instance of Nimbits here:


We encourage anyone who has written an app using the public nimbits server, to test their code on the BETA to ensure their code will not break when the public server is upgraded sometime in April 2012.


Please report any bugs or issues you encounter while testing the beta using the issue menu on our github project portal: https://github.com/bsautner/com.nimbits



One of the most important things to keep in mind if you are running your own instance of Nimbits is to be sure to upgrade to version 3.1.19 which is available for download on nimbits.com.  When you upgrade, a task is executed to implement any changes we need to the back end database, and the 3.2 release assumes you are upgrading from version 3.1.19.

All of the web services behave the same way, except for the category service which has been completely removed - i.e if you are using /service/category you shouldn't be.

Here is a screen shot of the new UI, long time users should be getting excited right about now:


From the top of the screen to the bottom, here's whats new.


  • The data entry grid and navigation tree have been merged into one view
  • You can now organize your data any way you like, points can have child points, folders can go as deep as you like and so on.
  • The subscription data feed is a special point for each user that logs system messages, events, alerts from other points you are subscribed and more - I go into more detail on the subscription channel below. 
  • You can create calculation objects for a point and organize them in your tree - you can now have many formulas for a single data point. Here, I have two formulas under my temp probe data point, one for converting to Celsius and another for Kelvin each time a new temp is recorded.
  • The filter item in the tree called "ah" is for XMPP alerts. You can now have many JIDs for a single point - alerts and messages can be sent over XMPP using the points custom JID. 
  • Replacing the old style alerts - you now can create many subscriptions for a single point. Subscribe to one of your points, a friends, or anyone's public point to get alerts sent to you when the point goes into an alarm state, or receives new data.
  • Create a summary point which can compute averages, min, max, standard deviations, variance and sums of another point on an hourly basis. 
  • You can now upload any type of document and organize them with your points. You can share your files publicly or with your connections
  • You can create a connection to another Nimbits user by sending them an invitation. If they approve, you can see their points and data in their tree (if they set the objects permission level to public or connection viewable).



Introducing the Nimbits subscription data feed


On the right of your portal, you now have a data feed channel that shows you system events and messages, as well as data alerts you configure to show up in your feed. You can subscribe to any other users data point, and configure your subscription to send messages to your feed and you will see high, low and idle alerts here in real time.  Your data feed is just another Nimbits data point!

Simpler Data Entry

You may also notice on this screen shot that when you enter a value you can just type:  "42 ultimate answer" in the current value and Nimbits will record the number 42 to the points value and the words "ultimate answer"as an annotation for that value, simplifying data entry. 


Subscriptions

Subscriptions replace the old alert configurations. Before you would configure a point to alert you when it went high, low or idle using the point property menu. Now, you create a subscription to a point - configuring the subscription for how you want to be alerts (XMPP, Email, Twitter etc) and what event triggers the alert (new data, high alerts etc). You can have many subscriptions to a point, subscribe to a another Users point you've connected to or find a point using the search engine on nimbits.com and subscribe to a public point from another user you don't even know.




Monday, October 10, 2011

Machine To Machine (M2M) Communication with Nimbits in the Cloud

You can use Nimbits to do much more than record sensor readings to the cloud. You can create a data point and store a history of data objects in them in any format that can be serialized into a string i.e JSON or XML.  A Nimbits data point can be used to relay data between software systems, or hardware devices such as arduino, using the cloud as a back end.

You can create streams of data objects, and store them in a data point series. The data can then be accessed from anywhere, and used to shape the behavior of connected devices and software.

In other words, you can serialize a class containing any sort of data, feed it into a data point on the cloud, and then read that data from anywhere to drive software systems, or devices connected to Nimbits data points.

This tutorial shows you how I use the Nimbits SDK to relay the tell a robot how to feel. Take, for example this class called Robot - it contains three emotional states, happy, sad and angry.

public class Robot {
    Emotion emotion;

    enum Emotion {
        happy,sad,angry
    }
    
    public Robot() { //don't forget a no arg constructor
    }
    
    public Emotion getEmotion() {
        return emotion;
    }

    public void setEmotion(Emotion emotion) {
        this.emotion = emotion;
    }  
}


I have a data point on Nimbits call MyRobot. I want to transmit an emotional state to the robot, as well as GPS Coordinates of where it is, a number value ranking the level of that state, and an instance of the class above:

public void sendData() throws Exception {
 Robot robot = new Robot();
 robot.setEmotion(Robot.Emotion.happy);
 Value value = ValueModelFactory
    .createValueModel(40.111, -75.146,10, 
    "I'm a happy robot", robot);

 client.recordValue(pointName, value);
}



Let's imagine somewhere else in the world, the robot is connected to the internet and is monitoring this data point on Nimbits. It can get the latest value being sent to it, recieving the robot class and the emotion being sent to it:

public void getData()  {
  
 Robot robot = (Robot) client.getCurrentDataObject(pointName, Robot.class);
 if (robot.getEmotion() == Robot.Emotion.angry) {
   destroyAllHumans();
 }
 else {
   eatHerringSandwiches();
 }


Our Robot downloads the current emotional state from the cloud and carries out it's instructions. In this case, the robot is happy so the human race is lucky.

What's happened here is this:

  • The data was recorded to the cloud
  • All of the things Nimbits does when it gets a new value in a Point is available, such as alerts, compression, calculations and so on. 
  • The stored value is instantly accessible from anywhere
  • The current value can be tied into other systems to trigger behavior.
Learn more at nimbits.com 




Wednesday, August 24, 2011

[Data In] Linux Command Line Data Logging

Nimbits provides you with an SDK (software development kit) that allows you to develop your own software using libraries we provided. These libraries simplify the use of the REST API which are web services you can also use to access Nimbits data.

Nimbits has a collection of REST web services you can use to create data points, categories, and read and write data to data points.  These services use Nimbits' built in authentication process using a secret key, or you can use Google account credentials. The SDK just saves you the trouble of figuring out google's authentication process and gives you easy to use functions to automate your use of Nimbits.

Debian and Ubuntu users can write scripts that use Nimbits by installing the SDK from our repository. This  simply installs the java nimbits library JAR on your system and puts a shell script called nimbits in your path so you can script against nimbits at the command line like this:


$nimbits action=recordvalue point=foo value=42.0 email=test@example.com password=secretpassword


Installing

Before installing Nimbits, you'll want to install Java and ensure it's on your class path. You can know this by opening a teminal and typing

$java -version

You can install java using a command like this:

$sudo apt-get install openjdk-6-jdk

You can install the Nimbits SDK using aptitude from our repository. This will keep you up to date with new versions of the SDK. Learn more about debian repositories here:

https://help.ubuntu.com/community/Repositories/Ubuntu

Simply edit your /etc/apt/sources.list file and add the following line to the bottom

deb http://deb.nimbits.net/packages/ ./
then run:


sudo apt-get update
sudo apt-get upgrade
You will be prompted to install nimbits. (you may also get a warning that you are unable to authenticate us, please enter Y To continue).


You can add /opt/nimbits/nimbits.jar to your java programs and use all of the methods in there to program against a Nimbits Server, or you can use the command line interface to write bash scripts. The SDK provides a tools for storing URL, password and account information in a hash file on your system.

Please visit the Nimbits SDK WIKI for more samples and information.

Example

Here is an example command the records a value to a data point using the hashed credential file mentioned above. The file contains the url of the Target Nimbits Server and credentials.

#!/bin/bash


KEY=[path to my key file]
nimbits -point=foo -action=recordvalue -value=1.0 -i=$KEY









Sunday, July 3, 2011

Process Diagrams with SVG



Whenever you walk in to manufacturing or process control centers, you see the giant overhead displays and monitors with graphics on them used to monitor systems.

These graphics were drawn by engineers using very expensive tools that then connect those graphics to data historians and process control systems. It was my desire to have this kind of interface for a home, small business, or hobbyist that first inspired me to develop Nimbits.


You can create a Process Diagram in Nimbits in SVG format to make a visual representation of your systems and display changing point values, color changing alerts and link to other diagrams and web pages.

I use InkScape to draw my diagrams. It is a free and open source project that runs on Windows and Linux. You can install it on Ubuntu right from the repository, or download it on windows, macs and Linux systems from http://inkscape.org/. Nimbits can display most SVG Diagrams, and the official doc for adding live point data to them is available here on the Nimbits Wiki.

I like to use my aquarium as an example for Nimbits, even though it can do much more than monitor fish. First, it's a simple, closed system with a lot of moving parts. Secondly, i'd like to see if I can make my fish tank a tax deduction.

Since Nimbits diagrams can be made public, so anyone can see them.  Feel free to take a look at my Fish Tanks current status right now here: Link to My Aquarium's Process Control Diagram.  You should see something like this (if your using IE, you may need to install Chrome Frame):



This graphic demonstrates the following features:


  • Try clicking on the F Temp to see a live chart of current values. 
  • Check out the Temp C value. This is the result of a real time calculation that converts my F reading to C. 
  • The two charts, one showing a water level and the other showing the Temp F are just Google Charts using the Nimbits Chart API. You can create any type of chart and embed them in your diagram. 
  • Check out the light, indicating the light is on or off. If you load this graphic at night on the east coast of the US, the light should be black.
  • Click on the Pump.  It will take you to a drill down graphic to see some data about the pump. 
  • Click on the links, they will take you to external web sites. 
  • The Data points in this diagram are shared as public points, you can mix private and public points, Nimbits will only show you what you're authorized to see.
  • Some of these values are entered manually. The Nitrate value expects a new value once a week, if I forget, the point goes Idle and turns red to remind me to do a test.

  • Log into Nimbits and create your Data Points.
  • Download InkScape and start a new drawing. 
  • Add a Text Box to your drawing
  • Click on the Text Box and Select Edit - Edit XML
  • Add a point=pointname value, and an action=[value,onoff,alert,idle]
  • Based on the action=value, Nimbits will do the following when the graphic is viewed:
    • value - show the current value in the box
    • onoff - if the value is zero, turn the shape black, else turn it yellow
    • alert - turn the shape red if the point is in an alert state.
    • idle - turn the shape red if the point is idle.
  • You can also draw a shape and add diagram=diagramName action=self to add a link to another diagram.

On the Nimbits Portal, select the "Upload Diagram" button above the Point tree to upload the diagram just like a Point, you can drag it into folders and organize it with your point.

You can then configure that graphic to share it publicly, or with your connected users.

When you are in the Nimbits Portal, you can click on the graphic to add it to the data display, enter values and trend by clicking on values.


Here is an XML snippet from my SVG diagram that shows the format of a Nimbits XML Node it is all generated by the drawing program, it's just a matter of adding the action and point properties.


<text action="idle,value" point="PH"  y="187.78877" x="-119.55978" id="tspan3034-1" id="tspan3198" /></text>





Please visit www.nimbits.com to learn more.


Wednesday, May 11, 2011

The Nimbits Annotated Timeline chart


The Nimbits Server UI uses the Google Visualization API to display data in the main portal. You can use the Annotated timeline to slice and dice your data by selecting time ranges and dragging other points into it. 

What's interesting about Nimbits and time ranges, is that it uses one logical algorithm to attempt to understand what time range you are requesting. This algorithm is used anywhere a timespan is being requested. So if you are pulling data from the chart API web service, downloading a series, or entering a time range into the chart, you can use the same date formats and nimbits will understand.


Some examples of timespan requests are a unix epoch time in milliseconds or seconds, a text date, or even an abstract date like "now" or "*" for the current time, and -1d for one day in the past from the end time. 

Nimbits interprets all date requests as a start and end time. Start times can be  relative to end times when using an abstract like -7d (the last 7 days) as above. 

When you double click on a data point in Nimbits, the portal opens with a point control window with the current value of the data point in it and the chart control.  In order to show useful information right away, Nimbits gets the date of the last recorded value for that point, and the last 100 recorded values from that date.  


Chart Features
  • Enter a start and end timestamp boxes integerate into Nimbit's date interpretation algorithm. Many combinations of date entiry is possible, including a Unix time, * or now in the end date for the current time, -1d -5h etc in the start time for a relative range to the start time. Read more here.
  • Annotations. Any recorded value with a text value in the note field will show as an annotation. Shown here as "Hello" 
  • Drag and drop. Drag your points or those from your connections into the chart to compare them on the same display.
  • Click the unpin button to move the chart, or maximize to full screen.
  • The zoom selection will zoom into the current data.  Note that you must select a time range and click the refresh button to first download the data you want, then use the zoom options to zoom into that data. The zoom options do not download data outside the selected time range. 
  • MinY and MaxY selections for changing the Y axis. 
  • Zoom slider at the bottom for zooming in.

Tuesday, April 19, 2011

Using Nimbits to log unit test performance metrics

Alternate title: Application, heal thy self.

This article will show you how to add a couple of lines of code to your Unit Tests to change them from simply telling you if your code passes or fails, to all singing all dancing integrations tests that log execution times, enforce SLAs, and even warn you if the test isn't running.

The reason for the alternate title above is that as I was writing the Nimbits source code, I realized that there was an incredible useful aspect to the Nimbits server: I could actually use the server to record how long my tests took to run, and see the history of this performance metric. Nimbits was actually using itself to improve performance. So I could see if changes I made to the code was improving performance or not.

Take, for example this test, which writes 10 random values to a data point called b1.


@Test
public void testLoadRandom() throws IOException {
Random r = new Random();
for (int i = 0; i < 10; i++) { Value v = nimbitsSDK.recordValue("b1", (r.nextDouble() * 100), new Date()); 

 Assert.assertNotNull(v); 
 Assert.assertTrue(v.getKey() > 0);
}
}


Good, this test will run once an hour on my Hudson server and fails if any of the 10 values do not get recorded into the b1 data point.   This of course, can be any test you are running on any project. In my example here i'm using Nimbits to test and tune Nimbits which I know is a little creepy.

You can add a couple of lines of code to this test to time how long it takes to write these 10 values - that is to make 10 round trips to Nimbits, waiting for each response.  We're then going to record how long the test took to run and save that time in another time series Data Point on a Nimbits Server. This way, we can better understand how our system is performing. What's more, there are many built in features we can take advantage of. Check this out.

Here is the same unit test with references to the Nimbits SDK, and some code to time the test. We then write the result to the data point



@Test
public void testLoadRandom() throws IOException {
Date start = new Date();
Random r = new Random();
for (int i = 0; i < 10; i++) { Value v = Common.client().recordValue("b1", (r.nextDouble() * 100), new Date()); Assert.assertNotNull(v); Assert.assertTrue(v.getKey() > 0);
}
Date end = new Date();
long time = (end.getTime() - start.getTime()) / 1000;
nimbitsSDK.recordValue("Time to record 10 values to b1", time, new Date());
}


I have a data point setup called "Time to record 10 values to b1" that holds the time each test took to run. Let's take a look at some of the value Nimbits brings to the table besides being able to see how our system performs.

Alerts:  A data point can be configured to send you an email, text message, twitter etc if a value exceeds a threshold. 


Why yes, I'd like to know if the time is over 20 seconds or below 0, both indicating something was wrong.


Idle Alarms: Nimbits can alert you if a data point is idle when it should have received a value.


Sure, the build can fail on an error, but what if the whole build server or Hudson is down? Nimbits can alert you if a point has not received a new value. Yes, i'd like to know if my hourly tests are not generating data.





See the impact of your code changes:


The test I wrote above has been humming away for several weeks. Here is a snapshot of the resulting times that spanned the upgrade of the Test Nimbits Server Instance to Nimbits 3.1, being released this week on www.nimbits.com.  Imagine going into your boss's office with a chart like this showing real improvement as a result of a code change. (Yes, Nimbits Server 3.1 is way faster)





Obvious improvement with the upgrade, one of the many uses of a Nimbits Server. You can use the live nimbits server, download your own, and get the Nimbits SDK for java and .net on www.nimbits.com

Sunday, March 6, 2011

Chat with your Nimbits Data Points over IM

You can chat with your Nimbits data over any XMPP compatible instant messenger client, And there are a lot of them! Chances are, the IM client on your phone, desktop or browser can talk to a Nimbits Server.

Using IM, you can create points, get current values, record values and list points just like a command prompt at a terminal.

To get started, log into your Nimbits Server - if you are using the public server, just log into app.nimbits.com. Click the "Request IM Invite" button on the top menu, and then check your IM client. You should get a request authorize the Nimbits Server to send you instant messages.


From there, you can send an IM to Nimbits with the following commands


  • ?  - Help - list these commands
  • c foo - create a new data point called foo
  • foo=42 - set the current value of foo to 42
  • foo=bar - set the current text value of foo to bar
  • ls - list all points and categories
  • foo?  get the current value of foo.


Your chat session may look like this:



After you authorise Nimbits to send you instant messages, you can also configure a data point to send you an alarm using IM, or just send you an IM any time a point gets a new value.

To set this up,  log into Nimbits and double click on a point, you can select the Point Properties menu to configure that data point.  You can then check off the box to receive an instant message from Nimbits any time that point records a new value or, under the alarms tab, get an IM if the value is above or below a limit you set.




Then, when a new value comes into that data point from anywhere, you'll get an IM from Nimbits.