Tuesday, November 17, 2009

A quick view of nimbits

Nimbits is a free, social data logging service. It provides web services to feed and store time series data (such as a changing temperature) into "Data Points" online. As your data is fed up into Nimbits you'll be able to use its many online data processing services, perform calculations, and receive alerts. You can also pull your data down into any connected system. Today, as values are recorded into Nimbits users see their changes in Spreadsheets, Visio Diagrams, on Facebook, IM, or using our free windows desktop interface. Software developers also use our free and open source SDK to write their own interfaces to the Nimbits Engine. There is even an app on the Android Phone Marketplace to view and enter data.

We're adding to Nimbits every day. Please consider following this blog, becoming a fan on Facebook, or joining our discussion forum so we can keep you up to date on changes to the system.

Here is a sample screen showing the main view of nimbits. It shows a category i made on my account called "Aquarium" and a Data point called TempF. I'm feeding a temperature reading from a temp probe connected to USB port on my computer.

I show how to do this on this blog posting. Using .Net and A windows Program.

In Another Blog Posting here, I do the same thing using Java and Linux.

When your data is fed into a Data Point hosted on the cloud, you can monitor your changes using the web portal, our Android Phone Interface, Spreadsheets, Visio Diagrams, our Desktop Client or any software you develop yourself using our SDK or the REST Web Services.

Here, a google visualization API time series chart just like a google financial page! You can zoom and analyse your data just as easily.  You can also set GPS Coordinates for data points and see your changes on Google earth.






The Blue line shows the temperature in degrees F being stored in a data point called TempF. I'm also performing a calculation on TempF each time it's written to.

After creating a new point called TempC we then click on the TempF point in the navigation tree and set a calculation. The Calculation is executed each time the Trigger (TempF) is written to. The formula is executed using the X Point (TempF again - usually the trigger is also the x variable) and the result is fed into the target point (TempC)









I also want an alarm on TempF. This will send me an email from the Nimbits system if a values is ever written that is greater than or equal to 90 degrees.



Lastly, lets set our properties on TempF.


Data compression means only new values written outside this value will actually be recored. I'm writing to nimbits every second so with compression of .5 - repeated writes of 80.0 will be ignored. As will 79.6 or 80.2 etc. 80.6 will be recorded.


Keep your data clean by setting experiation dates - if you don't care what a Temp was last month, then set data to be permenatly deleted if it's older than 30 days like i do here.


If you want to share your data point with other people or applications without giving out your SOA Key, make it public. I do here, so anyone can read this value via this URL:


http://app.nimbits.com/nimbits/Service/value?pointname=TempF&owner=bsautner@gmail.com

(If you do click on this link, take a look at the resulting page source, you should just see one value returned, no XML)

Just the point name and owner email is enough if the point is public. Anyone can get this value to read into an excel spreadsheet, google spreadsheet or whatever. Click the link above - provided my Temp Probe is running you should see the current value.

You can also see here the options to have new values posted to your Facebook Wall or to send you an Instant Message.



Thanks again for using nimbits!

Friday, October 30, 2009

Introduction to Nimbits Data Historian


The best example I like to give that helps explain why we developed Nimibits is:


Say you want to record a reading from a probe thermometer that read a temperature every second. How would you do it? An excel spreadsheet? SQL Server? Flat file? Well, that's 86,400 new records in your database or spreadsheet every day - 31,556,926 a year. Want more than one? Want to record a humidity level or anything else. You start to get a real performance and volume problem.


A data historian solves this basic need. Advanced compression algorithms store this data more efficiently than a table based system. Some other basic compression can take place as well, such as only recording changes in the data - so if your temperature stays at 80 degrees for an hour, that's really only one record - even though your probe is feeding you that same temp over and over.


It then provides a means to ask it: What was the last reading? What was the average over the last 8 hours? This time last year? Record high ever?


On top of that, let's say you'd like to get an email if your temperature goes over 80 degrees or how about automatically plugging your temperature reading into a calculation for relative humidity and storing the result of that calculation in another data point? Then, let's see the average value over the last year.


One of my favorite examples is the home brewer who connects his carboy full of beer to a data historian to log the flow of C02 coming out, the PH, Specific Gravity and temperature over time. Not only to automatically calculate the % alcohol but to permanently store this information in the database to compare to future brews.


Another is the system administrator who reads all of his servers memory/CPU utilization every second into data points. They always know what the current memory usage is, but what about last night at 3am? Can he get paged if the number is ever 100%?


A data historian solves these problems.


Now, we did not invent the concept of a data historian. In fact, I spent 10 years in the chemical industry working with historians that logged changes in chemical plant components - the ppm of a chemical in a vat, the vibrations of a pump - all feeding calculations to chemical engineers.


These systems cost 100's of thousands of $ and I always wondered what it would be like to hook my aquarium up to one and monitor my temp and PH from my IPhone.


I believe Nimbits is a brand new type of data historian. First, all of the data is fed directly into a web service and stored on Google's Cloud Computing Environment. This means virtually limitless storage and scalability. Further, since your data is already on the internet, sharing that data with other interested parties or even posting on social sites like Facebook and Twitter becomes a no-brainer.


Once again, imagine two lab techs on either side of the world with their changing data feeding nimbits data points on their accounts. They can both share points with each other and view real time changes in one shared spreadsheet.


Nimbits provides SDK's and Web Services so Software Engineers can plug into it any time their software needs to store a changing timer series value. Another one of my favorite uses of Nimbits is to store the time it takes for a long running function to complete. Drop the execution time in milliseconds into a Nimbits Data Point and at any time, view averages or changes to help optimize your program. Perhaps see how enhancements improved your performance. "Hey boss, that memory purchase improved performance by 300% - and here's the Nimbits data to prove it"


Any values that change over time can be stored and retrieved in the Nimbits System.


In my Blog, you'll see how to pull the current values of your data points into spreadsheets and graphical representations of your systems. View your data real time on mobile devices, and even tie into Wolfram Alpha which gives you an incredibly powerful tool to analyze your data using predictive algorithms and mathematics I will never understand.


There's so much more....but i hope this gives you enough insight to get as excited about Nimbits. Nimbits is currently in Beta mode and is due to release in early 2010. Check out www.nimbits.com for more updates!

Friday, October 23, 2009

A basic .net Client for Reading and Writing Nimbits Data

One of the most common tasks with Nimbits is to write a value to a data point, and request the most recent value. So much so, there is an entire web service dedicated to this task.

You can call these services from any platform or programing language. This example is for C# developers. If you want to do the same thing in JAVA, check out this post.

A developer can read and write data to nimbits by calling our REST based web services. In order to do this, you need to log into app.nimbits.com and create your data points in advance. Then get your SOA key. Learn more about SOA keys here in my blog.

With a SOA key, you can access your Nimbits data from other applications you develop without being logged into Nimbits with your Google Account.

Here is a C# .net class you can add to any project to read and write data to and from your data points. It's just two functions, getCurrentValue and recordValue



using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;

namespace MyClient
{


class Program
{
private static String soaKey = "drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC";

private static String serviceURL = "http://app.nimbits.com/nimbits/Service/value";

private static void recordValue(double value, DateTime timestamp, String pointName)
{

DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
long ms = (long)(DateTime.UtcNow - epoch).TotalMilliseconds;
long result = ms;

HttpWebRequest request = null;
Stream stream = null;
String postData = "key=" + soaKey + "&pointname=" + pointName + "&timestamp=" + result + "&value=" + value;
StreamReader streamReader = null;
HttpWebResponse response = null;
request = (HttpWebRequest)WebRequest.Create(serviceURL);
request.Method = "POST";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream newStream = request.GetRequestStream();
// Send the data.
newStream.Write(data, 0, data.Length);
newStream.Close();
response = (HttpWebResponse)request.GetResponse();
stream = response.GetResponseStream();
streamReader = new StreamReader(stream);
string feedData = streamReader.ReadToEnd();
}


private static Double getCurrentValue(String pointName)
{


String postData = "key=" + soaKey + "&pointname=" + pointName;

HttpWebRequest request = null;
HttpWebResponse response = null;
Stream stream = null;
StreamReader streamReader = null;
request = (HttpWebRequest)WebRequest.Create(serviceURL + "?" + postData);
request.Method = "GET";
request.ContentLength = 0;
response = (HttpWebResponse)request.GetResponse();
stream = response.GetResponseStream();
streamReader = new StreamReader(stream);
string feedData = streamReader.ReadToEnd();
response.Close();
stream.Dispose();
streamReader.Dispose();
return Convert.ToDouble(feedData);


}
}
}

A basic Java Client for Reading and Writing Nimbits Data

Update 7/2010 - this code is out of date now because we're no longer supporting the SOA key. Check out my more recent postings that have the Java code you need to authenticate directly to Nimbits with your Google Account.



One of the most common tasks with Nimbits is to write a value to a data point, and request the most recent value. So much so, there is an entire web service dedicated to this task.

You can call these services from any platform or programing language. This example is for Java developers. If you want to do the same thing in C# check out this post. I would love to see someone post a variation in another language like python.

A developer can read and write data to nimbits by calling our REST based web services. In order to do this, you need to log into app.nimbits.com and create your data points in advance. Then get your SOA key. Learn more about SOA keys here in my blog.

With a SOA key, you can access your Nimbits data from other applications you develop without being logged into Nimbits with your Google Account.

Here is a java class you can add to any project to read and write data to and from your data points. It's just two functions, getCurrentValue and recordValue

the magic happens when you go to nimbits and set up calculations and alerts to trigger every time you write to a data point. Other users around the world who you share your data with will see your changes in real time.


import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import java.util.Date;


public class NimbitsClient {

private final static String soaKey = "drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC";
private final static String serviceURL = "http://app.nimbits.com/nimbits/Service/value";

public static void recordValue(double value, Date timestamp, String pointName) throws IOException
{
String data = URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(soaKey, "UTF-8");
data += "&" + URLEncoder.encode("pointname", "UTF-8") + "=" + URLEncoder.encode("TempF", "UTF-8");
data += "&" + URLEncoder.encode("value", "UTF-8") + "=" + value;
data += "&" + URLEncoder.encode("timestamp", "UTF-8") + "=" + timestamp.getTime();
// Send data
URL url = new URL(serviceURL);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
wr.close();
rd.close();
}

public static Double getCurrentValue(String pointName) throws IOException
{
Double retVal;

String data = URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(soaKey, "UTF-8");
data += "&" + URLEncoder.encode("pointname", "UTF-8") + "=" + URLEncoder.encode(pointName, "UTF-8");
URL url = new URL(serviceURL + "?" + data);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
String result ="";

while ((line = rd.readLine()) != null) {
result +=line;
}
rd.close();
retVal = Double.valueOf(result);

return retVal;

}


}

Understanding the Nimbits SOA Key

Please note we are phasing out SOA Key's in favor of OAuth. Please contract me if you have any questions - bsautner@ google's free mail service.

This blog posting will help you understand how to access your private Nimbits historical data using a SOA key instead of your google account. Please note you can always make your points public so anyone can see the data and make it much easier to access. See my other posts on sharing your data if that interests you.

When you log into Nimbits with your Google Account, you have access to all of your historical data and applications. However, we need to provide you with a way to tell the nimbits system who you are when your using or developing an application that dosn't have the ability to provide Google account information. The solution is your SOA key.

When you log into app.nimbits.com you can create a SOA key which is a random 64 character string like this:

drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC

just click the Manage SOA Key Menu and you'll be prompted to either rest your key to a new one or to have your current key emailed to you. Your key will be emailed to the Google account you're logged in with.

You should keep this key as safe as you keep your google account password since it can be used to read and write values to your data points. In order to do more advanced things like delete data or create points, you need to log into the application normally.

If you feel you need a new key, just use the menu to generate a new one. The old key will no longer work. Keep this in mind since any apps or spreadsheets you have will need to have their key updated.

Using a SOA key extends your ability to access your Nimbits data from apps outside the Google environment. Read more in this blog about making your data points public so they can be accessed without a key by anyone anywhere (for those of you who don't care who knows what the current humidity level of your humidor is but still want to give your facebook friends hourly updated :-)

Read more on this blog about using your SOA Key in a .net or java application to use nimbit.com as a data source.


Saturday, September 26, 2009

Connecting a SensaTronics Model F temperature monitor to Nimbits

Update: A more recent article has a more generic Java Class for reading and writing to Nimbits or any App Engine Project that uses REST web services. Check it our here or enjoy the more rugged code below.



In this post, I'm going to show you how to use a temperature probe connected to your computer and feed that temperature into Nimbits so you can access that reading anywhere in the world. Each time the Fahrenheit temperature is read into the system, we'll automatically convert it to Celsius using the formula engine and set up an alert to email us automatically if the temp goes beyond a certain range.

Before you attempt this, please understand how nimbits SOA keys work. You'll need to create a SOA key by logging into app.nimbits.com. Read more about SOA keys here.


I'm going to cover how this is done in Java using Eclipse on an Ubuntu workstation. If you'd like to accomplish the exact same result using a Windows bases PC and .Net see my other post on this topic.

It's a little more difficult to get things going on your desktop using linux, but IMO it's a much more satisfying result!

This example can be uses as a general purpose tutorial for connecting most serial or USB devices to nimbits. If it produces a number value that changes over time, then it can be done.

Hardware:

As i said, this tutorial can be modified to work with most serial devices. For this post, I'm using a SensaTronics Model F temp monitor available here

I like this monitor becuase it supports multiple probes and they also sell a 50' probe which frees you from having to put the workstation next to whatever you're monitoring.

Since the PC i'm using dosn't have a serial port, i'm using Belkin USB to Serial Adapter which works just fine on Ubuntu without any driver futsing.

Software
I'm not going to go into any great detail on setting up the development environment. One should have Eclipse up and running and follow the instructions for installing RXTX on linux - Thankfully, you can skip to the bottom of these instructions and just do Method 3.

Step 1. Create your project
  • Launch Eclipse and select File - New Java Project
  • Name the project TempProbe
  • Click Finish
  • Right click on the TempProbe Project in Package Explorer and Select New - Class
  • Name the Class runProbe and Check to create a public static void main
  • Right click the project again and select Build Path and then Configure Build Path
  • On the Libraries tab, select to Add An external JAR and browse to where you dowloaded the RXTXcomm.jar file.
Step 2. Gather information

When you connected your Temp Probe USB adapter and new tty device was added to your system. You'll need that port information. Open a terminal and type:
ls /dev/tty*

when i connected my temp probe, i saw a new device called /dev/ttyUSB0 yours may be different.

You'll also need your nimbits.com SOA Key. Using a key allows you to feed data into nimbits from a system that's not logged into google with a google account.

When you log into nimbits you're authenticated using your google account and can access all of your information. When your not logged into google, you need to provide your personal key which is a 64 character string the application assigns to you. It's important you keep this key as private as you keep your google password since anyone who has it can access your Nimbits data via the web services (but not any other information related to your google account)

Log onto Nimbits and select Manage SOA Key at the top menu. Here, you can have your key emailed to the google account your logged in as. Be careful if you choose to get a new key. You can do this at anytime but any software you already have using the nibmits service will need to be updated.

While your logged into Nimbits, let's also create two new Data points under any catagory you like. Call the first point TempF and the next Point TempC

Click on the TempF point. You'll see an empty chart, and above it, the menu for the many ways you can work with this point. Select the point name from the menu to edit it's properties.

Select the calculations tab to edit the calulation. Read more about how calculations work in Nimbits. When your probe writes to the TempF point, it will trigger the calculation, use the new TempF value as the x parameter and send the result into the TempC point.

So the tigger is LabTemF
Formula : (x-32)*(5/9) (converts F to C)
Target: TempC
X= TempF

Save, you should be good to go. In face, if you click test you should get -17.777 because your curret value for the new TempF point is 0.

You'll also want to set the properties of the point. You can enter C and F for the units of measer. Also, note the compression setting. Compression means you'll only record values that are outside the compression range. So if your reading 77 degrees and your compression is set to 0.5, a reading of 77.1 will get ignored but 77.6 will be recorded. This is usefull for filtering out noise and small changes your not interested in. Let's give our temp points a compression of 0.5

Now, paste this code into your class:


public static java.sql.Date Now()
{
Calendar cal1 = Calendar.getInstance();
Date _now = new java.sql.Date((cal1.getTime()).getTime());

return _now;

}
public static void main(String[] s)
{
//System.out.println(getTemp());

while (true)
{

try {
Double v = getTemp();
System.out.println("Got Temp: " + v);

if (v > 0)
{
updatepoint(v);
}

Thread.currentThread();
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (java.lang.NumberFormatException e)
{

}

}


}
private static void updatepoint(double temp)
{
try {
//my SOA key from nimbits.com
String key = "drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC";

// Construct data
String data = URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(key, "UTF-8");
data += "&" + URLEncoder.encode("pointname", "UTF-8") + "=" + URLEncoder.encode("TempF", "UTF-8");
data += "&" + URLEncoder.encode("value", "UTF-8") + "=" + temp;
data += "&" + URLEncoder.encode("timestamp", "UTF-8") + "=" + Now().getTime();
// Send data
URL url = new URL("http://app.nimbits.com/nimbits/Service/value");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();

// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
wr.close();
rd.close();
} catch (Exception e) {
e.printStackTrace();
}

System.out.println("wrote value " + temp);


}
private static double getTemp()
{
Double TempV = 0.0;
SerialPort serialPort = null;
OutputStream mOutputToPort = null;
InputStream mInputFromPort = null;

try {
CommPortIdentifier portIdentifier = CommPortIdentifier.getPortIdentifier("/dev/ttyUSB0");

System.out.println(portIdentifier.getName());

serialPort = (SerialPort) portIdentifier.open("ListPortClass", 300);
int b = serialPort.getBaudRate();
System.out.println(Integer.toString(b));
serialPort.setSerialPortParams(9600, SerialPort.DATABITS_8, SerialPort.STOPBITS_1,
SerialPort.PARITY_NONE);
serialPort.setDTR(false);
serialPort.setDTR(true);
Thread.sleep(1000);
mOutputToPort = serialPort.getOutputStream();
mInputFromPort = serialPort.getInputStream();
String mValue = "AT\r";

System.out.println("beginning to Write . \r\n");
mOutputToPort.write(mValue.getBytes());
System.out.println("AT Command Written to Port. \r\n");
mOutputToPort.flush();
System.out.println("Waiting for Reply \r\n");
Thread.sleep(1000);
byte mBytesIn [] = new byte[20];


mInputFromPort.read(mBytesIn);
String value = new String(mBytesIn);
String[] s1 = value.split((""+(char)10));
System.out.println(s1[0].trim());
System.out.println("Response from Serial Device: ");
System.out.println(value);
TempV = Double.valueOf(s1[0].trim());



} catch (NoSuchPortException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (PortInUseException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (UnsupportedCommOperationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally
{
serialPort.setDTR(false);
try {
Thread.sleep(500);
mOutputToPort.close();
mInputFromPort.close();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Closing");

serialPort.close();
}
return TempV ;

}


Run the app. You should start to see temp readings from your probe show up on your nimbits portal.

Now you can:
  • use this temp in a google doc, or excel spreadsheet
  • get an email alert when the temp goes out of a range
  • Get powerful analytics from Wolfram Alpha like a possible formula and continuation for a sequence
  • View this on your android based phone and more!