Friday, October 30, 2009

Introduction to Nimbits Data Historian


The best example I like to give that helps explain why we developed Nimibits is:


Say you want to record a reading from a probe thermometer that read a temperature every second. How would you do it? An excel spreadsheet? SQL Server? Flat file? Well, that's 86,400 new records in your database or spreadsheet every day - 31,556,926 a year. Want more than one? Want to record a humidity level or anything else. You start to get a real performance and volume problem.


A data historian solves this basic need. Advanced compression algorithms store this data more efficiently than a table based system. Some other basic compression can take place as well, such as only recording changes in the data - so if your temperature stays at 80 degrees for an hour, that's really only one record - even though your probe is feeding you that same temp over and over.


It then provides a means to ask it: What was the last reading? What was the average over the last 8 hours? This time last year? Record high ever?


On top of that, let's say you'd like to get an email if your temperature goes over 80 degrees or how about automatically plugging your temperature reading into a calculation for relative humidity and storing the result of that calculation in another data point? Then, let's see the average value over the last year.


One of my favorite examples is the home brewer who connects his carboy full of beer to a data historian to log the flow of C02 coming out, the PH, Specific Gravity and temperature over time. Not only to automatically calculate the % alcohol but to permanently store this information in the database to compare to future brews.


Another is the system administrator who reads all of his servers memory/CPU utilization every second into data points. They always know what the current memory usage is, but what about last night at 3am? Can he get paged if the number is ever 100%?


A data historian solves these problems.


Now, we did not invent the concept of a data historian. In fact, I spent 10 years in the chemical industry working with historians that logged changes in chemical plant components - the ppm of a chemical in a vat, the vibrations of a pump - all feeding calculations to chemical engineers.


These systems cost 100's of thousands of $ and I always wondered what it would be like to hook my aquarium up to one and monitor my temp and PH from my IPhone.


I believe Nimbits is a brand new type of data historian. First, all of the data is fed directly into a web service and stored on Google's Cloud Computing Environment. This means virtually limitless storage and scalability. Further, since your data is already on the internet, sharing that data with other interested parties or even posting on social sites like Facebook and Twitter becomes a no-brainer.


Once again, imagine two lab techs on either side of the world with their changing data feeding nimbits data points on their accounts. They can both share points with each other and view real time changes in one shared spreadsheet.


Nimbits provides SDK's and Web Services so Software Engineers can plug into it any time their software needs to store a changing timer series value. Another one of my favorite uses of Nimbits is to store the time it takes for a long running function to complete. Drop the execution time in milliseconds into a Nimbits Data Point and at any time, view averages or changes to help optimize your program. Perhaps see how enhancements improved your performance. "Hey boss, that memory purchase improved performance by 300% - and here's the Nimbits data to prove it"


Any values that change over time can be stored and retrieved in the Nimbits System.


In my Blog, you'll see how to pull the current values of your data points into spreadsheets and graphical representations of your systems. View your data real time on mobile devices, and even tie into Wolfram Alpha which gives you an incredibly powerful tool to analyze your data using predictive algorithms and mathematics I will never understand.


There's so much more....but i hope this gives you enough insight to get as excited about Nimbits. Nimbits is currently in Beta mode and is due to release in early 2010. Check out www.nimbits.com for more updates!

Friday, October 23, 2009

A basic .net Client for Reading and Writing Nimbits Data

One of the most common tasks with Nimbits is to write a value to a data point, and request the most recent value. So much so, there is an entire web service dedicated to this task.

You can call these services from any platform or programing language. This example is for C# developers. If you want to do the same thing in JAVA, check out this post.

A developer can read and write data to nimbits by calling our REST based web services. In order to do this, you need to log into app.nimbits.com and create your data points in advance. Then get your SOA key. Learn more about SOA keys here in my blog.

With a SOA key, you can access your Nimbits data from other applications you develop without being logged into Nimbits with your Google Account.

Here is a C# .net class you can add to any project to read and write data to and from your data points. It's just two functions, getCurrentValue and recordValue



using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;

namespace MyClient
{


class Program
{
private static String soaKey = "drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC";

private static String serviceURL = "http://app.nimbits.com/nimbits/Service/value";

private static void recordValue(double value, DateTime timestamp, String pointName)
{

DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
long ms = (long)(DateTime.UtcNow - epoch).TotalMilliseconds;
long result = ms;

HttpWebRequest request = null;
Stream stream = null;
String postData = "key=" + soaKey + "&pointname=" + pointName + "&timestamp=" + result + "&value=" + value;
StreamReader streamReader = null;
HttpWebResponse response = null;
request = (HttpWebRequest)WebRequest.Create(serviceURL);
request.Method = "POST";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream newStream = request.GetRequestStream();
// Send the data.
newStream.Write(data, 0, data.Length);
newStream.Close();
response = (HttpWebResponse)request.GetResponse();
stream = response.GetResponseStream();
streamReader = new StreamReader(stream);
string feedData = streamReader.ReadToEnd();
}


private static Double getCurrentValue(String pointName)
{


String postData = "key=" + soaKey + "&pointname=" + pointName;

HttpWebRequest request = null;
HttpWebResponse response = null;
Stream stream = null;
StreamReader streamReader = null;
request = (HttpWebRequest)WebRequest.Create(serviceURL + "?" + postData);
request.Method = "GET";
request.ContentLength = 0;
response = (HttpWebResponse)request.GetResponse();
stream = response.GetResponseStream();
streamReader = new StreamReader(stream);
string feedData = streamReader.ReadToEnd();
response.Close();
stream.Dispose();
streamReader.Dispose();
return Convert.ToDouble(feedData);


}
}
}

A basic Java Client for Reading and Writing Nimbits Data

Update 7/2010 - this code is out of date now because we're no longer supporting the SOA key. Check out my more recent postings that have the Java code you need to authenticate directly to Nimbits with your Google Account.



One of the most common tasks with Nimbits is to write a value to a data point, and request the most recent value. So much so, there is an entire web service dedicated to this task.

You can call these services from any platform or programing language. This example is for Java developers. If you want to do the same thing in C# check out this post. I would love to see someone post a variation in another language like python.

A developer can read and write data to nimbits by calling our REST based web services. In order to do this, you need to log into app.nimbits.com and create your data points in advance. Then get your SOA key. Learn more about SOA keys here in my blog.

With a SOA key, you can access your Nimbits data from other applications you develop without being logged into Nimbits with your Google Account.

Here is a java class you can add to any project to read and write data to and from your data points. It's just two functions, getCurrentValue and recordValue

the magic happens when you go to nimbits and set up calculations and alerts to trigger every time you write to a data point. Other users around the world who you share your data with will see your changes in real time.


import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import java.util.Date;


public class NimbitsClient {

private final static String soaKey = "drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC";
private final static String serviceURL = "http://app.nimbits.com/nimbits/Service/value";

public static void recordValue(double value, Date timestamp, String pointName) throws IOException
{
String data = URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(soaKey, "UTF-8");
data += "&" + URLEncoder.encode("pointname", "UTF-8") + "=" + URLEncoder.encode("TempF", "UTF-8");
data += "&" + URLEncoder.encode("value", "UTF-8") + "=" + value;
data += "&" + URLEncoder.encode("timestamp", "UTF-8") + "=" + timestamp.getTime();
// Send data
URL url = new URL(serviceURL);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
wr.close();
rd.close();
}

public static Double getCurrentValue(String pointName) throws IOException
{
Double retVal;

String data = URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(soaKey, "UTF-8");
data += "&" + URLEncoder.encode("pointname", "UTF-8") + "=" + URLEncoder.encode(pointName, "UTF-8");
URL url = new URL(serviceURL + "?" + data);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
String result ="";

while ((line = rd.readLine()) != null) {
result +=line;
}
rd.close();
retVal = Double.valueOf(result);

return retVal;

}


}

Understanding the Nimbits SOA Key

Please note we are phasing out SOA Key's in favor of OAuth. Please contract me if you have any questions - bsautner@ google's free mail service.

This blog posting will help you understand how to access your private Nimbits historical data using a SOA key instead of your google account. Please note you can always make your points public so anyone can see the data and make it much easier to access. See my other posts on sharing your data if that interests you.

When you log into Nimbits with your Google Account, you have access to all of your historical data and applications. However, we need to provide you with a way to tell the nimbits system who you are when your using or developing an application that dosn't have the ability to provide Google account information. The solution is your SOA key.

When you log into app.nimbits.com you can create a SOA key which is a random 64 character string like this:

drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC

just click the Manage SOA Key Menu and you'll be prompted to either rest your key to a new one or to have your current key emailed to you. Your key will be emailed to the Google account you're logged in with.

You should keep this key as safe as you keep your google account password since it can be used to read and write values to your data points. In order to do more advanced things like delete data or create points, you need to log into the application normally.

If you feel you need a new key, just use the menu to generate a new one. The old key will no longer work. Keep this in mind since any apps or spreadsheets you have will need to have their key updated.

Using a SOA key extends your ability to access your Nimbits data from apps outside the Google environment. Read more in this blog about making your data points public so they can be accessed without a key by anyone anywhere (for those of you who don't care who knows what the current humidity level of your humidor is but still want to give your facebook friends hourly updated :-)

Read more on this blog about using your SOA Key in a .net or java application to use nimbit.com as a data source.