Friday, October 30, 2009

Introduction to Nimbits Data Historian

The best example I like to give that helps explain why we developed Nimibits is:

Say you want to record a reading from a probe thermometer that read a temperature every second. How would you do it? An excel spreadsheet? SQL Server? Flat file? Well, that's 86,400 new records in your database or spreadsheet every day - 31,556,926 a year. Want more than one? Want to record a humidity level or anything else. You start to get a real performance and volume problem.

A data historian solves this basic need. Advanced compression algorithms store this data more efficiently than a table based system. Some other basic compression can take place as well, such as only recording changes in the data - so if your temperature stays at 80 degrees for an hour, that's really only one record - even though your probe is feeding you that same temp over and over.

It then provides a means to ask it: What was the last reading? What was the average over the last 8 hours? This time last year? Record high ever?

On top of that, let's say you'd like to get an email if your temperature goes over 80 degrees or how about automatically plugging your temperature reading into a calculation for relative humidity and storing the result of that calculation in another data point? Then, let's see the average value over the last year.

One of my favorite examples is the home brewer who connects his carboy full of beer to a data historian to log the flow of C02 coming out, the PH, Specific Gravity and temperature over time. Not only to automatically calculate the % alcohol but to permanently store this information in the database to compare to future brews.

Another is the system administrator who reads all of his servers memory/CPU utilization every second into data points. They always know what the current memory usage is, but what about last night at 3am? Can he get paged if the number is ever 100%?

A data historian solves these problems.

Now, we did not invent the concept of a data historian. In fact, I spent 10 years in the chemical industry working with historians that logged changes in chemical plant components - the ppm of a chemical in a vat, the vibrations of a pump - all feeding calculations to chemical engineers.

These systems cost 100's of thousands of $ and I always wondered what it would be like to hook my aquarium up to one and monitor my temp and PH from my IPhone.

I believe Nimbits is a brand new type of data historian. First, all of the data is fed directly into a web service and stored on Google's Cloud Computing Environment. This means virtually limitless storage and scalability. Further, since your data is already on the internet, sharing that data with other interested parties or even posting on social sites like Facebook and Twitter becomes a no-brainer.

Once again, imagine two lab techs on either side of the world with their changing data feeding nimbits data points on their accounts. They can both share points with each other and view real time changes in one shared spreadsheet.

Nimbits provides SDK's and Web Services so Software Engineers can plug into it any time their software needs to store a changing timer series value. Another one of my favorite uses of Nimbits is to store the time it takes for a long running function to complete. Drop the execution time in milliseconds into a Nimbits Data Point and at any time, view averages or changes to help optimize your program. Perhaps see how enhancements improved your performance. "Hey boss, that memory purchase improved performance by 300% - and here's the Nimbits data to prove it"

Any values that change over time can be stored and retrieved in the Nimbits System.

In my Blog, you'll see how to pull the current values of your data points into spreadsheets and graphical representations of your systems. View your data real time on mobile devices, and even tie into Wolfram Alpha which gives you an incredibly powerful tool to analyze your data using predictive algorithms and mathematics I will never understand.

There's so much more....but i hope this gives you enough insight to get as excited about Nimbits. Nimbits is currently in Beta mode and is due to release in early 2010. Check out for more updates!

No comments:

Post a Comment