How to chart large amounts of data?

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

User avatar
flying_circus
Forum Regular
Posts: 732
Joined: Wed Mar 05, 2008 10:23 pm
Location: Sunriver, OR

How to chart large amounts of data?

Post by flying_circus »

I've been super-nerding lately and built an arduino based fermentation monitor. In the end, this device will be a climate controller for my fermentation chamber and provide statistics about the fermentation of the beers that I brew. Right now, I am trying to figure out the logistics of graphing the large amount of data it produces.

How it works: At a specified interval (currently 60 seconds) this device will take a reading from the sensors attached, and then send a post request to my webserver, which will log the sensor readings in a mysql database.

The Problem: I want it to be able to run for atleast a month, minimum, and collect data for the duration. That means I've collected almost 45,000 sensor readings for a 31 day month.

I have been toying with some php based chart software, I've had the best luck with pChart (ver 1) so far. However, based on the 200 test entries in my database, pChart seems kludgy at best. It seems more suited for a dozen or so data points.

The best type of system I have seen is what google is using for their stock system, example: http://www.google.com/finance?client=ob&q=NASDAQ:GOOG#

Does anyone have any experience in this area, or recommendations? I'd rather not reinvent the wheel, as this is a pet project.
User avatar
egg82
Forum Contributor
Posts: 156
Joined: Sat Oct 01, 2011 9:29 pm
Location: Colorado, USA

Re: How to chart large amounts of data?

Post by egg82 »

you call that a pet project? Holy... I don't have the time or money for that kind of thing, how did you find both?

Anyway, in answer to your question -
It's not re-inventing the wheel. In fact, compared to what you're doing it's minuscule. Are you collecting 45,000 per day or 45,000 per month?
In any case, i'm not 100% sure MySQL would be the BEST way to go, but its SELECT, WHERE, and SORT BY functionality may come in useful there. Minimizing data output is a great way for the browser (and PHP) to be able to handle data. Instead of getting a while bunch of data at once, only collect what is relevant. If many things are relevant, then use several pages for your data. If you have too many rows, just grab a few at a time. 30 is a good number.

Anyway, hope I helped
Eric!
DevNet Resident
Posts: 1146
Joined: Sun Jun 14, 2009 3:13 pm

Re: How to chart large amounts of data?

Post by Eric! »

You're talking about Google's flash charts. I think they have released an API for their visualizations using this time series type of charts.

You can put your beer data into a database then pass the info to the google API dynamically and make nice charts on the fly.
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: How to chart large amounts of data?

Post by Benjamin »

You should also put up a blog about this (if you haven't). A lot of people would be interested in viewing pictures and reading about what works, what doesn't, how it tastes etc. This is one of the projects on my list actually.
User avatar
egg82
Forum Contributor
Posts: 156
Joined: Sat Oct 01, 2011 9:29 pm
Location: Colorado, USA

Re: How to chart large amounts of data?

Post by egg82 »

didn't think about that. It's a good idea, blog about your creation.

The question still remains, however. Where are you people getting the time and money for this?
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: How to chart large amounts of data?

Post by Benjamin »

:arrow: Charge more.
User avatar
flying_circus
Forum Regular
Posts: 732
Joined: Wed Mar 05, 2008 10:23 pm
Location: Sunriver, OR

Re: How to chart large amounts of data?

Post by flying_circus »

Eric! Thanks for the tip. I briefly checked out the google api the other night, I guess I didn't dig deep enough. I spent a little more time looking it over tonight and I think it will work for what I am trying to do. I also like that it's free, that will be a big plus if I give someone else the code, let alone, set it up for my father :)

egg82, the project is actually pretty cheap and easy to get into, google arduino. The hardware for the microcontroller and ethernet shield (with microSD socket) are less than $100. If electronics interest you, I'd highly recommend you check it out, it's very beginner friendly (and that's what I consider myself). Time wise, I've got maybe a couple evenings worth of work into it? Every once in a while you gotta take a break from work and blow off a little steam, for me that's beer!

Benjamin, you should DEFINATELY get into home brewing if it interests you. The downside is that it will totally ruin "regular" beer for you. I typically do 5 gallon batches (about 48 bottles) at a time. Brew day takes about 4 hours from start to finish, then you just wait for 3 or so weeks before you drink it. Each batch runs about $35. I'd love to set up a blog, I have so many cool projects going on that I'd like to document, but I'm only about half way done writing my blog system. I built a cnc machine last spring and I really wish I had blogged about it. If you're entertaining the idea of brewing, shoot me a PM, I'll share some info you'll need to get started :D
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: How to chart large amounts of data?

Post by Benjamin »

I will definitely get in touch with you. I still need to finish my first batch of 18 solar panels though...
User avatar
flying_circus
Forum Regular
Posts: 732
Joined: Wed Mar 05, 2008 10:23 pm
Location: Sunriver, OR

Re: How to chart large amounts of data?

Post by flying_circus »

Benjamin wrote:I will definitely get in touch with you. I still need to finish my first batch of 18 solar panels though...
Share your solar panel project with us!

Here is a screen shot of the graph I ended up with. Pretty nifty for a minimal amount of effort 8)
Attachments
Fermonitor Ver 0.01 - alpha
Fermonitor Ver 0.01 - alpha
Eric!
DevNet Resident
Posts: 1146
Joined: Sun Jun 14, 2009 3:13 pm

Re: How to chart large amounts of data?

Post by Eric! »

What does that tell you about the beer?
User avatar
flying_circus
Forum Regular
Posts: 732
Joined: Wed Mar 05, 2008 10:23 pm
Location: Sunriver, OR

Re: How to chart large amounts of data?

Post by flying_circus »

The graph is just a datalogger at this point. What I haven't yet described is the climate controller.

Yeast is a living organizm and the active ingredient in beer, it is also VERY finicky. It eats sugar, farts CO2, and produces alcohol in the process. Fermentation really is nothing more than a controlled "rot".

A traditional ale yeast performs best when kept within the temperature range of about 65-68 degrees (F). The trouble is, fermentation is an exothermic activity. The peak of the fermentation process tends to happen within about 72 hours of pitching the yeast into the beer and it is not uncommon to see a dramatic temperature increase within that time period. We also have to consider our local climate, in the summer time it is pretty consistently in the 90's outside, and there is no A/C in my home, which means my home is pretty close to the same temperature. Not ideal for fermentation.

If your temperatures are out of range during the fermentation process you can kill the yeast, have the yeast become inactive, or more likely, the yeast will be sluggish and produce off flavors. If the yeast die, or grow too slowly, it gives wild bacteria (bateria naturally occuring in air) a chance to take over and infect your beer.

Keeping fermentation temperature within range is probably the second most important part of brewing beer, right behind sterilizing/sanitizing all of your equipment.

With that said, I am working on designing a fermentation chamber with 2 thermally isolated compartments. The temperature will be regulated by the device that produces the data log, effectively activating a hot or cold source based on the temperature readings.
Attachments
Fermentation Chamber Rough Sketch
Fermentation Chamber Rough Sketch
Eric!
DevNet Resident
Posts: 1146
Joined: Sun Jun 14, 2009 3:13 pm

Re: How to chart large amounts of data?

Post by Eric! »

Got it. Just a test run. Send us all a cold one when you get the process under control!
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: How to chart large amounts of data?

Post by Benjamin »

flying_circus wrote:Share your solar panel project with us!
We'll see what the market does. Right now China is dumping cells into the market at below cost because of subsidies provided by the government over there. So at this point it appears that buying completed panels is cheaper than building them. When I started the project I could build them for 33% of what the completed price would be. I certainly would never buy any from China though, so there still might be some room there if you stick with US/German components.

Anyway, building these is not for the faint of heart, there's no way to build them in a way that will last a long time without a vacuum chamber of some sort. Luckily I was able to build one of those, but the cost for that alone was $3k or so.
User avatar
egg82
Forum Contributor
Posts: 156
Joined: Sat Oct 01, 2011 9:29 pm
Location: Colorado, USA

Re: How to chart large amounts of data?

Post by egg82 »

why wouldn't you buy anything from china? They actually make some good things (besides cheap BK toys)
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: How to chart large amounts of data?

Post by Benjamin »

egg82 wrote:why wouldn't you buy anything from china? They actually make some good things (besides cheap BK toys)
We can't discuss politics here. But if you live in the U.S., you should research this issue yourself if you care about the financial health of yourself, your family, your neighbors and the country itself.

I'll just leave this here: http://www.madeinusa.org/nav.cgi?info/whybuy
Post Reply