Massive Image Organizer

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Massive Image Organizer

Post by Skara »

This may be a bit simple, but I'd really like to get some feedback before I start this project. I've been pondering this for a while, and the last time I tried to start it, it got messy real fast.

I'd like to design a rather simple image organizer, similar to iPhoto, Aperture, Lightroom, etc. (without the image adjustment features--I just want to organize them)
Reason being, (as a photographer) I'd like to be able to access my files from anywhere and this seems to be the best (if not easiest) way to do it.

That said, I need something that will first:
- track 5000-20000 image files (yeah, I know... I overshoot)
- add tags to each image (and sort by combos of those tags)

I'm sure I'll add more stuff as it develops, but that's where I want to start.

So. Obviously, if I were only doing this for a couple hundred files, I could be done in 30 minutes. :roll:
LAST time I tried this though, it gave out and stopped working very well around 3000 images.

Therefore, questions:
- Is there some different way I should store this data given how many entries I'll have?
- Is there anything special I should know about pagination other than "LIMIT 10,20" ?

Hopefully these aren't dumb questions. I just don't want to start on it and then have it freak out once I add so many images...

Thanks!!
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: Massive Image Organizer

Post by Christopher »

Skara wrote:So. Obviously, if I were only doing this for a couple hundred files, I could be done in 30 minutes. :roll:
LAST time I tried this though, it gave out and stopped working very well around 3000 images.
What part if it gave out and stopped working very well around 3000 images?
(#10850)
User avatar
inghamn
Forum Contributor
Posts: 174
Joined: Mon Apr 16, 2007 10:33 am
Location: Bloomington, IN, USA

Re: Massive Image Organizer

Post by inghamn »

This sounds like an exact description of my own photo database project. Over the years, I've finally gotten it to a point I consider elegant. You're welcome to take a look at my solution, either for ideas, or to even try out and use.

http://sourceforge.net/projects/photodatabase/

I completely understand about the volume of photos being a problem. After a few thousand photos, organizing them into albums stops making sense. You don't want to find photos from just one album at a time. You're really wanting to have a big sea of photos that you can pull sets of photos that you want to look at. Not just tagging, but other meta-information as well: year, roll, location, etc.

The uploading system is also a big consideration. If you're talking about thousands of photos, you don't want to upload files one photo at a time. Unfortunately, multi-file upload on the web is a bit of a tricky problem. You're forced into looking at a Java or Flash solution for the multi-file uploader. Maybe even a service-based approach, where your users use a custom client. (Which is ultimately what I did)

The only other recommendation I can suggest is not to have PHP open and stream the photos. The photo files (thumbnails and all) need to be accessible by apache. Let the web server handle streaming the files to the user, it what it's good at. I ended up doing a custom 404 handler to generate and cache thumbnails, which then redirects to let apache try to server the request photo again. But definitely keep the photo files on the hard drive. Don't even think of storing blobs in the database.
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: Massive Image Organizer

Post by Skara »

arborint wrote:What part if it gave out and stopped working very well around 3000 images?
inghamn wrote:I completely understand about the volume of photos being a problem. After a few thousand photos, organizing them into albums stops making sense. You don't want to find photos from just one album at a time. You're really wanting to have a big sea of photos that you can pull sets of photos that you want to look at. Not just tagging, but other meta-information as well: year, roll, location, etc.
this ^ for one.
That and I goofed something somewhere because it put a massive draw on my database and kinda freaked it out. It kinda just told me "too much stuff" and didn't want to load anymore. So rather than fix it, I gave up.

Your photodatabase project looks interesting. I'll give a longer look at it later when I have time. The 404 handler sounds really weird, though. Definitely have to get a good look at how that works.
As for uploading them, this will run on a computer the images are already on. Yay me.
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: Massive Image Organizer

Post by Benjamin »

Skara wrote:it put a massive draw on my database and kinda freaked it out. It kinda just told me "too much stuff" and didn't want to load anymore.
Can you post the code please.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Massive Image Organizer

Post by josh »

Were you storing binary data in the table? Where you selecting joining or enforcing RI on an un-indexed column? On marinas.com they have 1.5TB of images ( 130k ) all in one big directory, the database tells us which photos belong together, to a single location, we just group on those field(s), since they are indexed there is no performance hits selecting from the 130k row table. I don't think the fact the data is about photos should have any performance hits as long as you store the binary data on disk or in another database.
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: Massive Image Organizer

Post by Skara »

Hmph.. well, ok. No, I can't display the code, I lost it a long time ago. This isn't something I've messed with for a while. I was really just looking for suggestions, etc, before I started. :mrgreen:
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: Massive Image Organizer

Post by Christopher »

Hey Josh, what is that style of organizing data where you have everything in one table with a metadata columns? I think it uses tuples? What's it called?
(#10850)
User avatar
allspiritseve
DevNet Resident
Posts: 1174
Joined: Thu Mar 06, 2008 8:23 am
Location: Ann Arbor, MI (USA)

Re: Massive Image Organizer

Post by allspiritseve »

arborint wrote:Hey Josh, what is that style of organizing data where you have everything in one table with a metadata columns? I think it uses tuples? What's it called?
Are you thinking of EAV? Entity-Attribute-Value?
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: Massive Image Organizer

Post by Benjamin »

Skara, tell us where you have been and we will tell you where to go. It sounds like you either wrote some very bad code or you were storing your files in the database. Don't be embarrassed. We are here to help.

Maybe you were lost the last time and gave up, but it could have been something as simple as adding an index to a field in a database. Things aren't always what they seem. So show us some love, give us your worst code, and we will have fun making it better, not picking on you. That's why we are here buddy. Seriously...
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: Massive Image Organizer

Post by Christopher »

allspiritseve wrote:Are you thinking of EAV? Entity-Attribute-Value?
Yeah ... thanks. :)
(#10850)
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: Massive Image Organizer

Post by Skara »

I really don't have the code anymore. I've switched computers since then and I lost a lot of things I had written at one point. I'm sure it was bad code, though. :crazy:

Looking at EAV (haven't heard of it before), would this be a good way to tag images? I had it in my head to simply comma separate them within one row entry, such as
"id", "filename", "tag1, tag2, tag3", "moredata"
but would..
"id", "tag1"
"id", "tag2"
"id", "tag3"
work better then? It would be easier to pull data, I think, but I could almost as easily search the tag-string.
(Maybe that's not quite EAV since the attribute is assumed tag, but it's close, I think.)
User avatar
allspiritseve
DevNet Resident
Posts: 1174
Joined: Thu Mar 06, 2008 8:23 am
Location: Ann Arbor, MI (USA)

Re: Massive Image Organizer

Post by allspiritseve »

Skara wrote:I had it in my head to simply comma separate them within one row entry, such as "id", "filename", "tag1, tag2, tag3", "moredata"
I would suggest a table for images, a table for tags, and an association table between them. This is called a "many to many" association and will allow you the most flexibility.
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: Massive Image Organizer

Post by Christopher »

From Wikipedia, EAV is useful "in circumstances where the number of attributes (properties, parameters) that can be used to describe a thing (an "entity" or "object") is potentially very vast, but the number that will actually apply to a given entity is relatively modest."

I don't think that the problem you are trying to solve? I think you have lots of image -- all with the same attributes. You just want to be able to associate any number of tags with an image, and select images using a set of tags.
(#10850)
User avatar
Skara
Forum Regular
Posts: 703
Joined: Sat Mar 12, 2005 7:13 pm
Location: US

Re: Massive Image Organizer

Post by Skara »

Ok, what do you mean by an association table between them? As in..

image: id, info, info, info
tag: id, name
assoc: imageid, tagid

?? Why not just have the two tables?
Post Reply