Your biggest load and/or database
Posted: Wed Nov 03, 2010 8:49 am
Just curious as to what the largest load (site access and/or size of database) you deal with?
When you are used to smaller sites, optimizing isn't such a big concern, over the past two years, I have had the fun of two projects that forced me to learn to handle things differently.
1. was a database with a few thousand products, but also merged with many other databases, which when you went to a page, produced a 10-20 second page load. This taught me the use of temp tables to get the main list of what matched the "WHERE" clause, setting up indexes on that, and then doing a join to the others (these were databases synced over nightly from the client's POS software, and hindsight (I got put into this project in the middle), I would have built tables to build the databases better on our end instead of just rebuilding it the way it came in).
2. We have a site, that consistently each month get 30,000+ unique visitors, and 80,000+ total page loads. The required that just about everything in the site that you can link to be in a full drop down navigation, plus the site consisted of news, events, products (with many attributes tied to their databases, again), and applications, comments, and several places where that data needs randomized when displayed. The core site system we use utilizes smarty, however on this scale, combined with the fact they are in there every day changing/adding things, made it real fun. I ended having to write my own caching routine. Getting it to cache wasn't so bad, however dealing with trying to avoid what we had with using smarty, (any change in the Admin side blew away all cache) and switching it so there were triggers on cached items so say you edit a product, only the parts of the product you edited will delete caches matching was real fun.
So I know in my world, those are two of my bigger challenges, but I know for others you might be like "ha! few thousand, try few million". For those that have dealt with larger projects, what are some recommendations/considerations can you share for that compared to working on "smaller" sites?
-Greg
When you are used to smaller sites, optimizing isn't such a big concern, over the past two years, I have had the fun of two projects that forced me to learn to handle things differently.
1. was a database with a few thousand products, but also merged with many other databases, which when you went to a page, produced a 10-20 second page load. This taught me the use of temp tables to get the main list of what matched the "WHERE" clause, setting up indexes on that, and then doing a join to the others (these were databases synced over nightly from the client's POS software, and hindsight (I got put into this project in the middle), I would have built tables to build the databases better on our end instead of just rebuilding it the way it came in).
2. We have a site, that consistently each month get 30,000+ unique visitors, and 80,000+ total page loads. The required that just about everything in the site that you can link to be in a full drop down navigation, plus the site consisted of news, events, products (with many attributes tied to their databases, again), and applications, comments, and several places where that data needs randomized when displayed. The core site system we use utilizes smarty, however on this scale, combined with the fact they are in there every day changing/adding things, made it real fun. I ended having to write my own caching routine. Getting it to cache wasn't so bad, however dealing with trying to avoid what we had with using smarty, (any change in the Admin side blew away all cache) and switching it so there were triggers on cached items so say you edit a product, only the parts of the product you edited will delete caches matching was real fun.
So I know in my world, those are two of my bigger challenges, but I know for others you might be like "ha! few thousand, try few million". For those that have dealt with larger projects, what are some recommendations/considerations can you share for that compared to working on "smaller" sites?
-Greg