very large arrays

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
yamobe
Forum Newbie
Posts: 13
Joined: Mon Jun 19, 2006 9:42 am

very large arrays

Post by yamobe »

Hello

I have a BIG problem!!

I was making a Game which needed very heavy processing (I didn't know that at the beggining).
The incredibly big arrays used, use server memory up to 1Gb, which is ridiculous.

I have been working on this more than 2 months, and I don't want to start all over again.


Do you think I would resolve this using an external C/C++ program on the server to make the heavy processing instead of using PHP?

Are there incredible and magic techniques to use very large arrays in php without using so much memory?

What do you think?

Cheers
User avatar
Apollo
Forum Regular
Posts: 794
Joined: Wed Apr 30, 2008 2:34 am

Re: very large arrays

Post by Apollo »

When handling data amounts of over 1GB, optimized C++ is usually TONS faster than PHP.

However, first of all, you should take a step back and reconsider your whole agorithm. Is it REALLY necessary to have over 1GB of data? Are you by any chance storing temporary stuff that isn't really necessary or can be calculated on the fly? Isn't there some kind of smarter approach?

If you're dealing with all kinds of in-between states or implicit results, you could also take a look at a function programming approach (e.g. Haskell) which does lazy evaluation.
User avatar
tr0gd0rr
Forum Contributor
Posts: 305
Joined: Thu May 11, 2006 8:58 pm
Location: Utah, USA

Re: very large arrays

Post by tr0gd0rr »

Is it data that could go into a MySQL table?
yamobe
Forum Newbie
Posts: 13
Joined: Mon Jun 19, 2006 9:42 am

Re: very large arrays

Post by yamobe »

I can unset some unused data.. that's true... that would reduce to something around 300Mb perhaps... which is still very high... I don't think I can do better than that.
The algorithm is a heavy transport model that was done by a mathematician, so I can't do anything about that... It is what it is.

Instead of using C++ I could use html5 web workers or something like that and start from scratch ... but I am scared of debugging issues since I have never faced something like this before... Do you know other tools I could use?

function programming approach was my first approach and it was SLOW AS HELL so I changed all the code... all the data goes to a mysql table, but making sql requests each time I need something is heavier and slower... I was my approach also at some point.

I have 3 possible solutions as I see
- Using c++ in the server to make the heavy lifting
- Using some javascript/HTML5 thing to make the heavy lifting on the web browser and then throw everything into mysql
- Optimize my php code to kill unused data when no longer needed

2 first options force me to start from scratch!!!
Any other thought?

Thanks :)
Post Reply