Page 1 of 1
very large arrays
Posted: Wed Jun 22, 2011 1:29 am
by yamobe
Hello
I have a BIG problem!!
I was making a Game which needed very heavy processing (I didn't know that at the beggining).
The incredibly big arrays used, use server memory up to 1Gb, which is ridiculous.
I have been working on this more than 2 months, and I don't want to start all over again.
Do you think I would resolve this using an external C/C++ program on the server to make the heavy processing instead of using PHP?
Are there incredible and magic techniques to use very large arrays in php without using so much memory?
What do you think?
Cheers
Re: very large arrays
Posted: Wed Jun 22, 2011 3:37 am
by Apollo
When handling data amounts of over 1GB, optimized C++ is usually TONS faster than PHP.
However, first of all, you should take a step back and reconsider your whole agorithm. Is it REALLY necessary to have over 1GB of data? Are you by any chance storing temporary stuff that isn't really necessary or can be calculated on the fly? Isn't there some kind of smarter approach?
If you're dealing with all kinds of in-between states or implicit results, you could also take a look at a function programming approach (e.g. Haskell) which does
lazy evaluation.
Re: very large arrays
Posted: Wed Jun 22, 2011 11:14 am
by tr0gd0rr
Is it data that could go into a MySQL table?
Re: very large arrays
Posted: Thu Jun 23, 2011 3:53 am
by yamobe
I can unset some unused data.. that's true... that would reduce to something around 300Mb perhaps... which is still very high... I don't think I can do better than that.
The algorithm is a heavy transport model that was done by a mathematician, so I can't do anything about that... It is what it is.
Instead of using C++ I could use html5 web workers or something like that and start from scratch ... but I am scared of debugging issues since I have never faced something like this before... Do you know other tools I could use?
function programming approach was my first approach and it was SLOW AS HELL so I changed all the code... all the data goes to a mysql table, but making sql requests each time I need something is heavier and slower... I was my approach also at some point.
I have 3 possible solutions as I see
- Using c++ in the server to make the heavy lifting
- Using some javascript/HTML5 thing to make the heavy lifting on the web browser and then throw everything into mysql
- Optimize my php code to kill unused data when no longer needed
2 first options force me to start from scratch!!!
Any other thought?
Thanks
