Hi, I'm a PHP/Javascript developer working a web project that will require constant database updating. This updating will be independent of what the web users are doing. I'm willing to dive into another language to get this done. I've thought about using PERL, or PYTHON, or even C. Since I'd have to learn any of these from the ground up, I have the opportunity to select the ideal platform for this. My number one requirement should be database performance.
Any suggestions?
Back-end service
Moderator: General Moderators
Re: Back-end service
what's the kind of data would require constant updating? And what do you mean by constant? A thousand times per second, once a second, once an hour?
Re: Back-end service
I haven't decided how constant constant is, the tables will be large (100's of millions of records) though each record will be small (less than 100 bytes). Every record will need an opportunity to update based on other records in the same table. Once a run through completes, it'll be time to start another. I suspect it will take several seconds for each query to run though. Once I get my proof of concept going, I will likely be looking for ways to optimize the database, right now it's just mysql with godaddy, ultimately, I'll need to have this on a cloud I think. Right now, I'm just trying to find the best server side language to use.
Thanks
Jason
Thanks
Jason
Re: Back-end service
I think, based on your description the mysql server would be the biggest bottleneck here, so client language wouldn't really matter.
Re: Back-end service
Client side language isn't in question, I'm looking for a recommendation on the server side. I know the db will be the bottleneck, I think I mentioned that much. Still, if I ignore the issue of language now, when the time comes to move to a better database platform, I may end up having to rewrite the server side.Weirdan wrote:I think, based on your description the mysql server would be the biggest bottleneck here, so client language wouldn't really matter.
Re: Back-end service
By client I meant mysql's client - anything that would be accessing the database.
Generally the language choice affects CPU and memory usage, but your task doesn't seem neither CPU bound or memory hungry (unless you have to fetch data into application to process before updating the database). So, unless I missed something, it always will be disk-bound at the database server, regardless of the database software you use.
Generally the language choice affects CPU and memory usage, but your task doesn't seem neither CPU bound or memory hungry (unless you have to fetch data into application to process before updating the database). So, unless I missed something, it always will be disk-bound at the database server, regardless of the database software you use.
Re: Back-end service
Well, that's the thing, I will need to query the table to calculate the next round of updates. I don't expect the individual queries to return more than a few hundred records at a time, then maybe perform 10 to 20 updates based on what it finds in the query. My problem is that I would like this task performed constantly with different recordsets. So, one may not be processor/memory intensive, I'd like to run several of these simultaneously. Or, if I'm not able to execute queries simultaneously, then maybe run a select query to gather the data, and while the info is being calculated for the update, at least the next select can get started.
My whole question is evolving as I type this, I hope it's making sense. I envision it going something like this:
Process A responsible to gathering data from database (select * from whatever where this is that)
Results passed on to Process B
Process B is responsible to taking in data and calculating updates
Results passed on to Process C
Process C is responsible for writing updates to the database (update whatever with this where this is that)
Each process should be able to queue up it's requests such that Process A isn't waiting for Process B before moving on, and etc......
All the while this is going on in the database server, there's a web server (PHP/JavaScript) that can also hit this database with select, update, and insert queries.
I hope that paints a better picture of what I'm trying to accomplish.
Thanks
Jason
My whole question is evolving as I type this, I hope it's making sense. I envision it going something like this:
Process A responsible to gathering data from database (select * from whatever where this is that)
Results passed on to Process B
Process B is responsible to taking in data and calculating updates
Results passed on to Process C
Process C is responsible for writing updates to the database (update whatever with this where this is that)
Each process should be able to queue up it's requests such that Process A isn't waiting for Process B before moving on, and etc......
All the while this is going on in the database server, there's a web server (PHP/JavaScript) that can also hit this database with select, update, and insert queries.
I hope that paints a better picture of what I'm trying to accomplish.
Thanks
Jason