How to scraping?
Posted: Tue Aug 10, 2010 8:04 am
HI
As for example my site is http://stockpedia.info/price.php. If you open this page in your browser, it scraps and displays a page from a remote server to you. So if hundreds of users open this page at the same second, then my server will send thousand of requests at the same time to the remote server. My question is how the remote server handle the requests or how it detects each request at the same time? Does it use server's IP or user's IP to identify each request? I am telling this because I observed that during trading hour, when hundreds of users visit my page at the same time, it becomes slow and non-responsive. But if there is only 1 or 2 users, that page responses quickly. My question is, is it possible to show the data from the remote server by this type of scraping or I have to use cron jobs for make any single request and save the response to the database. After that when users will visit my page it will load the data from my database. Which one is workable?
As for example my site is http://stockpedia.info/price.php. If you open this page in your browser, it scraps and displays a page from a remote server to you. So if hundreds of users open this page at the same second, then my server will send thousand of requests at the same time to the remote server. My question is how the remote server handle the requests or how it detects each request at the same time? Does it use server's IP or user's IP to identify each request? I am telling this because I observed that during trading hour, when hundreds of users visit my page at the same time, it becomes slow and non-responsive. But if there is only 1 or 2 users, that page responses quickly. My question is, is it possible to show the data from the remote server by this type of scraping or I have to use cron jobs for make any single request and save the response to the database. After that when users will visit my page it will load the data from my database. Which one is workable?