Curl with .aspx file login
Posted: Sat Dec 13, 2008 3:25 pm
Hello,
I am trying to use CURL to scrape the data from a page that requires a user to log in. The problem is that CURL always seems to get the HTML from the login page even if I am logged in all ready by using the browser.
So if I am logged in the browser seems to store information somewhere because any time I go back to that link I am still logged on except when using CURL. If I use any of the buttons on this aspx page via the browser to perform a search for the data I need, the link does not change but the html displayed does. Then if I use CURL it will always go to the login page as if I never had logged on.
So is there any way to get the html that the browser is displaying based on choices made by the user for a .aspx file? If I just use CURL to go to that link it will always take me to the login screen as if the user has not done anything yet.
If there is another tool that would be better for scraping this type of data please let me know.
I am trying to use CURL to scrape the data from a page that requires a user to log in. The problem is that CURL always seems to get the HTML from the login page even if I am logged in all ready by using the browser.
So if I am logged in the browser seems to store information somewhere because any time I go back to that link I am still logged on except when using CURL. If I use any of the buttons on this aspx page via the browser to perform a search for the data I need, the link does not change but the html displayed does. Then if I use CURL it will always go to the login page as if I never had logged on.
So is there any way to get the html that the browser is displaying based on choices made by the user for a .aspx file? If I just use CURL to go to that link it will always take me to the login screen as if the user has not done anything yet.
If there is another tool that would be better for scraping this type of data please let me know.