Page 1 of 1

trickly crawling question

Posted: Wed Sep 22, 2004 11:19 am
by ultraPeanut
I need to retrieve and parse results of a search (a site that lists a bunch of restaurants according to various criteria). It'd be nice if the returned results had a nice URL (i.e. blah.com?q=restaurant) that I could simply cURL, but the site uses ColdFusion, so no such nice handy URL (i.e. just blah.com/results.cfm).

So--is there a way to crawl a website by submitting a form using PHP? I'd also need to press several "next" buttons, using PHP, for more results. Or any other workarounds?

Thanks!

Posted: Wed Sep 22, 2004 11:23 am
by feyd
curl can fake a form submit without much trouble. [devnet]+curl +post[/devnet] may help.

Posted: Fri Sep 24, 2004 11:10 am
by ultraPeanut
Thanks. The problem though is that I don't know the URL for posting, since ColdFusion is used. e.g. can't use something like

Code: Select all

curl_setopt($ch, CURLOPT_POSTFIELDS, "url=index%3Dbooks&field-keywords=PHP+MYSQL");
since I dont' know the post url (the search results just come up like blah.com/results.cfm). This is the site I need to get results from: http://lapublichealth.org/rating/

Maybe I'm not using curl_setopt to its full advantage?

Posted: Fri Sep 24, 2004 12:45 pm
by feyd
their page source wrote:

Code: Select all

<FORM ACTION="/rating/ratesearchaction.cfm" method="post">
how is that not knowing where to post to?