trickly crawling question

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
ultraPeanut
Forum Newbie
Posts: 2
Joined: Wed Sep 22, 2004 11:14 am

trickly crawling question

Post by ultraPeanut »

I need to retrieve and parse results of a search (a site that lists a bunch of restaurants according to various criteria). It'd be nice if the returned results had a nice URL (i.e. blah.com?q=restaurant) that I could simply cURL, but the site uses ColdFusion, so no such nice handy URL (i.e. just blah.com/results.cfm).

So--is there a way to crawl a website by submitting a form using PHP? I'd also need to press several "next" buttons, using PHP, for more results. Or any other workarounds?

Thanks!
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

curl can fake a form submit without much trouble. [devnet]+curl +post[/devnet] may help.
ultraPeanut
Forum Newbie
Posts: 2
Joined: Wed Sep 22, 2004 11:14 am

Post by ultraPeanut »

Thanks. The problem though is that I don't know the URL for posting, since ColdFusion is used. e.g. can't use something like

Code: Select all

curl_setopt($ch, CURLOPT_POSTFIELDS, "url=index%3Dbooks&field-keywords=PHP+MYSQL");
since I dont' know the post url (the search results just come up like blah.com/results.cfm). This is the site I need to get results from: http://lapublichealth.org/rating/

Maybe I'm not using curl_setopt to its full advantage?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

their page source wrote:

Code: Select all

<FORM ACTION="/rating/ratesearchaction.cfm" method="post">
how is that not knowing where to post to?
Post Reply