I need to retrieve and parse results of a search (a site that lists a bunch of restaurants according to various criteria). It'd be nice if the returned results had a nice URL (i.e. blah.com?q=restaurant) that I could simply cURL, but the site uses ColdFusion, so no such nice handy URL (i.e. just blah.com/results.cfm).
So--is there a way to crawl a website by submitting a form using PHP? I'd also need to press several "next" buttons, using PHP, for more results. Or any other workarounds?
Thanks!
trickly crawling question
Moderator: General Moderators
-
ultraPeanut
- Forum Newbie
- Posts: 2
- Joined: Wed Sep 22, 2004 11:14 am
Thanks. The problem though is that I don't know the URL for posting, since ColdFusion is used. e.g. can't use something like
since I dont' know the post url (the search results just come up like blah.com/results.cfm). This is the site I need to get results from: http://lapublichealth.org/rating/
Maybe I'm not using curl_setopt to its full advantage?
Code: Select all
curl_setopt($ch, CURLOPT_POSTFIELDS, "url=index%3Dbooks&field-keywords=PHP+MYSQL");Maybe I'm not using curl_setopt to its full advantage?
- feyd
- Neighborhood Spidermoddy
- Posts: 31559
- Joined: Mon Mar 29, 2004 3:24 pm
- Location: Bothell, Washington, USA
how is that not knowing where to post to?their page source wrote:Code: Select all
<FORM ACTION="/rating/ratesearchaction.cfm" method="post">