Page 1 of 1

Login and retrieve

Posted: Thu Jul 21, 2005 4:16 am
by bladecatcher
G'day All,
I'm trying to automate a download from a site where I'm a paid member.

To do this I must log in and then fill out a form (see below).

The site is MS using a VB_Script.

Once I figure out how to log in and the url to download the data, I think I have worked out the rest from the php manual.

My questions are:
How can I log in using php run from my web server?

What is the URL to download the data for today?

-----------------------------
LOGIN FORM
------------

Code: Select all

<BODY class=Body2 onload=document.frmLogin.memberID.focus();>
<FORM name=&quote;frmLogin&quote; action=&quote;https://www.xxxx.com.au/mb_login_page.asp?_sn=ED4EEE9E-7E68-4D97-8B70-E96EB1C20895&_c=&_dt=17%2F03%2F2005+10%3A12%3A01+AM&popUpType=Login&quote; method=&quote;POST&quote;>

<INPUT type=hidden name=&quote;GoBack&quote; value=&quote;&quote;>

Member ID

<INPUT type=&quote;text&quote; name=&quote;memberID&quote; size=&quote;20&quote; class=&quote;clsFieldData&quote; value=&quote;myname&quote;>

<br><br>

Password

<input type=&quote;password&quote; name=&quote;pswd&quote; size=&quote;20&quote; class=&quote;clsFieldData&quote; value=&quote;xyz123&quote;>

<br><br>

<INPUT TYPE=&quote;checkbox&quote; NAME=&quote;remember_me&quote; VALUE=&quote;Y&quote; checked>

remember my details

<INPUT class=&quote;clsBtnGo&quote; type=submit name=&quote;subLogin&quote; value=&quote;Login&quote;>

  

<INPUT class=&quote;clsBtnGo&quote; type=submit name=&quote;subCancel&quote; value=&quote;Cancel&quote;>

</FORM>
------------------------------------------------------
DOWNLOAD FORM
-------------------

Code: Select all

<!-- http://www.xxx.com.au/MB_Download_Price.asp -->

<!-- 
http://www.xxx.com.au/MB_download_price_show.asp?mode=all&oneday=20&onemon=07&oneyear=2005&dataformat=metastock&B1=Get+Historical+Data
http://www.xxx.com.au/MB_download_price_show.asp?mode=all&oneday=20&onemon=07&oneyear=2005&dataformat=metastock

Referring URL: http://www.xxx.com.au/Shares/default.asp?_sn=568D09AF-D6DA-4BC7-867C-0618B0F3C7D9&_c=71290&_dt=21%2F07%2F2005+3%3A42%3A70+PM

 -->


All Securities -  Only last 4 weeks available.
<br><br>

<FORM name=&quote;AllSec&quote; method=POST action=&quote;http://www.xxx.com.au/MB_download_price_show.asp?mode=all&quote; target=&quote;_blank&quote; <!-- onsubmit=&quote;return ValidateAll(this)&quote; -->>

Date
 
<input type=text name=oneday size=2 maxlength=2 value=01>/
<input type=text name=onemon size=2 maxlength=2 value=07>/
<input type=text name=oneyear size=4 maxlength=4 value=2005>
(DD/MM/YYYY)

<br><br>

Data Format:
<select name=dataformat class=&quote;clsFieldData&quote;>
<option value=metastock><FONT FACE=&quote;Arial&quote; SIZE=2>MetaStock</FONT>
<option value=ezy><FONT FACE=&quote;Arial&quote; SIZE=2>Ezy Chart</FONT>
<option value=ezy2><FONT FACE=&quote;Arial&quote; SIZE=2>Ezy Chart (Indices * 100)</FONT>
<option value=insight><FONT FACE=&quote;Arial&quote; SIZE=2>Insight Trader</FONT>
<option value=insight2><FONT FACE=&quote;Arial&quote; SIZE=2>Insight Trader (Indices * 100)</FONT>

<option value=metastockvol><FONT FACE=&quote;Arial&quote; SIZE=2>MetaStock (Vol % 100)</FONT>
</select>

<br><br>

<input type=submit name=B1 value=&quote;Get Historical Data&quote; class=&quote;clsBtnGo&quote;>

</FORM>
d11wtq | Please use

Code: Select all

or

Code: Select all

tags where required!   [/color]

Login & retrieve data

Posted: Fri Jul 22, 2005 12:56 am
by bladecatcher
G'day Guys,

Do you require more information?

Is the login process just a matter of submitting the name and password contained in a url?

then retrieve the serial number?

then submit the data request in the correct format?

tia
build

Posted: Fri Jul 22, 2005 1:32 am
by josh
I think a lot of people on this forum are going to question whether what you are doing is legit or not, as a lot of the members here have strong ethics, and people have posted a lot before about downloading entire image galleries via php and stuff like that, but anyways what do you mean by how do you login? Is there a specific question, or are you at just a total loss?

For a start try opening a socket to the url that handles logins, and sending memberId with your username and send your password and send whatever other variables their login form utilizes, you would then have to read in the response and parse it for any cookies they are sending, then you would subsequently place a request to the download page, passing along any cookie data to let the server know you're logged in, then parse the response for the url you want and you said you already know how to download it once youve got the URL

Hope I can help with anything else,
Good luck

Posted: Fri Jul 22, 2005 4:26 am
by timvw
Tools like wget are build for this...

appreciate your replies

Posted: Fri Jul 22, 2005 5:01 am
by bladecatcher
Quite legit guys, I'm a paid up member and have permission to automate the download process and permission to share the method I use.

I was going to use something like:

Code: Select all

$wpage = fopen(&quote;https://www.xxxx.com.au/mb_login_page.asp?_sn=ED4EEE9E-7E68-4D97-8B70-E96EB1C20895&_c=&_dt=17%2F03%2F2005+10%3A12%3A01+AM&popUpType=Login&quote;, &quote;r&quote;);
then parse for the stuff I need, that's why I wanted help working out the URL, the one I guessed was wrong.

Sockets huh? OK I've started with http://au.php.net/sockets

any other suggestions appreciated.

GNU wGet looks interesting too, might be a better answer but both will be educational.

Thank you both.
build

Posted: Sat Jul 23, 2005 1:47 pm
by josh
Regular expressions will help you find the URL once you are logged in.

Posted: Sat Jul 23, 2005 7:33 pm
by timvw
I've found the http://simpletest.sourceforge.net browser to be quite handy too ;))

I use it to log in on a website, click a few links, post a form automagically...
http://timvw.madoka.be/programming/php/smscity.php.txt

Posted: Sat Jul 23, 2005 9:32 pm
by Ambush Commander
Most of the time, authentication is done via cookies. For instance, I use a Fetch_WikipediaRecentChanges class to automatically grab my Watchlist every five minutes. Wikipedia, when you select Remember Me, sets this cookie that you simply have cURL or SimpleTest's simple browser present during the automated downloading.

So, your first job is to figure out how the website authenticates users.