Page 1 of 1

Decoding HTTP socket request

Posted: Sun Mar 14, 2010 5:13 pm
by iLdn
Some days ago i decided to try to understand how iTunes and the AppStore works.

So i've installed Wireshark and i've found the request that iTunes makes to the Apple server

Then i created my PHP page that send a GET socket request to the server and everything looks like working...

This is how the script looks like

Code: Select all

<?php
 
$socket = fsockopen ("", 80);
fwrite ($socket, "GET HTTP/1.1\r\n");
fwrite ($socket, "Host: \r\n");
fwrite ($socket, "User-Agent:\r\n");
fwrite ($socket, "X-Apple-Store-Front: 143441-1\r\n");
[altri header]
fwrite ($socket, "Connection: Keep-Alive\r\n\r\n"); // da notare il doppio a capo
 
 
while (!feof ($socket)) {
$part = fgets ($socket, 255);
print $part ;
}
 
fclose ($socket);
?>
The server send me a HTTP/1.1 200 OK page with some header (and that's ok) but also some text that i cant decript...

That's the page

Code: Select all

HTTP/1.1 200 OK Content-Type: text/xml x-apple-application-site: CUP X-Apple-Partner: origin.0 Vary: X-Apple-Store-Front Vary: Accept-Encoding x-webobjects-loadaverage: 0 x-apple-application-instance: 1101 x-apple-date-generated: Sun, 14 Mar 2010 21:45:22 GMT x-apple-request-store-front: 143441-1 x-apple-max-age: 3600 x-apple-asset-version: 70761 Content-Encoding: gzip Cache-Control: max-age=310 Date: Sun, 14 Mar 2010 21:51:55 GMT Content-Length: 8336 Connection: keep-alive 
 
‹í=ëwÚ>–Ÿ‡¿Â‡³ÙÝMð0ð[HOÚô‘™¦Í&i»3_z„ €'Æöø‘4ýëW’_’-#ÀI;6Á/ùê¾ï•®4}õceKÐ,×™uÕžÒ• c¸¦å,fÝ/·ïNÆ])cÛuà¬ë¸ÝW§Ž„>Ós׈VР   %Ԇ̺Ë0ôþåÇÇÇð<öw%[á*»’ðöÉC-øÆ²‹újÁÇw>0Bòr¥7Rº§¤iüéÌßìOüþ+ß ]õÓg¦žm!Û›¼½©ia~˜5XøJ:JŸ½‡O§x˜*ãïù•¸Õ?QO‘›^¢ ¹w |à¿ñ½äzh…6Œ¯OƒÐGè>½ŒË˜ÊÉQ|[äÛìM   ŽÁžF(LƒóÏó@#ä˿߄®{.ü€ðü:>|e™³þ G'~‹ÂUlkP¥¯–    Ý`ûnä¡Ê@w¦2o ”Ö-~Ÿô6“ZÀnTm¨ªh¯|xòÙ7óî Ìk×]½º3.z2Œ&ý+ïÔªÄ1\º¨é¦0oƒWM₏֔ö,[4˜£Ñ ÌTæLë‰O¢N¦¡Aù´i—ÎìúQÏÚé“w¼¶£Æl±%xнYZ¶éCç?‚yC„}ß,}dÔ,àHDT¥÷nàA»%M&¯ „OЈBº+h>µc¶„À‹œÐo>]¾sàí¨u,Þ[5í»NK"=ñ¢×‰@sý{7 [3N"ºüƒåI\O¾^;”q .ӂҍ熶µX¶ƒÐ"è_ÀÏŸí QÄ2~D~…ã¶Ÿˆ~¼„aKÆEÓ'#så¶$%"À] óZ¾q£v°¨Š™k¸X€–œ[~í÷í@'â3Þ _ÐG$v> îýv-bY:f胖ЩŠû«Û–;«‰È r l³ðøÖNNó[Íô÷&ù‚mò\ŒqËs}‚RèJŸr2}m<«}}’ædú"^Pò5töç•`©[Áüá|³,Òæ¿‰‚Ð]]|倜átô‡óâK÷“¬!éíÄÙ?3a"¿%Ó¦)Bö#‰á[ÊÞ EÔ^‹a<QÄp¤$ ¥pÃ)3Ÿû`ZƒPHC¸¾ï¶ãÆà<³ˆ'óWËLCúw`eÙ ‘¼/Eµ—ÌÛ_»«Ö2MF‘`êÆ°NÞYÕ‘¤í‘]DiÞ,]?”ÞYöª5åÞÑœ7‚³=ETÒíÒ·l»¥Ø Y‘jÚÌù¾ý*!xlÃýfrObî7òo¿¶1:Q”ÉDUõÌ‹âYŒXé/ÈÓDnž    ÷’±€u]W¶ü]ÿ>µ^7adZ{×çùâ    ûvÅk°7Þ’ÌŽ„ttâ}·4Ä×WDRØ :ß#!®nÓ©õEÜnì1¶ P.Áuî,£MO¹†À¶Â§}ÚŠ"˜B©Îgs‡ª"#´é„!ùAÂfþÍ™çIäE-88±·° gãí=ÛÈw¤9baü÷x°ûµSTOW4]WGƒÔ…ІÈ×äv 9 éÎõ¥½ê7ªÃád8ìëÝõÀºrMém÷Ò·%„¶ôù)í:D„Â[÷â½Ü¹»èÝk×½o!ÖÑu¼ ¿½ŽS¡ HeHßš‘±gšBêh+Pú!°œÕ>‡Bhpù¡± :ûMwQ€·t¿º°àŒ·âÖÈ1—ðaêuÅÚ®$÷·ü£uƒðÉn‡´m@½„æ~§{æ€j[qCKõŸê6`¶àýѰn…ÒO°T"2Z“m ¼Zº¡»ð·Üc¼E»•/på»ÈƆÈé  Ûv+#{ ï [²[ÊVöÆ5,`KI¦iä Lí¾p Î â¥[<ì³&‚s+«ú%´l+leöu+«ú ‚p¹ÏÁ% Ð5æj³Š[ ´Áµì¬±¼Ëêös!å+² šc砐ºX‰ÈäD\„ÙzÔA¹Ák×І Ø£`2SÖú#m8ʧÙÕåFw‘…HêP¯,ã>øcÿ¤â¦“‚ïe6RçlÑY+ÆVí+Bå” åˆÔ¾P½ç¾‡)ü‰f´˜¹BŠLx"‰•D´>¸óy+~ "¶6ß»Ð'™µàÏþ8ÖO‚ávÀ™¯'„Zá‘!ë§RŠ ²¥\ŠÚWE„qZ"Áã•‹=þ}Ne¦`Q›Èó±ùñÏòd@»˜…ê‹Çü È$ˆlS+à ÍÒD‘?ŸpßDö~'àS:J¹$èO€½††[3[B¥z·ÐX:®í.ÚáW¡üÄ×,ÿo¯ZÑ©üÙ6ÎeÀἝ̡‚?Ú–:·€¤÷`liµésÚÀ1÷˜Ë b»úQ†-› ˜Õ–.Ä,ûúƤc}¨è»»…I[eÈ•Fï*HK¤¹¥^f!œÁDDGæ¢,mT^¿U’FQ„æ"XÉPIfÝW®ÕÒ$wu(¤Þ÷³@ MRl·¦NêB«"µ8 WŽEÜ#2 7sîa[+åhª®ˆHþ¾HÆ¡k~°Ôlkx‰p&ÿ͍œ…tf"?¾-iÒ'B+ÏgE[³eÔ‘ÿ~ù 3ÚõÅfÊ·=]E˜ôÊǶs•å çú¤Š±-eŸ%ÁÂ-TÞj‰¨:á?Û¬u<±Ir§-'"‚ví;é>@ÛõZtþ'bÙ2Ͻ‡-ki•‘RŸ6k=ÔÉP|–És-¤¡ðùt³”N2ü¥!Ð&´VnЧ@¶ÓZ¸ò ½äWïÇʦáËÙl@•Á¢0ù&Eãkß}`ÂFø¶ž:Ç ¡Edþ ³@@Bü½-8G‚¹¡´ÁPòÖP ù¦ñH˜XKˆ‡ŠÐ ÎfQÊÖ0 9$"äl2Ãlk0bK<ƁI[@ ©ûxá:°ŠßÌ‘ˆ ]âyz¸ª¼qjgk8'ûwë¶…QJ–$St[Uò@ã1Ŷ`Ššo!0–èKš.ßpÌÖ@Uôv²¥M4€aàŰOðf1ˆì=õÝoÒÍõs= ñ “Ú¬Ô;LÀº‰V+fe/áI¨q{Ïoãü€    a£S!ø|Ò<2Jx×™+Æ.5ñ›,¥ÈEhÈÜ«u0’xà  qxÐq~“ͼ¨¹Ò°Ý&mwáFá7+\âW¡7ýòé]”Ál;›Âq‡Ù‡¿c鎔ŠûÝð÷î!ˆÑ¡Awr.LƒÈƒþÂvçÀ–Uù}O?9ùtóÊ3†‰j34mö—ãïË™’|¤)7d§!ôóT@®>Pw.føõGý³#íú‡"Äxk¢ÊÑÅéè1G|#Â::|Ä¿âGúï‚h¾²Â£þ9"<ÒtäĬÐÑ•kÜÃð}d…À'o'«Þàé¹q['_ng¥[—Áo¥ä —½ÉŽ¢ éù!CÍâÑŠJÌiÓ‰éÝ rN†LDpºžŒ;óÉFðª¢[õT   îfê ?Ä…†‹)ca΍ÄÚ œÚýJ˜?€á‰çûT¨Ë±8IÌ'—N!Õwðàð É‡¬Ð†”E=/[íð/Me²#V|8•Ùý²Ò_Ó+äÁR§Èq¢ø$Ó <<ŠvcÊI×H&í0èJ–³´æVxîƒ Í³î°Ø¥§B  š6&¬ÂâG^Åb4ÃBD˜‹Ãä¨;9œüøÑrqO’ÿô? Uª¯1æ07K>ÞGàÂAäu•®„8=?˜»aè®òcÞQ·"¢Á;_0f]ÝŒû7®íú³îd qGíL&Moßµí]¿‰ÑϸåY’‘5»x´°|šâ€¯Ý&‚Ýǰ#JõüBœ}êJÈteû¾m–bÃŽb|ß?ï ?è €>®wù÷sËGÏÄ+çÊw.ò]p-tÐE†+ã´S4/Ô6Ö1ã^]}¡7‚»…A(¡÷{À·¼Qœ"¬;B·Ž{JOëJžï"³>ͺIÊ.ÙT®{ÚaÅŸ¼i    1ÑŠtLÀ0§HLxl*c˜îªa[À¶Qo°²ŒJÉ¢Ö<2îú9Ôtè@¿Ë¼þó‹{y'ð‡‡ÐZuØÆnCfÇ}¼B ;v60€o²4¬D ܈Q"ˆ)G G½Q žX©Éè™ó EBÃ0u¨f¼3ýÞ“R\×™ö4 £6,‹%ňÙ+nŽ)H—ÍYiz…ô|ä«[Џ–.gÝ£= )ÌøB Þҝ»A{ÐD¶éYØs`([+lª‰wù=Ææ÷ɰç9Hó ¿”rᣃ‹¹¦øaeêGHc”)…4•K6¤qnð‘ƒ-³ Êú7D ãõ÷¡ÆÐë   Û ¶$ÌC¦TÒ;P¬tëJ¦ûèØ.0%ëjé:PBAD– ÝÙZL+ŽÚƒc)ò>À:Ø•Â%”lÂ|¿Kɽ“bâô_ÀtŠ-Æ#lÛP?™ðF˜L]Ëä„X¢´·áúXR?{Ðùrý±Æ· ÅX‘ST˹’wEV4]¾Š|t^žCÌeŐW?íÞôB±è‘FO†ñÔY   ˜æ !͍õ¿‘Ê'MXQél”…g³ì忨4?O1‘7•†ÄKj§pApº¹8¯]Û\3»÷jí0&3G0ëÜËT½ô©Éð13èÚ)Îj@™þ(óKQé D ÓržÀÉ=@‹¢Í¨H–r«ô×äÖ¸ÂÙçCyH©ßeæ]×ÔTÔH‰uÀ6#íâÓ–6cÃMyp:?ý7¥7™¬ÝòªÐèh§0"*ØÔnÙ¤nºó)ŸyÄµË}Â$âðåNœÄ·\Lœ¸|±-8)ü0p˜gÕ”a—3iÄ´0·¡)½Žž¤3Ï;fTñáѰ:t¾Ï£'t×w¥0(߯ ¿>åŽeÙ¿¨K» §‹öíÓI&ÒïkÆúö´žä¥ß÷¿s¿Ôúó3äaÜ—;´h6    màgß„~<'µ8s`+ÛŽ×̼"ܺW&mÆ¢¥ŒÛïÞa–A r‰Ï éùZeÉdÒå_Ôej|t«¹j;pȨ&hHKÎ;#§XP_*p®¨Ãf&‚P¯£Ûî!è0X[祒š4> tq±Ñ°<—¸T)œ÷•^ý?³‰dòÀ¡PòP(y(”<J %ÁBÉ|wƒbI`Û¤¿_ðúfx™Å—_Ý!s]GLò#m¨)_ªê=è{Ô}зӾ)‡½”âÊžÝX²ÂT#D‡"Ë(39oYÎwT”ÉèCaæA¸Â½?á>rî@º;RyزÃÈ|m8t(ä<r 9·ÄÇ¡sMÞ©†æ‡BÎC!硐óPÈy(ä<r 9…œ‡BÎC!硐óPÈy(ä<r>!çF[á#B\õT}¨ëý±:š3 ·ÐX:HâOÒ €°ð<ˆdnYgm q‰_,IÉ– Zzü!“Ÿíjý8ýÊKþêñi9¦µpeeS•GЬAôÝTGŠÁ|döÕ‰e>˜kƒ± UÓõNhæŽ4E3ôÁx¤ÁùØ04U›÷áXÓO4E)ª6èc½Ñ¤ÄO’r†e”C1uËc¶CqÞË'Ε¥ß½bý>›#?Łқ%òB8ÒÇ"&>|jŒ‚ÃDÙôÌóN” ¾¹Ê-©Á–û±°¬ÙCeƒi¼Ìš­k”O–Êý{âáÌOà!KxAÿ&ãN)Ž' î%éùRžåu¨<à@;ŽŽÇúnóú¬^Xʳ¯&Çúd×Ïm;Ë$=ÙP{#4¬P2:çÎV 6%£ÆoøƒL‰Üï–´.`˜Ò@2¬ªmŸ?„²ëniUÝÊ57ìÏŽð!ü`©Gí RýíYêÑÚA½†¥;&ͤµ³‚¿Ö„^º#.Û—ôÞÕ{/Ä.Ö[æ/íÓÄoEܼã}¨Tg‘ µ>|Ÿ-mYžUM   íVž#âÊõ"rX™XߎËd@]‹s d¢'®w/ÐÝÐ)U³µÒHI|¤Ÿ]O%À7šy{:½„NtÂÕé™mK×È·¶ÃE·éY*·“Aþ²º±ŠË ;r‰OüZ]pã1]À'~-2„7K÷‘îÄíW   Ÿúµº‘nØÃH†çýZ "ñÕ2¡[’‰œþµº"Órç®{Ï%=ùkõÅsM!Ý“«øÔ¯Õ8¡ñ…îG^a·¶S¹`™|LcC_šeÏu¶ä½ø‰'ìfÑ²çªæphÆg_µPdzÙ×WÎе   ›W‰3ÃJ±Ï¸Í|œL8.d»r¡/Åç,ÄÇ•¬¤Ïý«°%¿FéEqgmN]ëòK¸4Á®ç®ÙÚ c_A«Q(ðbg/åéÿâ'~ÕWþ@$ïáf§$îº;•=”æÑ"~E$½p >ôƒ¬ÂSjô°"ÏaEžÃŠ<1„‡y^ÄŠ
I think that this text is compressed using gzip, but i'm not sure about it, and i think that this text is (when you uncompress it) a XML, and so the real important part of the response.

Chi mi saprebbe dire come posso fare a decomprimerlo? Perchè una parte della risposta è in chiaro mentre il resto no?
Can you help me to understand how to uncompress it? Why i can read the header but the XML code is compressed?

thank you so much for the help!!

Re: Decoding HTTP socket request

Posted: Sun Mar 14, 2010 6:11 pm
by requinix
iLdn wrote:I think that this text is compressed using gzip, but i'm not sure about it, and i think that this text is (when you uncompress it) a XML, and so the real important part of the response.
Correct.

zlib
SimpleXML

For the record, cURL is much better at this than the code you'll write.

Re: Decoding HTTP socket request

Posted: Mon Mar 15, 2010 8:55 am
by iLdn
Why this code doesn't work?

Code: Select all

 
 
while (!feof ($socket)) {
$part = fgets ($socket, 255);
$notcompressed = gzuncompress($part);
print $notcompressed;
}
 
Maybe because not all the request is compressed?

Re: Decoding HTTP socket request

Posted: Mon Mar 15, 2010 10:18 am
by AbraCadaver
It's probably encrypted. Maybe using the fairplay keys?

Re: Decoding HTTP socket request

Posted: Mon Mar 15, 2010 10:40 am
by iLdn
AbraCadaver wrote:It's probably encrypted. Maybe using the fairplay keys?
Mmm but Wireshark is able to show me the XML text... How it is possible?


EDIT
I've made some others tests and I found that Wireshark shows me an header that i don't receive that is about Chunked Transfer Encoding.
i don't know this mechanism but maybe this need some passage more..

EDIT 2!
looking around i found this function:

Code: Select all

function read_chunked($stream){
    $data = "";
    do {
        for ($size = ""; $char != "\n"; $size .= $char) {
            if (feof($stream)) return $data;
            $char = fgetc($stream); // character in chunk size
        }
        if (feof($stream)) return $data;
 
        // size = [hex data]\r\n
        $size = base_convert(trim($size), 16, 10);
        if ($size == 0) break;
 
        $data .= fread($stream, $size);
        fread($stream, 2); // terminating \r\n
    } while (!feof($stream));
 
    return $data;
}
 
it seems working but first of all i think that i've to decompress the string. is it true?[/color]

Re: Decoding HTTP socket request

Posted: Mon Mar 15, 2010 12:06 pm
by AbraCadaver
I think you want to gzinflate() NOT gzuncompress(). You would do this after de-chunking, however if that header is not in the response then it shouldn't be chunked. If you look at cURL, it should do all of this for you if you set the proper options.

Re: Decoding HTTP socket request

Posted: Mon Mar 15, 2010 2:16 pm
by iLdn
i can't solve... i'm not able to set correctly cURL and gzinflate don't works :S

i tried also to use a function like that but without any result :cry: :cry: :banghead: :banghead:

Code: Select all

if (!function_exists('el_zip_gzDecode')) { 
    function el_zip_gzDecode ($data, &$filename = null, &$comment = null) { 
        $gzMagic = sprintf("%X%X",ord(substr($data, 1, 1)),ord(substr($data, 0, 1))); 
        if($gzMagic != "8B1F"){ 
            $unpacked = @gzinflate($data); 
            return $unpacked; 
        } 
        $flags = ord(substr($data, 3, 1)); 
        $headerlen = 10; 
        if ($flags & 4) { 
            $extralen = unpack('v' ,substr($data, 10, 2)); 
            $extralen = $extralen[1]; 
            $headerlen += 2 + $extralen; 
        } 
        if ($flags & 8){ // Filename 
            $new_headerlen = strpos($data, chr(0), $headerlen) + 1; 
            $gzfname = substr($data, $headerlen, $new_headerlen - $headerlen - 1); 
            $headerlen = $new_headerlen; 
            if(isset($filename)) $filename = $gzfname; 
        } 
        if ($flags & 16){ // Comment 
            $new_headerlen = strpos($data, chr(0), $headerlen) + 1; 
            $gzcomment = substr($data, $headerlen, $new_headerlen - $headerlen - 1); 
            $headerlen = $new_headerlen; 
            if(isset($comment)) $comment = $gzcomment; 
        } 
        if ($flags & 2) // CRC at end of header 
            $headerlen += 2; 
        $old_size = strlen($data); 
        $new_size = $old_size - $headerlen - 8; 
        $packed = substr($data, $headerlen, $new_size); 
        $unpacked = gzinflate($packed); 
        return $unpacked; 
    } 
} 
 
while (!feof ($socket)) { 
$part .= fgets ($socket); 
 
} 
$a = el_zip_gzDecode($part); 
echo $a;
Something comes back from the server... i've only to unpack it :(

Re: Decoding HTTP socket request

Posted: Mon Mar 15, 2010 5:38 pm
by iLdn
Uff.. When i'm solving a problem i find that there is something more that doesn't works..

I've just understood that the page go in a loop!!! Infact the script print the header, print the content crypted, and than it stop working and all the code after the while is ignored!!! And the browser continue to load the page...

What will be the problem ? : :cry: cry: