Page 1 of 2

analyze PHP 7 Segfaults

Posted: Tue Nov 22, 2016 11:18 am
by Futureweb
Hey there,

I hope someone here can help me with this rather special Question! ;-)

It's about analyzing sporadic PHP 7 Segfaults on one of my Servers.

I found some Information about analyzing Segfaults with GDB / Backtraces here --> https://bugs.php.net/bugs-generating-backtrace.php <-- But I guess that's not valid anymore with PHP 7 as I can't find "execute" calls anymore - only "zend_execute_scripts", ...
I already opened a Bug Ticket @ PHP - including some Backtraces & Error Logs - but unfortunately no reply for some time now: https://bugs.php.net/bug.php?id=73183

Hopefully someone here can help me finding the cause of those Problems - no luck on German Boards so far ...

Thank you, bye from Austria
Andreas Schnederle-Wagner

Re: analyze PHP 7 Segfaults

Posted: Tue Nov 22, 2016 12:22 pm
by requinix
The backtrace mentions /var/www/mypath11/html/index.php. Is that a real file? What's the code in it?

Re: analyze PHP 7 Segfaults

Posted: Tue Nov 22, 2016 12:33 pm
by Futureweb
I modified Path/File Name for the posting. It's never the same File - it's completely random which file it happens!
Also those Memory Exhausted Errors are always shown after a SEGFAULT - so I guess that's just a side Effect of the Segfault and not the Cause?

Example of one of those memory exhausted Errors:
error_log-20161116:[Tue Nov 15 14:35:52.478805 2016] [:error] [pid 30926] [client 80.110.0.0:54073] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 140380662153216 bytes) in /var/www/path/html/templates/path/php/news_nuggets.php on line 218, referer: http://domain
216 Alle
217 </div>
218 <div class="referenzen_cst_filter_right_entry<?php if(isset($_GET['filter']) && $_GET['filter'] == 2) { ?> active<?php } ?>" id="referenzen_entry_2" rel="2">
219 Tourismus
220 </div>
There is nothing on Line 118 what could allocate 127TB of Memory... ;-)

That's why I want to analyze the Segfault - and not the OOM (I sense the Problem somewhere within ZEND OPCACHE ... it's just a feeling ... although this could be the completely wrong direction ...?)

Re: analyze PHP 7 Segfaults

Posted: Wed Nov 23, 2016 12:52 am
by requinix
The memory usage thing using ridiculous values is definitely a sign of a bug - uninitialized value, buffer overflow, something like that. I don't have actual expertise reading backtraces but I can read source code so I'll keep looking.

Oh. And in case you have it on, turn off opcache for a bit as a test. It has sometimes caused problems in the past.

[edit]

This is opcode stuff, which I'm unfamiliar with. It's clear PHP is trying to output something when it shouldn't because there's an opcode telling it to, but where that faulty opcode is coming from isn't clear in the backtrace. Perhaps a faulty optimization.

It looks like index.php is including some file, which is then including another file, which is where the faulty output is happening. At least with the 17683 backtrace. I'd use some of the opcache functions to get more information - like opcache_is_script_cached() to determine whether the second file is being compiled or read from cache, and if the latter opcache_invalidate() to purge the second file from the cache to see whether that affects anything.

Another option is to fiddle with opcache.optimization_level. Set to 0 and see if the faults still happen. If not, toggle each bit individually until the faults return.

Re: analyze PHP 7 Segfaults

Posted: Wed Nov 23, 2016 3:15 am
by Futureweb
One thing which makes this Bug so damn hard to hunt down is the irregular occurance of it. Sometimes the System is working for 3-4 Days without any problems - then on the other day the Segfaults occur 5-6 times ...
The System where this happens is a custom CMS developed by us with about 15.000 PHP Files and some hundred thousands PageViews per Day. Also for Page-Generation there are dozens includes/requires on every Call ... :/

I guess turning opcache off for validating that it is the cause of the problems is the right first step - will do so now.

Bit more Information: I parse the Apache Error Log every 30 Seconds - and if I encounter a Memory Error / Segfault / zend_mm_heap_corrupted (at this Time Calls to our CMS fail!) - I trigger a "opcache_reset();" - after that the CMS is working again - guess that's a clear sign that OPCache is the source of the Problems?!

Re: analyze PHP 7 Segfaults

Posted: Wed Nov 23, 2016 3:30 am
by requinix
If clearing the cache makes it stop happening for a while then that reinforces the theory that opcache is at fault.

Try the optimization thing before turning off opcache entirely. At the very least set it to 0 and see what happens - a few extra opcodes is better than no opcache at all.

Re: analyze PHP 7 Segfaults

Posted: Wed Nov 23, 2016 3:50 am
by Futureweb
alright - then I will go the opcache.optimization_level approach! ;-)
Still hoping to find the real Cause of the Problem - as OPCache is rather important for the good PageGeneration Times we got on our CMS ... not really happy to turn it off / disable some Optimizations! :-(

But no idea where to start debugging such a weird/irregular Problem.
#OPCacheMadness

Re: analyze PHP 7 Segfaults

Posted: Mon Nov 28, 2016 7:48 pm
by Weirdan
Getting VM stacktrace is rather easy: you copy .gdbinit from php-src to your home folder, then run the gdb with the core file you have and execute 'zbacktrace' function/macro. There's a lot more useful macros in that file as well.

Re: analyze PHP 7 Segfaults

Posted: Tue Nov 29, 2016 10:51 am
by Futureweb
Hi Weirdan,
thx for this Hint - pretty helpful those Functions! ;-)
Did it with all of my Coredumps ... but unfortunately it didn't really help on those ... different Files for each Coredump - sometimes Files/Lines are shown where no PHP Command is executed at all ... !?

Strengthens my beliefe that somehow OPCache get's corrupted and sends Bogus Data to PHP Engine ... but how to Debug that?

Re: analyze PHP 7 Segfaults

Posted: Thu Dec 01, 2016 8:32 am
by Futureweb
Some new Information - Today I set OPCache Log Level to Debug ... no Segfault so far - but some other weird Errors ... ?

At first some of those:

Code: Select all

Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
...
...
...
Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
Right after those Errors - some OOM Errors occured:

Code: Select all

[Thu Dec 01 11:45:52.556057 2016] [:error] [pid 11958] [client 80.110.0.0:50912] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 6935160699037703352 bytes) in /var/www/path/html/includes/path1/file.php on line 38, referer: http://www.domain.at/admin/menue/fil...111&id_usr=111
Line 38 of the File where OOM occured:

Code: Select all

if (!function_exists("get_galleries")) {
Did increase opcache.interned_strings_buffer from 8 to 16 now ... but there shouldn't be OOM Troubles when reaching Limit I guess ... (not to speak about Segfaults ... ;-))

Re: analyze PHP 7 Segfaults

Posted: Thu Dec 01, 2016 10:17 am
by requinix
Be sure to post the warnings about the interned string overflows in the bug report - it's an important lead that can be followed.

Re: analyze PHP 7 Segfaults

Posted: Thu Dec 01, 2016 10:43 am
by Futureweb
Updated PHP Bug Ticket with those Informations: https://bugs.php.net/bug.php?id=73183

Re: analyze PHP 7 Segfaults

Posted: Mon Dec 05, 2016 6:19 am
by Futureweb
raising opcache.interned_strings_buffer did not do the trick ... no "Interned string buffer overflow" - but OOMs again! :-(

Code: Select all

...
error_log-20161204.gz:[Sat Dec 03 18:11:13.865592 2016] [mpm_prefork:error] [pid 172] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
error_log-20161204.gz:[Sat Dec 03 18:12:08.563986 2016] [:error] [pid 12391] [client 83.65.0.0:34298] PHP Fatal error:  Out of memory (allocated 30466048) (tried to allocate 4096 bytes) in /var/www/docroot/html/sitemap.php on line 227
error_log-20161204.gz:[Sat Dec 03 18:12:14.884734 2016] [mpm_prefork:error] [pid 172] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
...
error_log-20161204.gz:[Sat Dec 03 18:16:18.952980 2016] [:error] [pid 12172] [client 47.64.0.0:51096] PHP Fatal error:  Out of memory (allocated 2097152) (tried to allocate 90112 bytes) in /var/www/docroot/html/includes/functions.inc.php on line 225, referer: https://someDomain.com
error_log-20161204.gz:[Sat Dec 03 18:16:24.641049 2016] [deflate:error] [pid 12518] [client 37.61.0.0:51591] AH01383: unable to init Zlib: deflateInit2 returned -4: URL /index.php, referer: http://www.someDomain.com
error_log-20161204.gz:[Sat Dec 03 18:16:24.662726 2016] [deflate:error] [pid 12364] [client 66.220.0.0:44444] AH01383: unable to init Zlib: deflateInit2 returned -4: URL /index.php
...

Re: analyze PHP 7 Segfaults

Posted: Mon Dec 12, 2016 8:28 am
by Futureweb
Since I increased opcache.interned_strings_buffer from 8 to 16 there was no single SEGFAULT anymore ... guess the Problem must lie somewhere within Interned Strings Buffer Logic then?

Re: analyze PHP 7 Segfaults

Posted: Tue Dec 20, 2016 3:10 am
by Futureweb
3 weeks without a single Segfault since I increased "opcache.interned_strings_buffer" ... Now I'm pretty sure there must be a Bug when Interned String Buffer is running full ... Hopefully someone over at PHP Bug Tracker will have a look into this! ;-)
Updates Bug Ticket with all relevant information ...

https://bugs.php.net/bug.php?id=73183

thank you all for your help
Andreas