analyze PHP 7 Segfaults

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

analyze PHP 7 Segfaults

Post by Futureweb »

Hey there,

I hope someone here can help me with this rather special Question! ;-)

It's about analyzing sporadic PHP 7 Segfaults on one of my Servers.

I found some Information about analyzing Segfaults with GDB / Backtraces here --> https://bugs.php.net/bugs-generating-backtrace.php <-- But I guess that's not valid anymore with PHP 7 as I can't find "execute" calls anymore - only "zend_execute_scripts", ...
I already opened a Bug Ticket @ PHP - including some Backtraces & Error Logs - but unfortunately no reply for some time now: https://bugs.php.net/bug.php?id=73183

Hopefully someone here can help me finding the cause of those Problems - no luck on German Boards so far ...

Thank you, bye from Austria
Andreas Schnederle-Wagner
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: analyze PHP 7 Segfaults

Post by requinix »

The backtrace mentions /var/www/mypath11/html/index.php. Is that a real file? What's the code in it?
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

I modified Path/File Name for the posting. It's never the same File - it's completely random which file it happens!
Also those Memory Exhausted Errors are always shown after a SEGFAULT - so I guess that's just a side Effect of the Segfault and not the Cause?

Example of one of those memory exhausted Errors:
error_log-20161116:[Tue Nov 15 14:35:52.478805 2016] [:error] [pid 30926] [client 80.110.0.0:54073] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 140380662153216 bytes) in /var/www/path/html/templates/path/php/news_nuggets.php on line 218, referer: http://domain
216 Alle
217 </div>
218 <div class="referenzen_cst_filter_right_entry<?php if(isset($_GET['filter']) && $_GET['filter'] == 2) { ?> active<?php } ?>" id="referenzen_entry_2" rel="2">
219 Tourismus
220 </div>
There is nothing on Line 118 what could allocate 127TB of Memory... ;-)

That's why I want to analyze the Segfault - and not the OOM (I sense the Problem somewhere within ZEND OPCACHE ... it's just a feeling ... although this could be the completely wrong direction ...?)
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: analyze PHP 7 Segfaults

Post by requinix »

The memory usage thing using ridiculous values is definitely a sign of a bug - uninitialized value, buffer overflow, something like that. I don't have actual expertise reading backtraces but I can read source code so I'll keep looking.

Oh. And in case you have it on, turn off opcache for a bit as a test. It has sometimes caused problems in the past.

[edit]

This is opcode stuff, which I'm unfamiliar with. It's clear PHP is trying to output something when it shouldn't because there's an opcode telling it to, but where that faulty opcode is coming from isn't clear in the backtrace. Perhaps a faulty optimization.

It looks like index.php is including some file, which is then including another file, which is where the faulty output is happening. At least with the 17683 backtrace. I'd use some of the opcache functions to get more information - like opcache_is_script_cached() to determine whether the second file is being compiled or read from cache, and if the latter opcache_invalidate() to purge the second file from the cache to see whether that affects anything.

Another option is to fiddle with opcache.optimization_level. Set to 0 and see if the faults still happen. If not, toggle each bit individually until the faults return.
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

One thing which makes this Bug so damn hard to hunt down is the irregular occurance of it. Sometimes the System is working for 3-4 Days without any problems - then on the other day the Segfaults occur 5-6 times ...
The System where this happens is a custom CMS developed by us with about 15.000 PHP Files and some hundred thousands PageViews per Day. Also for Page-Generation there are dozens includes/requires on every Call ... :/

I guess turning opcache off for validating that it is the cause of the problems is the right first step - will do so now.

Bit more Information: I parse the Apache Error Log every 30 Seconds - and if I encounter a Memory Error / Segfault / zend_mm_heap_corrupted (at this Time Calls to our CMS fail!) - I trigger a "opcache_reset();" - after that the CMS is working again - guess that's a clear sign that OPCache is the source of the Problems?!
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: analyze PHP 7 Segfaults

Post by requinix »

If clearing the cache makes it stop happening for a while then that reinforces the theory that opcache is at fault.

Try the optimization thing before turning off opcache entirely. At the very least set it to 0 and see what happens - a few extra opcodes is better than no opcache at all.
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

alright - then I will go the opcache.optimization_level approach! ;-)
Still hoping to find the real Cause of the Problem - as OPCache is rather important for the good PageGeneration Times we got on our CMS ... not really happy to turn it off / disable some Optimizations! :-(

But no idea where to start debugging such a weird/irregular Problem.
#OPCacheMadness
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: analyze PHP 7 Segfaults

Post by Weirdan »

Getting VM stacktrace is rather easy: you copy .gdbinit from php-src to your home folder, then run the gdb with the core file you have and execute 'zbacktrace' function/macro. There's a lot more useful macros in that file as well.
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

Hi Weirdan,
thx for this Hint - pretty helpful those Functions! ;-)
Did it with all of my Coredumps ... but unfortunately it didn't really help on those ... different Files for each Coredump - sometimes Files/Lines are shown where no PHP Command is executed at all ... !?

Strengthens my beliefe that somehow OPCache get's corrupted and sends Bogus Data to PHP Engine ... but how to Debug that?
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

Some new Information - Today I set OPCache Log Level to Debug ... no Segfault so far - but some other weird Errors ... ?

At first some of those:

Code: Select all

Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
...
...
...
Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
Right after those Errors - some OOM Errors occured:

Code: Select all

[Thu Dec 01 11:45:52.556057 2016] [:error] [pid 11958] [client 80.110.0.0:50912] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 6935160699037703352 bytes) in /var/www/path/html/includes/path1/file.php on line 38, referer: http://www.domain.at/admin/menue/fil...111&id_usr=111
Line 38 of the File where OOM occured:

Code: Select all

if (!function_exists("get_galleries")) {
Did increase opcache.interned_strings_buffer from 8 to 16 now ... but there shouldn't be OOM Troubles when reaching Limit I guess ... (not to speak about Segfaults ... ;-))
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: analyze PHP 7 Segfaults

Post by requinix »

Be sure to post the warnings about the interned string overflows in the bug report - it's an important lead that can be followed.
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

Updated PHP Bug Ticket with those Informations: https://bugs.php.net/bug.php?id=73183
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

raising opcache.interned_strings_buffer did not do the trick ... no "Interned string buffer overflow" - but OOMs again! :-(

Code: Select all

...
error_log-20161204.gz:[Sat Dec 03 18:11:13.865592 2016] [mpm_prefork:error] [pid 172] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
error_log-20161204.gz:[Sat Dec 03 18:12:08.563986 2016] [:error] [pid 12391] [client 83.65.0.0:34298] PHP Fatal error:  Out of memory (allocated 30466048) (tried to allocate 4096 bytes) in /var/www/docroot/html/sitemap.php on line 227
error_log-20161204.gz:[Sat Dec 03 18:12:14.884734 2016] [mpm_prefork:error] [pid 172] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
...
error_log-20161204.gz:[Sat Dec 03 18:16:18.952980 2016] [:error] [pid 12172] [client 47.64.0.0:51096] PHP Fatal error:  Out of memory (allocated 2097152) (tried to allocate 90112 bytes) in /var/www/docroot/html/includes/functions.inc.php on line 225, referer: https://someDomain.com
error_log-20161204.gz:[Sat Dec 03 18:16:24.641049 2016] [deflate:error] [pid 12518] [client 37.61.0.0:51591] AH01383: unable to init Zlib: deflateInit2 returned -4: URL /index.php, referer: http://www.someDomain.com
error_log-20161204.gz:[Sat Dec 03 18:16:24.662726 2016] [deflate:error] [pid 12364] [client 66.220.0.0:44444] AH01383: unable to init Zlib: deflateInit2 returned -4: URL /index.php
...
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

Since I increased opcache.interned_strings_buffer from 8 to 16 there was no single SEGFAULT anymore ... guess the Problem must lie somewhere within Interned Strings Buffer Logic then?
User avatar
Futureweb
Forum Newbie
Posts: 10
Joined: Tue Nov 22, 2016 11:12 am

Re: analyze PHP 7 Segfaults

Post by Futureweb »

3 weeks without a single Segfault since I increased "opcache.interned_strings_buffer" ... Now I'm pretty sure there must be a Bug when Interned String Buffer is running full ... Hopefully someone over at PHP Bug Tracker will have a look into this! ;-)
Updates Bug Ticket with all relevant information ...

https://bugs.php.net/bug.php?id=73183

thank you all for your help
Andreas
Post Reply