Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.
Just imagine the discomfort involved if you had to juggle your testicles around a little... You would be longing for the simplicity of convention... You could easily lose a testicle!
My threshold is readability and I certainly don't find your first example readable, whereas the second is completely clear. I tend to only use ternary operators for assignment in cases where they, in my opinion, make the code clearer. Mainly because an if() block (5-7 lines min) for a trivial assignment check (e.g. isset()) visually spoils readability for me. But I think where you draw the line and if you even use ternary operators is personal taste. I doubt there is any speed difference.
I would say in general rule if you are not sure -- do not use ternary operators.