I think it depends on a few things, including the language. (Your examples aren't in PHP.) In PHP, I'll usually throw an exception or return NULL. In your specific scenario, I'd probably throw an exception.
You might want to create a custom exception type by extending the base Exception class. You can also use the
SPL exceptions, such as the InvalidArgumentException, or maybe the OutOfBoundsException or OutOfRangeException types; I'm not sure which.
Of course, a hex RGB is simply a number between 0 and 16777215 (a 24-bit number) in base-16 notation. So if your code implementation manipulates these values as numbers, the following might be natural: If you get a value that is valid hexadecimal, but exceeds FFFFFF, say you get BFFFFFF, then you might choose to reduce it to the maximum value in the acceptable range, FFFFFF, and return the corresponding value without an error. On the other hand, if it contains non-hexadecimal characters, such as GFFFFFF, then it is not a number and you should throw an exception or return NULL.