Page 2 of 3
Posted: Sun Dec 25, 2005 5:07 am
by matthijs
AGISB, I totally agree with you about the validating of all user input.
That's why I'm bothering everyone here asking a lot of questions
But I also learned about the defense in depth principle. So even if I (try to) validate the emailaddress a user has entered with a regex, I want or would like to double check and see if anything suspicious is being entered. (like a "multiparts*/s*mixed" or bcc:)
Because, maybe the regex I use is not sufficient? Regex's can be very hard, and I'm no regex expert, so maybe there's a small mistake somewhere in the email validation routine. Or maybe I made a mistake copy-pasting a piece of code (ok, that would be stupid. But we make mistakes, do we?).
For example, if I look at the regex provided by ilovejackdaniels.
http://www.ilovejackdaniels.com/php/ema ... alidation/
That was published 1 june 2004. Since then some people have commented on the code and found mistakes or errors. they were acknowledged by the author, and the regex was updated at some point. So at some time, and maybe even still at this time, the regex could be insufficient. In that case I would like to have a backup validation routine, checking for suspicious input or newlines etc.
(p.s. I'm not nitpicking on Dave, just using it as an example)
So, you are correct in saying that the solution "Just validate any user input" is simple. However, "Just validate any user input" is not so simple. As can be seen by the many many threads in which people discuss what's good or not, better or not.
Posted: Sun Dec 25, 2005 7:16 am
by Buddha443556
Matthijs, when you get done coding this email thing might you consider coming back here and posting it for review. [Just remeber such reviews take time so don't expect an immediate response.]
AGISB, your absolutely correct all user data should be validated.
Merry Christmas!
Posted: Sun Dec 25, 2005 7:39 am
by matthijs
Buddha, I will do that. Really appreciate the valuable feedback I get and have gotten so far. And I promise I will not get angry when I don't get an answer in 5 minutes
First I'm going to my family (in-law) to celebrate Christmas and eat a lot. I'm sure I'm not the only one. So Merry Christmas everyone and speak to you later!
Matthijs
Posted: Sun Dec 25, 2005 5:33 pm
by matthijs
Hi, I'm back full of energy (literally...) to tackle this issue.
I set up some test cases, as shown below.
Straight from the manual with a small addition:
Code: Select all
<?php
$strings = array('email'=> 'mymail@mail.com',
'string1' => "asdf\n\r\t",
'string2' => 'arf12',
'string3' => 'LKA#@%.54',
'string4'=> "sender@anonymous.www%0ACc:recipient@someothersite.xxx%0ABcc:somebloke@grrrr.xxx,someotherbloke@oooops.xxx"
);
foreach ($strings as $name => $testcase) {
if (ctype_print($testcase)) {
echo "The string '$name' consists of all printable characters.<br>";
} else {
echo "The string '$name' does not consist of all printable characters.<br>";
}
}
?>
This will print out:
Code: Select all
The string 'email' consists of all printable characters.
The string 'string1' does not consist of all printable characters.
The string 'string2' consists of all printable characters.
The string 'string3' consists of all printable characters.
The string 'string4' consists of all printable characters.
As you can see, the string I have taken from
http://securephp.damonkohler.com/index. ... _Injection gets through.
So, Buddha443556 seems to be correct when he says:
ctype_print wouldn't be a safe option because it doesn't catch url escaped characters like "%0A"
I did try some more stuff, using a form with POST fields, like:
Code: Select all
<?php
if (!isset($_POST["send"])){
// no post data -> display form
?>
<form method="POST" action="<?=$_SERVER['PHP_SELF'];?>">
To: webmaster@website.com<br>
From: <input type="text" name="sender"><br>
Subject : <input type="text" name="subject"><br>
Message : <br>
<textarea name="message" rows="10" cols="60" lines="20"></textarea><br>
<input type="submit" name="send" value="Send"><br>
</form>
<?php
}else{
// found post data .. deal with it
$from=$_POST['sender'];
if (eregi("\r",$from) || eregi("\n",$from)){
echo "Not ok<br>$from";
}
else
{
echo "Ok<br>$from";
}
?>
But one thing I don't understand is that with all examples given on the damonkohler.com site, it always echo's out "Ok", even with evil input as:
sender@anonymous.www%0ACc:
recipient@someothersite.xxx%0ABcc:
somebloke@grrrr.xxx,
someotherbloke@oooops.xxx
So the eregi("\r", $from) || eregi("\n",$from)) solution given doesn't seem to help much against email injection?
Or should I set up my test case different?
I remember from a couple of months ago when I had to deal with email-injection attacks on a couple of sites, this was the reason I chose for another solution (looking for the occurance of different strings as shown in the function isInjection($text) )
What I do now is a combination of regex to validate the input and then with the email-injection function look for suspicious input.
Posted: Sun Dec 25, 2005 6:48 pm
by josh
Well it is trivial wether or not url encoded newlines will actually be rendered by your MTA as a new header value, regardless you could try something like
Code: Select all
$email = 'somestring';
$email_to_check = urldecode($email);
// check the decoded email
// send the string to the mail function
the decoded string was checked for newlines, the 'raw' string was sent to sendmail.. this should check encoded characters, but I still wouldn't personally put this code into my app.
Re: ctype_print for email injection prevention
Posted: Mon Jan 02, 2006 4:30 pm
by shiflett
matthijs wrote:Code: Select all
$_POST['email'] = preg_replace("/\r/", "", $_POST['email']);
$_POST['email'] = preg_replace("/\n/", "", $_POST['email']);
This particular approach demonstrates a poor use of regular expressions, because both replacements match a literal string, not a pattern. It's best to use str_replace() in these cases.
You're right that ctype_print() can check for these cases, and it has the additional benefit of being locale-aware.
jshpro2 wrote:generally the header the attacker is overwriting is the "to" header
This isn't true, but I don't fault you - almost every example I've seen demonstrates this approach. The "real" attacks in the wild prefer to add BCC headers - lots of them.
Jenk wrote:To: is a field within the header, not it's own header.
The To header is indeed a header, so maybe you're just misunderstanding something?
matthijs wrote:Does it return false in any instance of a /r or /n?
Yes, assuming you mean \r and \n. Try this:
Code: Select all
<?php
var_dump(ctype_print("\r"));
var_dump(ctype_print("\n"));
?>
Buddha443556 wrote:ctype_print wouldn't be a safe option because it doesn't catch url escaped characters like "%0A".
What is there to "catch" in this case? :-)
I think you're getting your contexts confused.
matthijs wrote:But I also learned about the defense in depth principle.
Glad to hear this mentioned. :-)
To answer your original question. using ctype_print() seems like a perfectly reasonable Defense in Depth mechanism.
That's not evil. :-)
Try this:
Code: Select all
<?php
mail('you@example.org',
'My Evil Test',
'My Evil Message',
'From: sender@anonymous.www%0ACc:recipient@someothersite.xxx%0ABcc:somebloke@grrrr.xxx,someotherbloke@oooops.xxx');
?>
Passing a value in the URL offers no additional attack vectors. If you'll tell me where this is suggested, I'll try to email the author and get it corrected.
I hope this was helpful.
Posted: Tue Jan 03, 2006 3:06 am
by matthijs
Chris, that was certainly helpful. Thanks for your explanation.
using ctype_print() seems like a perfectly reasonable Defense in Depth mechanism
Good to know.
Passing a value in the URL offers no additional attack vectors. If you'll tell me where this is suggested, I'll try to email the author and get it corrected.
The examples are from
http://securephp.damonkohler.com/index. ... _Injection.
Although I understand the reasoning behing the examples, they do not work on my server.
My guess is that it has something to do with the php settings on my server, because your example doesn't either produce the results I expect.
Code: Select all
<?php
mail('you@example.org',
'My Evil Test',
'My Evil Message',
'From: sender@anonymous.www%0ACc:recipient@someothersite.xxx%0ABcc:somebloke@grrrr.xxx,someotherbloke@oooops.xxx');
?>
After replacing the email addresses with my own, only the '
you@example.org' receives an email. Whatever I try, no extra cc: or bcc: mail is sent. Of course, that's a good thing, but not being able to reproduce the examples is somewhat frustrating.
Posted: Tue Jan 03, 2006 5:12 am
by Maugrim_The_Reaper
Above post - surrounded by <strong>
User sends you data - you assume its bad - you make certain its not bad. If all you expect is an email, then all you need do is validate it (filtering emails is basically validation in this instance). The most comprehensive email format checker I know can be found right here at
viewtopic.php?t=38863 . This will validate any string as a valid email "form" (i.e. it will not check that the email exists on a remote server). There's a GPL'd (by Roja) and a Perl version of the same script linked from there somewhere.
As mentioned previously RFC compliant regex's are pretty large - but that's a small price to pay for something that works on all email addresses (not just the common variants). I still find applications which refuse to validate a simple "Padraic Brady"@example.com...
Posted: Tue Jan 03, 2006 7:59 am
by Roja
Maugrim_The_Reaper wrote:As mentioned previously RFC compliant regex's are pretty large - but that's a small price to pay for something that works on all email addresses (not just the common variants). I still find applications which refuse to validate a simple "Padraic Brady"@example.com...
Or Pádraic Brady, eh?
I think of email testing as a three step process.. someone correct me if I miss a potential step, please.
1. Validate email against RFC (Syntax checking)
2. Validate domain exists (DNS checking) - (Also, optionally check against spam blacklists)
3. Send the email and check for return or failure
The first I accomplish using the PHP variant of the definitive regex.
The second I accomplish by using a variant of checkdnsrr() and getmxrr(), with a fallback function for Windows to ensure cross-platform compatibility.
An article on devshed has a fairly decent implementation.
The third I accomplish using phpmailer. It makes mailing very easy.
By the time I get to actually sending the email, I've eliminated a considerable scope of false emails. Can someone get an email address through that doesn't exist? Sure. But the risk is relatively low, and I've rarely seen it in happen in my implementations.
With phpmailer, you have extremely fine-grained control over each element in the email, from the to/cc/bcc fields seperately, down to the encoding of the email. It makes it possible to create a very strong web-driven email sending system.
Posted: Tue Jan 03, 2006 8:41 am
by Charles256
what all headers do you include to make sure your e-mails get through? even with phpmailer I've seen..I'd say...30% failure rate of e-mails just getting bounced by the other server..any clues on what headers to include to make sure I get through?
Posted: Tue Jan 03, 2006 8:48 am
by Weirdan
Roja wrote:
someone correct me if I miss a potential step, please.
4. After a temporary failure (such as "try again later"),
re-send the message until it gets through (or the message could not be delivered in acceptable time-frame, usually 5 days). That's where most of implementation using phpmailer with smtp mailer fail miserably. And that's why I would recommend to use real MTA instead of sending mail directly to recipient's MX.
Posted: Tue Jan 03, 2006 8:50 am
by Charles256
MTA and MX...define please for the ignorant..myself included

Posted: Tue Jan 03, 2006 9:07 am
by trukfixer
Charles256 wrote:MTA and MX...define please for the ignorant..myself included

MTA = Mail Transport Agent (what it sounds like - Postfix, Exim, etc)
MX = Mail eXchange (a type of DNS record - there's A, CNAME, MX, etc records for various types of dns lookups)
Posted: Tue Jan 03, 2006 9:10 am
by Roja
Charles256 wrote:what all headers do you include to make sure your e-mails get through? even with phpmailer I've seen..I'd say...30% failure rate of e-mails just getting bounced by the other server..any clues on what headers to include to make sure I get through?
Personally, my rate is *far* lower than that. I'd say below 5%, but it would be a back-of-the-envelope number. Its certainly rare for me, so its not far off.
I make sure to set Address (To:), Subject, Body, From, FromName, Host, CharSet, WordWrap, and Mailer type.
Technically, the last two are used by phpmailer, not technically included in the mail itself.
With those set, and coming from a non-blacklisted email address, I have very few problems getting mail out.
Weirdan wrote:4. After a temporary failure (such as "try again later"), re-send the message until it gets through (or the message could not be delivered in acceptable time-frame, usually 5 days). That's where most of implementation using phpmailer with smtp mailer fail miserably. And that's why I would recommend to use real MTA instead of sending mail directly to recipient's MX.
Thats odd. If you send the mail using phpmailer to via your smtp server, your smtp server should handle the retries if you receive "Try again later" codes. Thats the smtp server's responsibility. However, I've never had an issue with that. Perhaps I've been lucky so far?
Charles256 wrote:MTA and MX...define please for the ignorant..myself included
Mail Transfer Agent, and Mail Exchange. MX is the DNS record for a mail server.
Posted: Tue Jan 03, 2006 10:22 am
by Maugrim_The_Reaper
I'd almost surrendered my entirely valid Gaelic name to oblivion on the internet before people started using charsets properly. There are still sites though that replace the gaelic á with the infamous ?. P?draic is not my name... Doesn't suit me...
Hotmail still gets it wrong, of course. As well as most of the sites that stick a UTF-8 charset sticker in their HTML but then forget to save the HTML file with a UTF-8 encoding...or use odd charsets on their database...or...
Lots of reasons...actually. Its depressing come to think of it. What does an Irish guy need to do to get a simple name with one á character displayed correctly these days?
On the email failure rate... What failure rate? Unless you're blacklisted, or you have the headers all messed up (and that's hard with phpMailer), you should be fine. Of course the email addresses may not exist, or you could be hitting some server imposed limit (check your host's conditions of use), but generally I rarely get problems. Based on a complaints level X 100 measure its miniscule.
Anyone think of any other reasons for a 30% failure rate? Kinda curious now.