Page 1 of 2

Stopping Spam Idea

Posted: Sat Feb 16, 2008 5:39 am
by mpetrovich
Like many of us, I have pondered the problem of how to stop Spam. So, here are my thoughts on the topic and an idea.

I am sure you are quite aware of the problems and economic burdens associated with Spam. Estimates are as high as 90% of the email traffic is Spam, and the United States is spending about $10 billion each year fighting the problem. AOL, for example, blocks 1.6 BILLION messages per day. That is a huge infrastructure burden. We have filtering systems, blacklists, whitelists, etc. trying to deal with the problem. Those systems are treating the symptoms and only add to the system overhead. What has also concerned me is seeing legitimate e-mail blocked by these systems. Filtering systems will always make two kinds of errors: rejecting good mail, and accepting bad mail, and those errors will never be both driven to zero.

I have also concluded that fundamentally changing the system is now a viable option. I believe you could shift the world e-mail systems within a year, if you could solve the problem and provide tangible benefit.

The fundamental change I am suggesting is moving mail from a push system to a pull system. Mail servers would only send header information, and the messages would be stored on the sender server, not the recipient server as they are today. That eliminates all messages with falsified sender information. If the sender is not legitimate, there would be no message to download.

In addition, I would establish an SSL/TLS connection to the sender server for authentication and certification. I think it is now time for secure e-mail. As you know the server-to-server connection is generally not secure. I would also compress messages. That will reduce bandwidth more, and increase transfer speed. I have not decided whether to connect from the recipient directly to the sender server or involve the recipient server for security.

Requests for messages would either download or delete them, or they would expire. Returned receipts would no longer be necessary and would be automatic. You would know when someone got your message.

Now, obviously to do this would require infrastructure changes. What is intriguing is that these changes are really not that difficult. Header transfer could take place with current SMTP. Mail server software would need a slight tweak. Mail servers need to keep the messages and need the retrieval mechanism. My current thinking is to provide sender e-mail address, recipient email address, and message ID to the sending server to retrieve the message. A reporting function would need to be created for the senders.

Then we have the client e-mail programs. These programs would need to retrieve the header information, and knowing messages are at the sender, provide the user the choice to retrieve or delete the message, or "always retrieve from this sender." Non-compliant older software could be given a link in the body, and the message could be viewed in a browser.

Obviously, filters, if necessary, would only be viable at the client, servers would no longer carry that burden. Servers may need to monitor hijacking, which unlike today, might be easier to detect. Servers could also be blacklisted. An advantage of blacklisting with this kind of system is that the messages could be stopped before they have invaded the users systems.

Companies/organizations could decide when to require messages sent in this format. If a company could reduce their server overhead they may want to require messages sent to them to follow this format by a specific date.

Well, that should give you some idea of my thinking. What are your thoughts?

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 10:25 am
by Bill H
How is that significantly different than the present? Whether it's just the header or the entire email, I still have to see if it's something I want to deal with or not. If it is I retrieve it, if not I delete it. I still have to deal with it, and the fact that it's merely a header and not the entire message doesn't seem to make a difference in the amount of time I spend dealing with it. How is a list of headers less cumbersome than an equally long list of complete emails?

In fact, the system seems more cumbersome, since the ones I do want to deal with are scattered all over the world on multiple servers and waiting for them to be retreived is going to be slower than simply deleting the unwanted ones from my server and making a single retreival from my server for all of the desired emails.

Plus, the header alone may not provide sufficient information, so to look at the content is going to require retrieving the mail itself from a different server than the one to which I am presently connected.

I agree that something needs to be done, but I don't think this looks like the best solution.

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 2:41 pm
by mpetrovich
Thanks for thinking about this. The difference is getting at the source instead of treating the symptom. My thought was to eliminate the ability of spammers to falsify the sending domain and email address. Nearly all spam e-mail messages carry fake sender addresses. What is really bad is when they use someone's real email address. Recently, three people I know had to change their email addresses because they were used by spammers. One person could not delete the bounce-back messages fast enough and had to kill their email address at the server. This is the point of the Sender Policy Framework (SPF). So, what I was proposing was to eliminate the ability of spammers to use a false email. That also eliminates the need for SPF and other schemes to certify the sender. If the sender is not legitimate, there will be no message to retrieve. Having invalid email addresses also waste resources with receiving servers sending useless bounce-back messages.

Spammers need to send out about one million messages to have one response. If you want to send out a million messages, you will need to
store a million messages. It also makes it a little harder for spammers. (Although, you might be able to program a special server to serve up a message.)

So, you would not need to read the messages to find out if it is legitimate, because it would have to be. I have considered your point regarding retrieval of messages. First, for some senders, I would want an option to "always download" and for others it would be the equivalent to pulling up a Web page.

Also, something I noticed this morning in the bounce-backs from a client's newsletter, is that for them, the number one reason for rejections is full mailboxes. This would also be prevented with the system I proposed.

The other feature would be the ability to send secure e-mail. That could be done other ways was well. Currently, server-to-server email transfer is not secure, but that could be a feature incorporated into such a system.

So, I think this would be a huge drop in server load. If 90% of email is Spam, that means we have to have 10 times the server bandwidth than is needed.

Again, thanks for thinking about this.

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 3:19 pm
by califdon
Disclaimer: Although I've used email since Arpanet days, I don't consider myself an expert on email protocols.

First, and most importantly, I was very glad to see someone giving thought to the spam problem at a fundamental protocol level. Even if your specific proposal doesn't develop into a workable and accepted plan, we (the Internet community) need to put as much thought into the problem as we can. It all has to start somewhere. Thanks!

To my non-expert mind, your overall procedure has some merits. Message storage at the originating server would seem to offer a number of advantages, as you outlined. It also carries some costs. But if it could significantly reduce spam volume, the costs might very well be outweighed by the benefits.

Do you know what may be being considered by W3C, IETF or other standards groups already? There is already a limited implementation of "Net 2.0", I believe. I recall hearing talk about a new email protocol in connection with that. A quick search on Google turned up a few related documents, but I would have expected more (maybe with different keywords):
http://www.tmcnet.com/usubmit/2006/04/18/1581774.htm
http://thespamdiaries.blogspot.com/2006 ... fussp.html
http://www.ietf.org/mail-archive/web/as ... 12778.html
http://amtp.bw.org/
http://blogs.techrepublic.com.com/security/?p=393

Hope this adds to the discussion.

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 6:08 pm
by mpetrovich
Califdon, thanks for the links. I went through all of them. The last link at TechRepublic outlines the what I am getting at, and the statement that mail needs to be like RSS gets exactly to my concept.

Also, I do need to add that most Spam today is generated through hijacked computers. It is estimated that 80% of spam is delivered this way. Viruses and Trojans send out messages from unwitting computers. Spammers also use special software with dynamic DNS to send Spam directly from a PC. These are the kinds of things that would be prevented.

Any proposed changes need to consider the transition. I think you could keep the SMTP structure to send header information. What would need to change is the mail server software and the client software. If the client software had not been changed, a link could be provided in the message body to retrieve the message via the Web. So, to get started, you just need a server that stores the messages and provides the retrieval mechanism. Modifying the client software should not be difficult. Webmail would be very easy to change. 100-200 lines of new PHP code would most likely get the basic job done (now there's a challenge). Modifying the server would be a little more work, but I do not think it would be that difficult. So, I think you could actually implement a system like this with the present infrastructure. As mail servers upgrade, all it would take is a few major companies to require messages to be received with a new format, and I think the transformation would be fairly quick if it worked well. I believe today if a system is easy to implement and could take a bite out of Spam you would see a transition faster than vinyl records to CDs.

The reason I posted this concept is because I keep thinking about this from time to time, and love getting feedback from my technical friends as to whether there is any validity to my thinking. There are probably lots of things I have not considered, and I appreciate the feedback.

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 6:17 pm
by Bill H
I see your point about the sender storing the sent mails, and that does make a good point. And the "always download" is a good point, but when would that happen? Would my server store those but not others?

For instance, I use webmail to filter my mail. I delete all but the ones I want from my hosting server and then download the remaining ones to my own computer. How would a system such as you propose interface with webmail? I can see ways that it would, but...

My own feeling is that rather than incremantal change a completely new protocol is needed, one that obsoletes smtp/pop completely, but I don't know what it would consist of. I'm reminded of Hyman G. Rickover, father of the nuclear Navy, and his "chariot thinking" philosophy. He said something along the lines that if conventional thinking were sufficient we would be fighting wars with armored chariots rather than tanks. I think a leap is needed here on the order of converting from sail to steamships, or from oil fired ships to nuclear powered ones. People with higher foreheads than mine will have to come up with it tho.

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 8:47 pm
by mpetrovich
My thinking regarding an "Always download" feature refers to the client. I am thinking of a process similar to what you do by checking online first and then downloading. Let's say you create a whitelist and when messages come through on your list, the messages from whitelist senders are automatically loaded into the client, and that could be done in the background. If a message comes from someone you do not know or have in your list, you might have the following options:
  • reject (and send a reject message back),
  • ignore and delete (so sender would not receive notice),
  • retrieve and add to whitelist,
  • retrieve and not add to whitelist
So, what this would actually do for you is not need to go online, because only headers would download. For example, Thunderbird (and others) have an option to only download headers. This would be very similar, except I am saying you could have an option where a whitelist entry would have the message downloaded as well. So, I am guessing 95% of your legitimate mail comes from trusted sources. So in most instances if you used a whitelist, most people would not even notice a difference.

I like the quote from Rickover. It's true that you always have to look at revolutionary versus evolutionary approaches. What I have suggested is probably somewhere in-between. There is a massive world infrastructure, and how you transition will be critical with whatever approach is used.

Re: Stopping Spam Idea

Posted: Sat Feb 16, 2008 11:44 pm
by Bill H
What happens when I see a message that I want and I go to retrieve it and the connection to the server that it's on fails? Or is extremely slow at that moment? At best I'm going to be retrieving messages from multiple servers. Despite it's issue with spam, people use email because it is instant and utterly reliable. (Until you put spam filters on it which generate false positives, anyway.)

You're also talking about my hosting server that now has a mix of emails some of which are headers only and some of which are complete messages. I can see that becoming significantly problematic in several ways. And the messages are treated in four different ways as determined not by the server, but by settings on my computer. Or are you suggesting those settings be on the server? That would be pretty massive when it amounts to dozens of settings for thousands of hosted customers.

Deisel-electric submarines were not true submarines at all, but merely ships that could submerge for limited periods. The GUPPY program in the 60's and 70's extended underwater periods for small percentages at a huge expense, but they were still merely ships that could submerge for limited periods. Not until Rickover rammed the nuclear power plant down the throats of the "armored oxcart" crowd did we have a true submarine - a ship that was in its element submerged.

Incremental change yields small improvement at large cost. For fundamental difference one needs fundamental change.

Maybe an incremental approach would have some usefulness, but the spammers are almost certainly going to subvert it and I don't see it making enough difference to offset the additional complexities and probable concomitant loss of speed and reliability.

It's not my intent to disparage constructive thinking, and certainly your thoughts are not without merit. But my hope is that there are some minds out there that are applying the Rickover kind of thinking to this issue.

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 4:02 am
by mpetrovich
If you want to retrieve a message and the sending server is slow, your message would of course be slow in arriving. I do not know if you get any mail where the graphics are on the sending server, but it would be equivalent to that. What I am finding is that email is actually less reliable today. Sometimes when you are trying to send messages to large email services like AOL, MSN, or Yahoo, the messages get bogged down and take awhile. If you consider AOL having to block 1.6 billion messages each day, even with the filters, the server load is quite high just from the Spam. So, I am thinking servers might get a little more reliable.

No, the servers would not determine if a message would be retrieved. That would be at the client. The server would just hold the messages until the client retrieved it. Whether to automatically retrieve a message would be a setting in your e-mail program.

I was just thinking that if I clicked on the PM link to send a Private Message via this phpBB, it would be a very similar process.

This probably is equivalent to putting a reactor in a submarine. By the way, I actually considered joining the Navy nuclear submarine program when I was in college.

Thanks for your comments.

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 8:39 am
by Bill H
You might guess from my comments that I did serve in submarines. In my case diesel-electric ones, from 1958-1963. We viewed the nuclear boats, which were coming online at the time, with something of a jaundiced eye.

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 11:35 am
by califdon
Since I have read that something like 90% of all spam is originated from less than 200 major spammers across the world, I have a far simpler and more effective proposal: FIND THEM AND SHOOT THEM.

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 12:37 pm
by Christopher
I have glanced at this thread occasionally ... interesting idea and discussion (including submarine history ;)), I think my question to mpetrovich is: What is the goal of this discussion? Certainly no one here can have much if any effect on the major email systems and protocols. Are you just interested in having your ideas vetted? Or is there something a group formed here could actually do to cause change?

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 12:39 pm
by John Cartwright
I don't know about you, but I wouldn't want to recipient server knowing I've read the email. If anything, that is worse for spam.

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 1:49 pm
by Bill H
If you want to send out a million messages, you will need to store a million messages. It also makes it a little harder for spammers.
I'll bet a spammer would not need more than a day to develop a method for sending millions of headers while needing to store a single message on his own, or someone else's, server.

Re: Stopping Spam Idea

Posted: Sun Feb 17, 2008 6:47 pm
by mpetrovich
Since I have read that something like 90% of all spam is originated from less than 200 major spammers across the world, I have a far simpler and more effective proposal: FIND THEM AND SHOOT THEM.
I am wondering if we could destroy them using our submarine technology?
What is the goal of this discussion?
This idea was something that was floating around in my mind for about 3 or 4 months. It seemed like a good idea to me, but I needed some feedback from some technically competent people to determine if it was crazy or not. I have pondered this problem from time-to-time over the years. I came up with another method a couple years ago, and then found IBM had patented it a few months before. Over the past six or seven years I have written a few different applications to send out daily messages and newsletters. It is always a battle to get legitimate mail through, especially with shared hosting systems. I have had more than one exchange with Spamcop, AOL, etc. working to help a client get their messages out. I have seen legitimate business mail blocked by various servers. I am finding the problem is getting worse than it was just a few years ago. As mentioned in my previous post, I have also seen hijacked email addresses that were essentially ruined. Although the filters are getting better, there is still a high server overhead. So, that is why I think this is big issue. I am not sure what will come out of this discussion. If it is a good idea, maybe someone will take it and run with it. If we can determine that this idea really sucks, then I can forget about it, free up my brain, and do something more productive with my life.
I don't know about you, but I wouldn't want to recipient server knowing I've read the email. If anything, that is worse for spam.
I have thought about that. As a sender, it would be wonderful to know your message was received. Now, I am assuming that the sender has to be legitimate. So, that might be OK for them to know that your message was received. But, if this was someone that you did not want, they could be blacklisted, with messages automatically deleted and ignored. You do have a much better chance of blacklisting if you have a verified sender. We also need to counter selling of email addresses, which is how snail-mail junk mail arrives. Although, again, you can blacklist anyone.
I'll bet a spammer would not need more than a day to develop a method for sending millions of headers while needing to store a single message on his own, or someone else's, server.
The battle that we are in is the Spammers versus us. We try to come up with a prevention method, and the Spammers come up with countermeasures. They are not going to give up easily -- big money is at stake. This quote is getting at the kinds of things that need to be though about. Currently, Spammers can easily send out millions of headers. You can buy software out of Russia that will send out bulk email directly from your PC and you can get a dynamic DNS service to make it work, and send out a million messages (you can do a search on this). Spammers also use hijacked PCs to do this. I am guessing this might be their counterattack. So, you could get Spam headers, but not the messages. So, first, the header needs to match the sender domain where the messages are stored. The domain could be verified. I think also that a sending server could be programmed to generate dynamic messages, instead of storing them. There might be some legitimate reasons for this as well. It would be good for newsletters. Now, one thing that is different is that since the message is stored externally, the recipient server could block messages before they are delivered, if spam is detected. That would be much less bandwidth than current filtering systems. That would require that the recipient mail server retrieve the messages. So, it is possible we might be able to nail down a Spam source more efficiently than we do now. So, we just have to think as Spammers and figure out how to break the system, and determine if countermeasures can fix it.

Thanks for all the comments, thoughts, and ideas!!