Forums

[Bock's note: original post deleted - context: someone posted spams on different forums]

Please don't advertise here

And in light of this guys spamming effort, maybe we should make the forum posting for registered members only?

I fully agree, it sems andy has posted this useless message on SMSPower's other forums. Maxim, these need to be deleted.

I deleted the spams.
They come from time to time, not so much, and right now I'd rather have them deleted manually (by Maxim or me) than disable guest posting.

Currently we delete all the spam we see - I've probably deleted 20-30 since the board changed. If it gets to be more of a problem we can increase our efforts but keeping guest posting is one of our goals to reduce obstacles for new posters.

It is possible with this board for us to appoint "moderators" with the power to delete posts. If we have more problems we can consider appointing some trusted regular members from different parts of the world to provide more 24-hour coverage.

Hello Maxim,

I run phpBB through IIS and MySQL on my local machine, I use it to simply muck around on and to try and I think that if you use a usergroups policy it should be possible to create a user group with the access to delete spam posts from the forums.

Let me go now and have a muck around with my phpBB and I will report back here with my findings

Have an enjoyable day all,

Jacko

Maxim,

I have done some investigating and this is what I have came up with.

You will need to be in the Administration panel to do this.

Go to the Forum Admin -> Permissions and then look up the forums in which you want to change the permissions. Once you have selected a forum make sure you are in advanced mode. Once in advanced mode go over to the delete field and then select the MOD option from the dropdown box. Repeat this step for all of the other forums.

Now go to User Admin -> Permissions and look up the users who you would like to be a moderator then when the list of all of the forums comes up change the users preferences to Is Moderator in the dropdown box.

This final step is optional, you may want to create a new rank and call it "spam admin" or something, make sure you tick the special rank option so it is displayed throughout the forum. Then go to use management and change their ranks accordingly.

This should also be possible with a user group but I am not farmilliar with this.

Thanks,

Jacko

We already have that set up, thanks. It's just a question of appointing moderators when we need them.

So by this you mean that you would only change preferences on certain instances and not make the changes permanent.

If you want to do this then you have got me baffled.

Huh? I meant, we already have things set up so moderators can delete posts, we just didn't appoint many yet.

Oh sorry, I couldnt quite understand what you said, you have cleared it up now, thanks.

Unfortunately, even with recents tweak, the amount of spam is raising nowadays (we keep deleting them manually). One possible solution would be to discard guest posts using HTML, but it has yet to be seen if this can be implemented in a decent amount of time.

I have to be frank Bock but the spam on the forums in the last few days, particullarly over easter has just been un-acceptable for me. Is there anyway in which we can block a guests IP address??

To combat the problem also I think you need to assign more members in which you trust to be able to delete posts.

Make sure you get members that are all over the globe so if you are asleep and someone from the southern hempisphere visits the forums whilst you are asleep the spam problems can still be combatted.

We need to get on top of the spam before it gets out of control.

I had previously been aggressively using IP address blocking to avoid spam. However, I'd been blocking entire ranges from China (since we never had any Chinese visitors, and they were spamming us a lot). This is a problem because we now have several Chinese visitors and it is unfair to block them. I therefore emptied the IP blocklist after adding the spam words blocking mod.

The result of this is that we got a small flood of spam that might have been blocked before, but was not blocked by the new filter because it hadn't been filled with enough blocking words yet.

Note also that spammers rarely use the same IP address more than twice, and getting a new one from an ISP by DHCP is simple, so blocking individual IPs is not very useful.

The recent update to phpBB has also more aggressively killed HTML posted by these spammers, such that they get no useful effect from spamming us; eventually this is likely to make them give up, but not for a long time while old forums are online. We are looking into making a custom board modification to disallow all HTML from guest users.

Maxim, I've already been a moderator on another forum some time ago, so I know how this works. I don't know if you think I'm trustworthy, but let me know.

Blocking HTML from guest users should be reasonably effective, especially if it's a custom solution. I doubt the spammers are targeting this site specifically or even looking at it so they won't expend any effort to circumvent it.

The problem with the spamwords filter is that the spambot is pretty good about not repeating keywords. It looks like it's blocking about 4 or 5 messages a day, if the log section is accurate, and I'm personally deleting about 10 a day all on my own.

It's not unmanagable at this point but I do hope it doesn't get worse.

Heliophobe, Maxim, ando others

Is the spam word you implemented ablt to add words to itself and in essence 'train' itself to getting so know spam type words.

Atleast the mods are blocking a portion of the spam and yes Heliophobe lets hope we can keep the spam inder control, remember if it gets unmanagable I am sure there are other members out there who are more than willing to help out.

The spam words blocker needs to have words manually added - we have to be careful not to block legitimate uses of words like "casino" and "poker", for example. I try to add a blocked word for every spam I delete. It's expected that there will be more spam while we get it trained.

Regarding custom solutions: they seem to work well. I've read a lot about automated bot signups for phpBB accounts and we seem not to have that because we have some (kind of stupid) customizations to the signup pages. I'd like to delete all registered users who haven't posted, for fear of some automated atttack later, but it'd be too cruel to the lurkers.

I saw a phpBB mod that added a selection box saying "submit this post" that defaulted to "no"; simply by being non-standard it blocked almost all spam, but as soon as it gets widespread it'll be handled and thus become useless.

More options:
- CAPTCHA authentication for guests. Tricky to effectively use any off-the-shelf code for that, tricky to do it right - see PWNtcha. Almost entirely ineffective against real-people spam, and good CAPTCHAs are hard to read for normal people. Japanese people might not be so good at reading English CAPTCHAs and we have a hard enough time getting Japanese posts already.
- Block posts using disallowed HTML. It's aggressively stripped anyway (notice <a> in spams, ie. using < etc and removing the href= attribute) so it doesn't even work; rejecting such posts won't lose anything. We allow some HTML to make life easier for people, but the list of allowed tags is rather small.
- Disallow links ([URL] tags) from guests and/or users who haven't posted yet. This can potentially hurt newcomers who just want to post a useful link; but it will be more effective against both bots and human spam posters.
- Block posts not using correct encoding. Some of our spam has even managed to fail by being posted with the wrong encoding and ends up looking like "?????? ???? ????". I added "????*" to the blocklist, hopefully that'll catch them - I would need a less well-written browser to test it.
- Some cool Sega 8-bit quiz/authentication: sadly, it'd be too hard to make it possible for "newbies" to post if we did that.
- Jig the forms around a bit to confuse the bots and/or people - hard to do too much without making it confusing for users.

For now, I want to see how the spam words mod goes - give it a few weeks and then we can reconsider the options. It has the potential to be quite effective against common spam, no matter whether bots or people, HTML or BBCode, no matter what IP it's posted from. However, weird crap will always potentially catch it out but each time the word list will be improved.

What about on some forums how they have those fuzzy picture thingys that you have to enter the number in the box otherwise a post will not be submitted. Could that help the situation??

That's called a CAPTCHA. Software exists that can defeat most commonly-used ones - for example, the phpBB registration CAPTCHA comes up as 97% computer readable - and the ones that are hard to defeat by computer are also usually hard to read by people too. Add the fact that Japanese people can't be expected to be able to read them easily and they're not very useful. It is a possibility for the future, if we can make some kind of customised version.

What about using pictures instead of words? For example, five pictures and you need to write how much of them are Alex Kidds.

It'd be a block to people who don't know Alex Kidd and/or can't easily read English. It'd be trivial to defeat if anyone was actually bothered to code an attack just for this site.

Well, it seems to have leveled off to one or two a day (at least that I've seen). Have we done anything besides add words to the blocking list?

Incidentally, anyone planning an offtopic post probably shouldn't talk about their sex life what medications they're taking. Except tetracycline. Feel free to discuss tetracycline.

Like I expected, the spamming is tailing off as the filters pick it up, and maybe the spammers take us off their lists when they realise it's not getting through.

I did grab the IPs from the spam log and stuck them in the banlist*, but they weren't particularly being reused - I only saw a few used twice, the same day. Botnets have a lot to answer for.

*and accidentally IP banned myself, luckily I have ways around that...

I 'reckon you should use KittenAuth:
http://www.thepcspy.com/articles/security/the_cutest_humantest_kittenauth
To test it:
http://www.thepcspy.com/kittenauthtest
Cute. But it should work better than CAPTCHAs.

CAPTCHAs are ultimately doomed - apart from silly tests like KittenAuth, they have had to evolve to a state where they are only marginally human-readable to reduce the computer readability to acceptable levels. Thus we find things like AJAX CATCHA changing for the unreadable cases, annoyed people using imageless browsers and poor accessibility in general. And what effect do they have on human-posted spam?

Well, we seem to be doing OK on spam now. Attached is a little graph I cooked up using my l33t Excel h4x0ring 5ki11z, showing how many spams our spam blocker has caught per day. It looks like the amount of incoming spam has decreased markedly, leaving just a background level that it much more manageable.

spams.png (4.26 KB)

Spams blocked per day

Nice Work

Update:

Recently there have been some major spam attacks on forums (XRumer being at the forefront), which has lead to an emerging black market in lists of forums where automated signups and guest posting are possible. In order to maintain their lists, the spammers occasionally post test messages with no spam content and generic subjects like "I just joined, I like your site", etc. These are very hard to block using keyword filters. They often try to provoke conversation, perhaps to measure the "activity" of the forum, so please try to avoid replying to them.

Real spam includes URLs. We have a custom mod that blocks all URLs from guest users and it works well, but these generic test messages are still annoying. Some spam uses things like phone numbers and IM usernames, but we can't easily filter them.

Almost all spamming is done using botnets. IP blocking is thus not very useful.

One possibility that I'm seriously considering is to make a trivial change to the guest posting form. No spammer is going to bother targeting a custom mod on a small site like this. Our modified signup form (the "Do you like SEGA?" question) seems to do a pretty good job of keeping out bot registrations.

Options we want to avoid:
- third-party CAPTCHAs (will be targetted and broken by spammers, unless we make them almost impossible for humans to read; not friendly to non-English speakers)
- third-party spam checking services (our site then depends on theirs to work)
- disallowing guest posting

I also mentioned spam users. We have had a few, but they are outnumbered by the spam registrations that never post. These set their profile details with a spam URL and just earn PageRank. The problem is, I can only really clean them out by visiting all the users' home pages to check what's spam and what isn't. I believe many, if not all, are not bots, they are low-wage people in poor countries doing it for a job. No technology can filter them. Manual spam deleting will always be necessary.

Apart from the annoying generic messages, I believe our spam levels are below one per day now. I think that's pretty good.

Keep up the good work.

It might be worth asking Stan Stepanic about sega8bit as we never get any spam on the forum there.

The unfortunate thing is that I don't think there is any universal solution.

I also have a site of my own running phpbb and get almost hourly attempts by spambots to register/spam.

I have found that even if I change the structure of the signup page, they eventually realise and modify their bot to suite.

The best defence so far has just been vigilance in trimming inactive/spam accounts.

The most persistant ones appear to originate from either china or russian addresses.

Botnets are mostly from the US. The spam sites themselves are mostly in China and Russia.

Here's a question: if I deleted every account which has never posted, would that be harmful to lurkers who've registered (to reserve their name) but never posted? Maybe after announcing it in advance for a while?

I know a racketboy (www.racketboy.com) when you register there is usually a mathematical question, its usually 4 separate equations and it asks you to answer X equation. I would not recommend this sort of system cause 1) I either got the captcha wrong (yep I have bad eyesight, dont hold it against me) or I got the mathematical question wrong, it took me 4 attempts to register on their forum.

Also I have seen some mods where you edit lines on one of phpBBs files, its ment to stop automated bot registrations in their tracks.

What about changing platforms for the forum?? I know its a pain in the ass, but I have heard punBB is quite a lot more secure. Although its like anything, we made these fancy things and although you think its secure there's always the factor of human error and it will always be broken no matter how complex, its just ultimately time and patience. Anyway this forums spam is quite low compared to alex-kidd.com, its really bad over there.

We chose phpBB because it's mature and well-supported. Unfortunately that also means it's widely used and thus targeted by spammers. It's the same deal with viruses on Windows/Mac/Linux. PunBB will be targeted in proportion to its usage.

All phpBB mods are edits to lines in the files. The anti-bot ones confuse the bots which are hard-coded to some standard layout/ordering/filed names. Newer bots are more resilient to it, which is where you start adding in CAPTCHA stuff like equations. Only a custom, non-trivial CAPTCHA will work. I have a few ideas for that.

I've been thinking of adding reCAPTCHA to the forums. It's pretty good - offering audio and AJAXy image replacement for the unreadable ones - and nicely geeky too, with the book scanning aspects, while also being about as bot-proof as you can get.

The only problem I see is that it's not very friendly to people not as used to Roman text (eg. Japanese people). It would only be used for guest posts, with the possibility of using it for registration if the need arises.

I'll be sure to style it to fit with the forums, rather than use the default huge ugly red box.

Any comments?

Yes Maxim I think this is a good idea, over the past couple of days the music and main forums have been hit hard with spam, one thread in the music forum had replies too, and they were rude to top it off. For the time being though I would suggest try adding it to guest only posting, and then as you say if the need arises add it to rego and stuff.

How bad are bot registrations here anyway??

I've probably killed a handful of spam accounts over the past few years. They looked more like people than bots.

Another issue is spam registrants who never post - just try to take some Google-juice from the website link in the profile. I need to take the time to clear them out. I hate to delete lurkers, though, so I can't just delete all accounts with no posts. If you have some spare time, you can go through the members with no posts and mail me a list of the ones that look like spam.

Well consider this, with full publically viewable forums and guest access there's really no reason for lurkers to register at all.

If the person is not going to contribute to the site and just likes to use it as a reference or catch up on the latest Sega gossip they don't need an account. Even if they wish to post once in a blue moon it wont be that incovenient to post as a guest.

So I would be inclined to check occasionally for any accounts older than x months and 0 posts and delete them.

There is a "last logged in" timestamp in the DB too, which would be even better. Note to self:

SELECT * FROM `phpbb_users` WHERE `user_lastvisit` < UNIX_TIMESTAMP(NOW() - INTERVAL 6 MONTH) AND `user_posts` = 0

returns 523 rows, but certainly not all should be deleted.

Lurkers often register to "reserve" their preferred username, and also to enable the extra features like the ability to see new posts since your last visit, and read/unread thread statuses.

What about disabling URL's in the user-page? That would eliminate the bias of PageRank'ings. Are there any other reason we don't like spam-accounts? I am a lurker, and I like the feature to view posts since last visit (as Maxim mentioned). I would not like to be deleted because of inactivity.

By the way, that reCAPTCHA is simply ingenious. I would vote for it, simply because of the pure brilliance of the idea.

On a forum I help run, there was a massive problem with bots registering. We stopped them from posting, as the accounts stayed unactivated (admin activation), but that just moved the problem to my inbox. The solution that ultimately worked was the VIP code mod. The general idea is that bots will trip up over anything non-standard (this doesn't include some popular mods).

Spam users are OK until they start posting spam.

In my experience, adding extra posting feature mods only stops the bots for a short while. They're getting very good at handling such things, even custom mods are not bot-proof. ReCAPTCHA seems the only thing likely to work since it is by definition showing text that cannot (easily) be OCRed. I don't know how good the audio part is, though.

I can add a rel-nofollow to the profile links (effectively the same as removing them, they will get no Google juice) but the bots will not bother to check for it, it needs to happen in the default phpBB for them to take notice.

You can probably prune accounts based on last logged time if 0 posts and 0 PM. In the event that a few of them were person interested in "securing" their name, well, they can probably recreate the account if they do care.

Since guest posting is allowed, I don't see the benefit of deleting spam user accounts.

I don't think audio CAPTHCA's are a good idea.

The point is that software exists that can register and post spam, I can only presume it hasn't done that here because guest posting works.

Audio CAPTCHAs are an alternative for people who can't read. Of course they are also language-dependent and if someone can neither read nor hear they may be in trouble.

I've already added reCAPTCHA to my development version of the forums, it is quite painless but needs some CSS work to make it fit the site.

Maxim I have been noticing a lot of this

Topic Title:
Goood aafternoon look at sites

Posted By:
BorBluelryraw (Guest)

I assume that you have blocked the IP from where this is coming from??

Also I looked at the user lists and its kinda hard to tell which ones are spam and which ones arent.

Jacko wrote

Maxim I have been noticing a lot of this

Topic Title:
Goood aafternoon look at sites

Posted By:
BorBluelryraw (Guest)

I assume that you have blocked the IP from where this is coming from??

To be honest I didn't bother checking the IPs because it's almost certainly coming from a botnet. The text is not very spammy, it's probably a precursor to a load of blocked spam messages. I'll just kill them until I get time to add the reCAPTCHA mod.

Here I am posting as a guest with reCAPTCHA installed.

Well, that seems to have worked. Now let's wait and see what the spambots do.

Ok, I figured I would test too.

Author	Message
geishaboy Joined: 28 Dec 2004 Posts: 67 Location: Osaka, Japan	Spamming Posted: Wed Jan 04, 2006 3:47 am
	[Bock's note: original post deleted - context: someone posted spams on different forums] Please don't advertise here And in light of this guys spamming effort, maybe we should make the forum posting for registered members only?

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Wed Jan 04, 2006 4:30 am
	I fully agree, it sems andy has posted this useless message on SMSPower's other forums. Maxim, these need to be deleted.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Wed Jan 04, 2006 9:06 am
	I deleted the spams. They come from time to time, not so much, and right now I'd rather have them deleted manually (by Maxim or me) than disable guest posting.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Jan 04, 2006 9:32 am
	Currently we delete all the spam we see - I've probably deleted 20-30 since the board changed. If it gets to be more of a problem we can increase our efforts but keeping guest posting is one of our goals to reduce obstacles for new posters. It is possible with this board for us to appoint "moderators" with the power to delete posts. If we have more problems we can consider appointing some trusted regular members from different parts of the world to provide more 24-hour coverage.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Wed Jan 04, 2006 9:33 pm
	Hello Maxim, I run phpBB through IIS and MySQL on my local machine, I use it to simply muck around on and to try and I think that if you use a usergroups policy it should be possible to create a user group with the access to delete spam posts from the forums. Let me go now and have a muck around with my phpBB and I will report back here with my findings Have an enjoyable day all, Jacko

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Wed Jan 04, 2006 10:23 pm
	Maxim, I have done some investigating and this is what I have came up with. You will need to be in the Administration panel to do this. Go to the Forum Admin -> Permissions and then look up the forums in which you want to change the permissions. Once you have selected a forum make sure you are in advanced mode. Once in advanced mode go over to the delete field and then select the MOD option from the dropdown box. Repeat this step for all of the other forums. Now go to User Admin -> Permissions and look up the users who you would like to be a moderator then when the list of all of the forums comes up change the users preferences to Is Moderator in the dropdown box. This final step is optional, you may want to create a new rank and call it "spam admin" or something, make sure you tick the special rank option so it is displayed throughout the forum. Then go to use management and change their ranks accordingly. This should also be possible with a user group but I am not farmilliar with this. Thanks, Jacko

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Jan 05, 2006 9:23 am
	We already have that set up, thanks. It's just a question of appointing moderators when we need them.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Thu Jan 05, 2006 9:49 am
	So by this you mean that you would only change preferences on certain instances and not make the changes permanent. If you want to do this then you have got me baffled.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Jan 05, 2006 10:26 am
	Huh? I meant, we already have things set up so moderators can delete posts, we just didn't appoint many yet.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Thu Jan 05, 2006 11:09 pm
	Oh sorry, I couldnt quite understand what you said, you have cleared it up now, thanks.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Tue Apr 18, 2006 8:35 pm
	Unfortunately, even with recents tweak, the amount of spam is raising nowadays (we keep deleting them manually). One possible solution would be to discard guest posts using HTML, but it has yet to be seen if this can be implemented in a decent amount of time.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Wed Apr 19, 2006 8:35 am
	I have to be frank Bock but the spam on the forums in the last few days, particullarly over easter has just been un-acceptable for me. Is there anyway in which we can block a guests IP address?? To combat the problem also I think you need to assign more members in which you trust to be able to delete posts. Make sure you get members that are all over the globe so if you are asleep and someone from the southern hempisphere visits the forums whilst you are asleep the spam problems can still be combatted. We need to get on top of the spam before it gets out of control.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Apr 19, 2006 10:02 am
	I had previously been aggressively using IP address blocking to avoid spam. However, I'd been blocking entire ranges from China (since we never had any Chinese visitors, and they were spamming us a lot). This is a problem because we now have several Chinese visitors and it is unfair to block them. I therefore emptied the IP blocklist after adding the spam words blocking mod. The result of this is that we got a small flood of spam that might have been blocked before, but was not blocked by the new filter because it hadn't been filled with enough blocking words yet. Note also that spammers rarely use the same IP address more than twice, and getting a new one from an ISP by DHCP is simple, so blocking individual IPs is not very useful. The recent update to phpBB has also more aggressively killed HTML posted by these spammers, such that they get no useful effect from spamming us; eventually this is likely to make them give up, but not for a long time while old forums are online. We are looking into making a custom board modification to disallow all HTML from guest users.

Tom Joined: 16 May 2002 Posts: 1356 Location: italy	Posted: Wed Apr 19, 2006 9:51 pm
Tom Joined: 16 May 2002 Posts: 1356 Location: italy	Maxim, I've already been a moderator on another forum some time ago, so I know how this works. I don't know if you think I'm trustworthy, but let me know.

Heliophobe Site Admin Joined: 25 Oct 1999 Posts: 2029 Location: Monterey, California	Posted: Wed Apr 19, 2006 11:32 pm
	Blocking HTML from guest users should be reasonably effective, especially if it's a custom solution. I doubt the spammers are targeting this site specifically or even looking at it so they won't expend any effort to circumvent it. The problem with the spamwords filter is that the spambot is pretty good about not repeating keywords. It looks like it's blocking about 4 or 5 messages a day, if the log section is accurate, and I'm personally deleting about 10 a day all on my own. It's not unmanagable at this point but I do hope it doesn't get worse.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Thu Apr 20, 2006 1:06 am
	Heliophobe, Maxim, ando others Is the spam word you implemented ablt to add words to itself and in essence 'train' itself to getting so know spam type words. Atleast the mods are blocking a portion of the spam and yes Heliophobe lets hope we can keep the spam inder control, remember if it gets unmanagable I am sure there are other members out there who are more than willing to help out.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Apr 20, 2006 7:30 am
	The spam words blocker needs to have words manually added - we have to be careful not to block legitimate uses of words like "casino" and "poker", for example. I try to add a blocked word for every spam I delete. It's expected that there will be more spam while we get it trained.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Apr 20, 2006 10:31 am
	Regarding custom solutions: they seem to work well. I've read a lot about automated bot signups for phpBB accounts and we seem not to have that because we have some (kind of stupid) customizations to the signup pages. I'd like to delete all registered users who haven't posted, for fear of some automated atttack later, but it'd be too cruel to the lurkers. I saw a phpBB mod that added a selection box saying "submit this post" that defaulted to "no"; simply by being non-standard it blocked almost all spam, but as soon as it gets widespread it'll be handled and thus become useless. More options: - CAPTCHA authentication for guests. Tricky to effectively use any off-the-shelf code for that, tricky to do it right - see PWNtcha. Almost entirely ineffective against real-people spam, and good CAPTCHAs are hard to read for normal people. Japanese people might not be so good at reading English CAPTCHAs and we have a hard enough time getting Japanese posts already. - Block posts using disallowed HTML. It's aggressively stripped anyway (notice <a> in spams, ie. using < etc and removing the href= attribute) so it doesn't even work; rejecting such posts won't lose anything. We allow some HTML to make life easier for people, but the list of allowed tags is rather small. - Disallow links ([URL] tags) from guests and/or users who haven't posted yet. This can potentially hurt newcomers who just want to post a useful link; but it will be more effective against both bots and human spam posters. - Block posts not using correct encoding. Some of our spam has even managed to fail by being posted with the wrong encoding and ends up looking like "?????? ???? ????". I added "????*" to the blocklist, hopefully that'll catch them - I would need a less well-written browser to test it. - Some cool Sega 8-bit quiz/authentication: sadly, it'd be too hard to make it possible for "newbies" to post if we did that. - Jig the forms around a bit to confuse the bots and/or people - hard to do too much without making it confusing for users. For now, I want to see how the spam words mod goes - give it a few weeks and then we can reconsider the options. It has the potential to be quite effective against common spam, no matter whether bots or people, HTML or BBCode, no matter what IP it's posted from. However, weird crap will always potentially catch it out but each time the word list will be improved.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Fri Apr 21, 2006 5:28 am
	What about on some forums how they have those fuzzy picture thingys that you have to enter the number in the box otherwise a post will not be submitted. Could that help the situation??

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Fri Apr 21, 2006 7:33 am
	That's called a CAPTCHA. Software exists that can defeat most commonly-used ones - for example, the phpBB registration CAPTCHA comes up as 97% computer readable - and the ones that are hard to defeat by computer are also usually hard to read by people too. Add the fact that Japanese people can't be expected to be able to read them easily and they're not very useful. It is a possibility for the future, if we can make some kind of customised version.

chas Guest	Posted: Fri Apr 21, 2006 4:13 pm
chas Guest	What about using pictures instead of words? For example, five pictures and you need to write how much of them are Alex Kidds.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Fri Apr 21, 2006 4:45 pm
	It'd be a block to people who don't know Alex Kidd and/or can't easily read English. It'd be trivial to defeat if anyone was actually bothered to code an attack just for this site.

Heliophobe Site Admin Joined: 25 Oct 1999 Posts: 2029 Location: Monterey, California	Posted: Fri Apr 28, 2006 9:43 pm
	Well, it seems to have leveled off to one or two a day (at least that I've seen). Have we done anything besides add words to the blocking list? Incidentally, anyone planning an offtopic post probably shouldn't talk about their sex life what medications they're taking. Except tetracycline. Feel free to discuss tetracycline.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Fri Apr 28, 2006 9:47 pm
	Like I expected, the spamming is tailing off as the filters pick it up, and maybe the spammers take us off their lists when they realise it's not getting through. I did grab the IPs from the spam log and stuck them in the banlist, but they weren't particularly being reused - I only saw a few used twice, the same day. Botnets have a lot to answer for. and accidentally IP banned myself, luckily I have ways around that...

thematrixeatsyou Joined: 27 Feb 2006 Posts: 37 Location: Wellington, New Zealand	Posted: Fri May 05, 2006 5:27 am
	I 'reckon you should use KittenAuth: http://www.thepcspy.com/articles/security/the_cutest_humantest_kittenauth To test it: http://www.thepcspy.com/kittenauthtest Cute. But it should work better than CAPTCHAs.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Fri May 05, 2006 7:34 am
	CAPTCHAs are ultimately doomed - apart from silly tests like KittenAuth, they have had to evolve to a state where they are only marginally human-readable to reduce the computer readability to acceptable levels. Thus we find things like AJAX CATCHA changing for the unreadable cases, annoyed people using imageless browsers and poor accessibility in general. And what effect do they have on human-posted spam?

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Jul 17, 2006 2:45 pm
	Well, we seem to be doing OK on spam now. Attached is a little graph I cooked up using my l33t Excel h4x0ring 5ki11z, showing how many spams our spam blocker has caught per day. It looks like the amount of incoming spam has decreased markedly, leaving just a background level that it much more manageable. spams.png (4.26 KB) Spams blocked per day

Tom Joined: 16 May 2002 Posts: 1356 Location: italy	Posted: Mon Jul 17, 2006 3:47 pm
Tom Joined: 16 May 2002 Posts: 1356 Location: italy	Nice Work

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Nov 06, 2007 4:14 pm
	Update: Recently there have been some major spam attacks on forums (XRumer being at the forefront), which has lead to an emerging black market in lists of forums where automated signups and guest posting are possible. In order to maintain their lists, the spammers occasionally post test messages with no spam content and generic subjects like "I just joined, I like your site", etc. These are very hard to block using keyword filters. They often try to provoke conversation, perhaps to measure the "activity" of the forum, so please try to avoid replying to them. Real spam includes URLs. We have a custom mod that blocks all URLs from guest users and it works well, but these generic test messages are still annoying. Some spam uses things like phone numbers and IM usernames, but we can't easily filter them. Almost all spamming is done using botnets. IP blocking is thus not very useful. One possibility that I'm seriously considering is to make a trivial change to the guest posting form. No spammer is going to bother targeting a custom mod on a small site like this. Our modified signup form (the "Do you like SEGA?" question) seems to do a pretty good job of keeping out bot registrations. Options we want to avoid: - third-party CAPTCHAs (will be targetted and broken by spammers, unless we make them almost impossible for humans to read; not friendly to non-English speakers) - third-party spam checking services (our site then depends on theirs to work) - disallowing guest posting I also mentioned spam users. We have had a few, but they are outnumbered by the spam registrations that never post. These set their profile details with a spam URL and just earn PageRank. The problem is, I can only really clean them out by visiting all the users' home pages to check what's spam and what isn't. I believe many, if not all, are not bots, they are low-wage people in poor countries doing it for a job. No technology can filter them. Manual spam deleting will always be necessary. Apart from the annoying generic messages, I believe our spam levels are below one per day now. I think that's pretty good.

james costello Joined: 17 Dec 2005 Posts: 537 Location: England	Posted: Tue Nov 06, 2007 6:46 pm
	Keep up the good work. It might be worth asking Stan Stepanic about sega8bit as we never get any spam on the forum there.

thatawesomeguy Joined: 25 Jul 2007 Posts: 732 Location: Melbourne, Australia	Posted: Wed Nov 07, 2007 4:30 am
	The unfortunate thing is that I don't think there is any universal solution. I also have a site of my own running phpbb and get almost hourly attempts by spambots to register/spam. I have found that even if I change the structure of the signup page, they eventually realise and modify their bot to suite. The best defence so far has just been vigilance in trimming inactive/spam accounts. The most persistant ones appear to originate from either china or russian addresses.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Nov 07, 2007 9:17 am
	Botnets are mostly from the US. The spam sites themselves are mostly in China and Russia. Here's a question: if I deleted every account which has never posted, would that be harmful to lurkers who've registered (to reserve their name) but never posted? Maybe after announcing it in advance for a while?

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Wed Nov 07, 2007 9:20 am
	I know a racketboy (www.racketboy.com) when you register there is usually a mathematical question, its usually 4 separate equations and it asks you to answer X equation. I would not recommend this sort of system cause 1) I either got the captcha wrong (yep I have bad eyesight, dont hold it against me) or I got the mathematical question wrong, it took me 4 attempts to register on their forum. Also I have seen some mods where you edit lines on one of phpBBs files, its ment to stop automated bot registrations in their tracks. What about changing platforms for the forum?? I know its a pain in the ass, but I have heard punBB is quite a lot more secure. Although its like anything, we made these fancy things and although you think its secure there's always the factor of human error and it will always be broken no matter how complex, its just ultimately time and patience. Anyway this forums spam is quite low compared to alex-kidd.com, its really bad over there.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Nov 07, 2007 10:05 am
	We chose phpBB because it's mature and well-supported. Unfortunately that also means it's widely used and thus targeted by spammers. It's the same deal with viruses on Windows/Mac/Linux. PunBB will be targeted in proportion to its usage. All phpBB mods are edits to lines in the files. The anti-bot ones confuse the bots which are hard-coded to some standard layout/ordering/filed names. Newer bots are more resilient to it, which is where you start adding in CAPTCHA stuff like equations. Only a custom, non-trivial CAPTCHA will work. I have a few ideas for that.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Jan 07, 2008 1:10 pm
	I've been thinking of adding reCAPTCHA to the forums. It's pretty good - offering audio and AJAXy image replacement for the unreadable ones - and nicely geeky too, with the book scanning aspects, while also being about as bot-proof as you can get. The only problem I see is that it's not very friendly to people not as used to Roman text (eg. Japanese people). It would only be used for guest posts, with the possibility of using it for registration if the need arises. I'll be sure to style it to fit with the forums, rather than use the default huge ugly red box. Any comments?

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Mon Jan 07, 2008 2:19 pm
	Yes Maxim I think this is a good idea, over the past couple of days the music and main forums have been hit hard with spam, one thread in the music forum had replies too, and they were rude to top it off. For the time being though I would suggest try adding it to guest only posting, and then as you say if the need arises add it to rego and stuff. How bad are bot registrations here anyway??

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Jan 07, 2008 3:06 pm
	I've probably killed a handful of spam accounts over the past few years. They looked more like people than bots. Another issue is spam registrants who never post - just try to take some Google-juice from the website link in the profile. I need to take the time to clear them out. I hate to delete lurkers, though, so I can't just delete all accounts with no posts. If you have some spare time, you can go through the members with no posts and mail me a list of the ones that look like spam.

thatawesomeguy Joined: 25 Jul 2007 Posts: 732 Location: Melbourne, Australia	Posted: Mon Jan 07, 2008 4:14 pm
	Well consider this, with full publically viewable forums and guest access there's really no reason for lurkers to register at all. If the person is not going to contribute to the site and just likes to use it as a reference or catch up on the latest Sega gossip they don't need an account. Even if they wish to post once in a blue moon it wont be that incovenient to post as a guest. So I would be inclined to check occasionally for any accounts older than x months and 0 posts and delete them.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Jan 07, 2008 5:14 pm
	There is a "last logged in" timestamp in the DB too, which would be even better. Note to self: SELECT * FROM `phpbb_users` WHERE `user_lastvisit` < UNIX_TIMESTAMP(NOW() - INTERVAL 6 MONTH) AND `user_posts` = 0 returns 523 rows, but certainly not all should be deleted. Lurkers often register to "reserve" their preferred username, and also to enable the extra features like the ability to see new posts since your last visit, and read/unread thread statuses.

MarcK Joined: 10 Apr 2002 Posts: 212 Location: Denmark	Posted: Mon Jan 07, 2008 9:52 pm
MarcK Joined: 10 Apr 2002 Posts: 212 Location: Denmark	What about disabling URL's in the user-page? That would eliminate the bias of PageRank'ings. Are there any other reason we don't like spam-accounts? I am a lurker, and I like the feature to view posts since last visit (as Maxim mentioned). I would not like to be deleted because of inactivity. By the way, that reCAPTCHA is simply ingenious. I would vote for it, simply because of the pure brilliance of the idea.

DMEnduro Joined: 27 Apr 2005 Posts: 420 Location: Australia	Posted: Tue Jan 08, 2008 1:45 am
	On a forum I help run, there was a massive problem with bots registering. We stopped them from posting, as the accounts stayed unactivated (admin activation), but that just moved the problem to my inbox. The solution that ultimately worked was the VIP code mod. The general idea is that bots will trip up over anything non-standard (this doesn't include some popular mods).

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jan 08, 2008 9:32 am
	Spam users are OK until they start posting spam. In my experience, adding extra posting feature mods only stops the bots for a short while. They're getting very good at handling such things, even custom mods are not bot-proof. ReCAPTCHA seems the only thing likely to work since it is by definition showing text that cannot (easily) be OCRed. I don't know how good the audio part is, though. I can add a rel-nofollow to the profile links (effectively the same as removing them, they will get no Google juice) but the bots will not bother to check for it, it needs to happen in the default phpBB for them to take notice.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Tue Jan 08, 2008 9:53 am
	You can probably prune accounts based on last logged time if 0 posts and 0 PM. In the event that a few of them were person interested in "securing" their name, well, they can probably recreate the account if they do care.

MarcK Joined: 10 Apr 2002 Posts: 212 Location: Denmark	Posted: Tue Jan 08, 2008 12:40 pm
MarcK Joined: 10 Apr 2002 Posts: 212 Location: Denmark	Since guest posting is allowed, I don't see the benefit of deleting spam user accounts. I don't think audio CAPTHCA's are a good idea.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jan 08, 2008 1:09 pm
	The point is that software exists that can register and post spam, I can only presume it hasn't done that here because guest posting works. Audio CAPTCHAs are an alternative for people who can't read. Of course they are also language-dependent and if someone can neither read nor hear they may be in trouble. I've already added reCAPTCHA to my development version of the forums, it is quite painless but needs some CSS work to make it fit the site.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Tue Jan 08, 2008 1:43 pm
	Maxim I have been noticing a lot of this Topic Title: Goood aafternoon look at sites Posted By: BorBluelryraw (Guest) I assume that you have blocked the IP from where this is coming from?? Also I looked at the user lists and its kinda hard to tell which ones are spam and which ones arent.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jan 08, 2008 2:25 pm
	Jacko wrote Maxim I have been noticing a lot of this Topic Title: Goood aafternoon look at sites Posted By: BorBluelryraw (Guest) I assume that you have blocked the IP from where this is coming from?? To be honest I didn't bother checking the IPs because it's almost certainly coming from a botnet. The text is not very spammy, it's probably a precursor to a load of blocked spam messages. I'll just kill them until I get time to add the reCAPTCHA mod.

Maxim (Guest) Guest	reCAPTCHA test Posted: Tue Jan 08, 2008 7:12 pm
Maxim (Guest) Guest	Here I am posting as a guest with reCAPTCHA installed.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jan 08, 2008 7:13 pm
	Well, that seems to have worked. Now let's wait and see what the spambots do.

Jacko (Guest) Guest	Posted: Wed Jan 09, 2008 12:41 am
Jacko (Guest) Guest	Ok, I figured I would test too.

Forums

View topic - Spamming