40,137 Spam were filterred out on a server per day, How well the solution protect yours Options

[pairote]

My latest stat.

Exim statistics from 2006-05-29 00:00:07 to 2006-05-29 23:59:37

Grand total summary
-------------------
At least one address
TOTAL Volume Messages Hosts Delayed Failed
Received 1307MB 8476 2261 350 4.1% 434 5.1%
Delivered 3042MB 19466 1396

User Specified Patterns
-----------------------
Total
Total mail blocked:........................................................................
......... 40137
Verify sender at SMTP time:.......................................................................... 2713
Spam mail blocked by manual blacklist sender at the SMTP time:....................................... 116
Spam mail blocked by manual blacklist host address at the SMTP time:................................. 0
Spam mail blocked by auto black list SA high score at the SMTP time:................................. 6391
Spam mail blocked by invalid HELO at the SMTP time:.................................................. 2543
Spam mail blocked by setting default address to :fail: at the SMTP time:............................. 13664
Spam mail blocked by auto black list after dictionary attack the SMTP time:.......................... 472
Spam mail blocked by RBL at the SMTP time:........................................................... 3019
Spam mail blocked by illegal File extension at the SMTP time and discarded after recieving email:.... 1254
Spam mail blocked by virus scanner at the SMTP time or discarded after recieving email:.............. 523
Spam mail blocked by SA at the SMTP time or discarded after recieving email:......................... 9442
Spam mail discared by SA high score (12-15) after recieving email:................................... 252

NOTE: My SA is running RAZOR, Pyzor, SARE, Custom SA rules, My own SURBL.

There are some optimization made on my server and is not updated in tutorial yet.
You may try it. It consume less CPU.
1. Disable Spam scanning for outgoing email. http://forums.rvskin.com/index.php?s=&...post&p=4901
2. Rearrange the ACL for RBL and unknown user http://forums.rvskin.com/index.php?s=&...post&p=4788
3. Auto blacklist the Server sending high score spam mail to the server.
http://forums.rvskin.com/index.php?s=&...post&p=4739
4. Auto disable spamd if the server is overloading. When the server is overloading, we should skip spam scanning to bring the load down. I am working on it.

My conclusion from the current stat:
- Changing the default email to :fail: is unevitable. You must do it to reduce the server CPU usage.
- Allowing user to run SA per domain basis is not a good idea. It increases the server load. You should not allow user to enable/disable spamassassin per domain basis. The configuration made by my tutorial is enough to catch the SPAM. If it still get through, there is very less chance the per-domain SA will help.
- SPAMs attempt will not decrease. The situation will worsen and worsen.
- You may need to cache RBL, and SURBL data on the local network. I don't run it yet.

Feel free to post your stat here.

Download the attached script to get the stat. Change its name to spamReport.pl.
http://www.rvskin.com/download/spamReport.txt

And run:

CODE

perl spamReport.pl

It will show you the yesterday stat.

#will show today stat

CODE

perl spamReport.pl 0

#will show 7 days old stat

You can set cron to generate the report daily. Run crontab -e and add line.

0 5 * * * perl /var/log/spamReport.pl

[Andr? Marcelo]

- You may need to cache RBL, and SURBL data on the local network. I don't run it yet.

This can reduce the load too?

thank you,

Andr?.

[pairote]

Getting these information faster will reduce the connection time between sender and mail server, and reduce CPU usage.

[Andr? Marcelo]

This server is a dual processor? and about the load, is normal?

Another question: on exim.conf, on RBL, you put the sbl.spamhaus.org, and in the /etc/mail/spamassassin/local.rf is defined URIBL_SBL and I read here: http://answers.google.com/answers/threadview?id=422251 that it used sbl.spamhaus.org, so my question, the sbl.spamhaus.org is verified 2 time by each email ?

thanks,

Andr?.

[pairote]

This server is a Xeon 2.8 GHZ with 2Mb of RAM. The load spike sometime when someone send the bulk emails. There are serveral optimizations on the server. You may not reach this level if you don't optimize it properly.

URI_SBL and RBL is a different thing. RBL test the blacklisted mail server. URI_SBL test the blacklisted domai name. For the URI_SBL, it query to multi.surbl.org only one time, and give you the result from 6 sources. It is very optimized.

However, SA also query the RBL. Due to the EXIM and SA is a different package, I don't think it is possible to make it run and query the RBL data only one time for both. If you check it separatly, EXIM has a cache system which cache the RBL result. Same as SA that cache the RBL queries. If you concern about the resource usage, you may drop the SA query for the duplicate RBL by give it score 0.00. I don't sure its rule name, please consult the SA manual.

[colorteck]

I am trying to use your script and getting these errors.

68) line 21, <FILE> line 35951.
Malformed UTF-8 character (unexpected non-continuation byte 0x6f, immediately after start byte 0xf1) in pattern match (m//) at (eval 68) line 21, <FILE> line 35951.


Also when i follow the instuction for adding

# Change the version number to the latest one
tar ?zxvf razor-agents-2.77.tar.gz
cd razor-agents-2.77/
# Update Digest::SHA1 before complie razor-agents
/scripts/perlinstaller --force Digest::SHA1
perl Makefile.PL
make
make test
make install

razor-client
razor-admin ?create
mkdir /var/spool/mqueue
chown mailnull:mail /var/spool/mqueue
razor-admin -d -create -home=/var/spool/mqueue/.razor/
razor-admin -register -home=/var/spool/mqueue/.razor/


I can not get the razor client to work or razor admin it gives me this error

root@svr06 [~]# razor-admin ?create
-bash: razor-admin: command not found

Thx

[Andr? Marcelo]

There are some optimization made on my server and is not updated in tutorial yet.

In your instalation service, do you include these optimization from the US 30,00 price?

thanks.

[pairote]

No. Due to it requires time to maintenance and adjust the appropriate values. The values appropriate on my server may not appropriate on yours.

[kevin2004]

I have tried to install this, but am not able to download the razor files as a gz. they are a tar.bz2. How do you unzip them? I keep getting a "not valid gunzip file" error.

Thanks, Kevin

[KevinUK]

I have tried to install this, but am not able to download the razor files as a gz. they are a tar.bz2. How do you unzip them? I keep getting a "not valid gunzip file" error.

Thanks, Kevin

bunzip2 -c razor-agents-sdk-2.07.tar.bz2 | tar -xf -

bunzip2 -c razor-agents-2.82.tar.bz2 | tar -xf -



-Kevin

[kevin2004]

wonderfull thanks

[Tomas]

Here are my stats!!

Exim statistics from 2006-10-13 00:00:10 to 2006-10-13 23:59:56

Grand total summary
-------------------
At least one address
TOTAL Volume Messages Hosts Delayed Failed
Received 700MB 8030 3017 186 2.3% 974 12.1%
Delivered 1026MB 9999 552

User Specified Patterns
-----------------------
Total
Total mail blocked:........................................................................
......... 30305
Verify sender at SMTP time:.......................................................................... 13333
Spam mail blocked by manual blacklist sender at the SMTP time:....................................... 0
Spam mail blocked by manual blacklist host address at the SMTP time:................................. 0
Spam mail blocked by auto black list SA high score at the SMTP time:................................. 5
Spam mail blocked by invalid HELO at the SMTP time:.................................................. 531
Spam mail blocked by setting default address to :fail: at the SMTP time:............................. 9361
Spam mail blocked by auto black list after dictionary attack the SMTP time:.......................... 1579
Spam mail blocked by RBL at the SMTP time:........................................................... 1116
Spam mail blocked by illegal File extension at the SMTP time and discarded after recieving email:.... 17
Spam mail blocked by virus scanner at the SMTP time or discarded after recieving email:.............. 23
Spam mail blocked by SA at the SMTP time or discarded after recieving email:......................... 4349
Spam mail discared by SA high score (12-15) after recieving email:................................... 733

[Ivan]

Hi Pairote, that is a really cool script. Thanks

If I just put it in a cron, would it email the results to me each day?

thanks
Ivan

[pairote]

If you add the cron by running crontab -e and add the cron line, cron will generate email and send to toot@yourHostName (which being forward to whatever you define in WHM/Contact Manager) automatically.

[Ivan]

If you add the cron by running crontab -e and add the cron line, cron will generate email and send to toot@yourHostName (which being forward to whatever you define in WHM/Contact Manager) automatically.

Thanks Pairote,
I addded it to my crontab -e and when it runs I just get an email with an error message that says:

sh: line 1: eximstats: command not found

the same command runs fine from the command line. What am I doing wrong?

Ivan

here is my crontab:

9 0 * * * /scripts/upcp
0 1 * * * /scripts/cpbackup
*/15 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap
9 23 * * * cd /usr/local/cpanel/whostmgr/docroot/cgi/fantastico/scripts/ ; /usr/local/cpanel/3rdparty/bin/php cron.php > /dev/null 2>&1
21 21 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
*/5 * * * * perl /etc/eximSpamDeny.pl > /dev/null 2>&1
0 6 * * * /scripts/exim_tidydb > /dev/null 2>&1
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
35 22 * * * perl /var/log/spamReport.pl
51 4 * * * /usr/local/cpanel/whostmgr/docroot/cgi/rvsitebuilderinstaller/autoinstaller.cgi

[pairote]

open the spamReport.pl and find eximstats change it to /usr/sbin/eximstats and try it again.

[Ivan]

open the spamReport.pl and find eximstats change it to /usr/sbin/eximstats and try it again.

Thanks Pairote, that worked.

Ivan

[michael_S]

This has been working great for me as well... Check out the stats. I am very pleased and spam has reduced to a trickle. The stats below are for 1 day on this server.

Exim statistics from 2006-11-03 00:00:01 to 2006-11-03 23:59:44

Total mail blocked:........................................................................
......... 86590
Spam mail blocked by auto black list SA high score at the SMTP time:............/spam attack/ 355006

Total mail blocked:....................................................................... per hour (each dot is 89 occurences)
------------------------------------------------------------------------------------------------------------------------------------------

00-01 2745 ..............................
01-02 3172 ...................................
02-03 3401 ......................................
03-04 3357 .....................................
04-05 3441 ......................................
05-06 3690 .........................................
06-07 3626 ........................................
07-08 4050 .............................................
08-09 3809 ..........................................
09-10 3951 ............................................
10-11 3834 ...........................................
11-12 3470 ......................................
12-13 3911 ...........................................
13-14 4283 ................................................
14-15 4221 ...............................................
15-16 4444 .................................................
16-17 3658 .........................................
17-18 3997 ............................................
18-19 3972 ............................................
19-20 3217 ....................................
20-21 2984 .................................
21-22 2965 .................................
22-23 3051 ..................................
23-24 3341 .....................................


Spam mail blocked by auto black list SA high score at the SMTP time:................................./spam attack/ per hour (each dot is 350 occurences)
--------------------------------------------------------------------------------------------------------------------------------------------------------

00-01 13500 ......................................
01-02 12443 ...................................
02-03 12918 ....................................
03-04 13323 ......................................
04-05 13306 ......................................
05-06 15369 ...........................................
06-07 15608 ............................................
07-08 16823 ................................................
08-09 16548 ...............................................
09-10 17477 .................................................
10-11 15799 .............................................
11-12 16789 ...............................................
12-13 16186 ..............................................
13-14 16434 ..............................................
14-15 16044 .............................................
15-16 16652 ...............................................
16-17 14614 .........................................
17-18 16333 ..............................................
18-19 14691 .........................................
19-20 14487 .........................................
20-21 11938 ..................................
21-22 11051 ...............................
22-23 11727 .................................
23-24 14946 ..........................................

[arteryplanet]

im having problem installing this looks like some tings has changed in the 2.82 version of razor....

root@server1 [/usr/src/razor-agents-2.82]# razor-client
This program is deprecated and no longer necessary. You may begin using Razor now.

root@server1 [/usr/src/razor-agents-2.82]# razor-admin ?create
An option needs to be specified, -h for help.

i did try runing with the new options.....

root@server1 [/usr/src/razor-agents-2.82]# razor-admin -create
nextserver: Bootstrap discovery failed. Giving up.


Any help will be appreciated.

Thank you!

[arteryplanet]

ok, got it working....now im seeing the following error in the logs....

Error in system filter: unknown filtering command "and" near line 85 of filter file

any idea?