Info
2006-01-22 greylisting might be ok
I've been working on a greylisting project in my spare time at home. It's been a bit of fun over the past few weeks. Everything looks good in test so I might be putting it on my own mail server in the near future.
The idea here being that once an inbound attempts to deliver mail, my server program delays the sender for 15 mins, if their IP address has not passed the greylist. Once past, it does a lookup against various RBL lists. If the mail is in an RBL list I have a choice, either put it on the shelf, or deny it.
If it gets shelfed, a challenge is sent to the sender, requesting they send a response to a randomized email address. If this challenge is responded to then the mail is passed along to the qmail-inject queue.
Serious thought has gone into these 1000-ish lines of perl. Most of the functions have been kept small. One afte thought is to run spamassassin on the mail, if the mail is not spammy then a challenge could be avoided, sine it passes the RBL and grey checks, the likelyhood of spam is far lower.
Other afterthoughts can include SPF checks, on big domain From: headers, such as paypal, microsoft, banks and other commonly spoofed addresses, requiring a challenge to be met.
Other recent thoughts include subjects such as why c# would be a preferred platform for some organisations, if offeres nothing IMO that cannot be offered via Sun's Java, which is free and taught by most education establishments.
A CSS design I have been working on has been submitted to oswd for their approval.
2006-01-16 sysadmin deleted the user's address books
problem: sysadmin deleted squirrelmail address books. solution: perl. Yes, another excuse to write some perl. Becomming quite a habbit this! Anyways, here's the perl script. addressbook.txt.
2006-01-08 subversion svn quick guide
I had a little learning exercise in subversion. Thought it would be worth the quick posting. In other news I think this site is outgrowing itself again. Having too many 'short' things in small entries is beginning to become disorganised through lack of structure.
apt-get install subversion subversion-tools svnserv -d -r /home/$USER/subv svnadmin create svn://localhost/sources cd /home/$USER/codes/java svn import svn://localhost/sources
Thats your subversion server setup. Simple huh. You can make it as complex as you wish through HTTP repository access, perhaps with a database backend if you want to be really posh.
Don't forget to alter the repository/conf/svnserve.conf file to enable write access.
2006-01-05 copy of copy of copy ...
Should the need ever arise, heres a smart way to download everything off your ftp server (perl required).
#!/usr/bin/perl -w
use strict;
use warnings;
use Net::FTP;
use Cwd;
# ftpcopy.pl, crude website search
# Copyright (C) 2006, ed neville
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
#
# purpose:
# to download a ftp directory and all subdirectories
#
# 20060105 ewn v0.1 inital program version.
my $ftp = new Net::FTP( "ftp.servername.com", Passive => 1 );
$ftp->login( "loginname", "password" );
my $localpath = getcwd;
my @remotepaths;
print $localpath;
sub listing
{
my $path = shift;
print "Path: $path\n";
if( $path eq "/" )
{
$path = "";
}
$ftp->cwd( "/$path" );
my @listing = $ftp->dir();
foreach my $file ( @listing )
{
$file =~ /^(.{10})(.*)([A-Za-z]{3}\s+[0-9]{1,2}\s+[0-9:]{4,5})\s+(.*)/;
my $filename = $4;
if( $file =~ /^d/ )
{
print "Dir $filename\n";
print "$localpath/$path$filename\n";
if( ! -d "$localpath/$path$filename" )
{
mkdir "$localpath/$path$filename";
}
listing( "$path$filename/" );
}
else
{
print "About to get $localpath/$path$filename\n";
$ftp->get( $filename, "$localpath/$path$filename" );
}
}
}
push( @remotepaths, "/" );
while( 1 )
{
my $seekdir = pop( @remotepaths );
listing( $seekdir );
if( scalar( @remotepaths ) == 0 )
{
last;
}
}
$ftp->quit;
2006-01-04 seek and ye shall find
Today I have rushed together a simple site search in perl. It has no caching and therefore quite slow. We did something a little more sophisticated at college and talleyed the word counts, matched them with similar words etc. I am surprised I've done it in ~100 lines though. That's not bad while remaining simple enough to read.
#!/usr/bin/perl -w
use CGI;
use strict;
use warnings;
# search.pl, crude website search
# Copyright (C) 2006, ed neville
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
#
# purpose:
# to make a simple search
#
# 20060104 ewn v0.1 inital program version. no caching.
#
my $query = CGI::new();
my $string;
my %form;
my $querystring = "";
my $referer;
my $siteroot = "/var/www/sites";
my %searchresult;
$referer = $query->referer();
$referer =~ s/^http:\/\///;
$referer =~ s/\/.*//;
print $query->header();
print $query->p( $query->referer() );
foreach my $p ( $query->param() )
{
if( $p eq "q" )
{
$querystring = $query->param( $p );
}
$form{$p} = $query->param( $p );
}
sub filecontains( $$ )
{
my $filename = shift;
my $searchstring = shift;
if( ! -f $filename )
{
$searchresult{$filename} = "Could not open source file";
return;
}
open( F, $filename )
or return( 0 );
while( my $line = <F> )
{
if( $line =~ /$searchstring/ )
{
$filename =~ s/^$siteroot\///;
$line =~ s/<.*>//g;
$searchresult{$filename} = $line;
}
}
close( F );
}
sub searchfor
{
my $path = shift;
my $querytext = shift;
opendir( D, $path ) or die( "cannot change to $path" );
my @files = readdir( D );
foreach my $file ( @files )
{
if( $file eq "." || $file eq ".." )
{
next ;
}
if( -d "$path/$file" )
{
searchfor( "$path/$file", $querytext );
next;
}
filecontains( "$path/$file", $querytext );
}
closedir( D );
}
$string = '<form method="post" action="/cgi-bin/search.pl" name="searchform"><input type="text" value="php" name="q"><input type="submit"></form>';
print $query->p( '<a href="/cgi-bin/search.pl">search</a>' . $string );
if( $querystring eq "" )
{
print $query->p( "Not given a query string" );
exit;
}
if( $referer eq "" )
{
print $query->p( "Cannot read referer value" );
exit;
}
if( ! -d "$siteroot/$referer" || $referer eq "" )
{
print $query->p( "$siteroot/$referer does not exist" );
exit;
}
searchfor( "$siteroot/$referer", $querystring );
while( ( my $key, my $value ) = each ( %searchresult ) )
{
print $query->p( '<a href="http://' . $key . '">' . "$key</a>: $value\n" );
}
2006-01-03 adventures getting listed on dnsbl
Just a report really of what happened this last week in the mail cluster. We have two boxes for clients who just use email forwarding with one box for hosted email.
The hosted email box has backup MX records pointing at the two forwarding boxes, they basically just queue the mail for the hosting box. This worked fine for quite some time with the validrcptto patch and mkvalidrcptto script. In the config we just had virtualdomain:primaryvirtualdomainbox in smtproutes, which was great, no problems there, except that when a backup received mail for the virualdomain, it could just accept anything@virtualhost, queue for a while and send to the primary, where it then bounced.
This vexxed me somewhat until I devised a plan to have two scripts, one which did about 1 quarter of the mkvalidrcptto script, ignores smtproutes, rcpthosts and virtual domains completely, just looks at the /etc/aliases.cdb, the virtual domains output, the system users, the .qmail-* files and builds a `hostname`.cdb, places that in a given directory and exits.
The second script merges all the .cdb files in the given directory into a single validrcptto.cdb, stored in /var/qmail/control.
Then all that was left was to distribute the `hostname`.cdb file among the cluster, and Bobs you're mother's brother, the jobs a good'un.
This works very well for my situation, if there's anyone who would like to make use of these scripts give me a nod and I'll post em, chances are this is too specific for individual cases, I doubt many people use the aliases.cdb file (despite it being less of a file system over head).
As I did give up three days of this long weekend to work on this, I'll post the scripts. Their usage is described above. If you notice anything with them not working how you would expect, let me know. As they're still alpha quality I advise that you pick them apart before doing anything in a production environment. View the qmail page for futher details.
In other news, MS have quite a serious WMF flaw on their hands. Effecting just about all versions of Windows. There's some interesting talk about this on the /. news page. I don't normally like to write about problems in other OS, and I never really want to get into a 'my os is better than your os' debate.
Oblig. Star Trek, Kirk: Fix the WMF hole.
Quick fix, regsvr32 -u %windir%\system32\shimgvw.dll
2006-01-02 Happy new maildrop
Been experimenting with maildrop this weekend (besides working on a cdb rcptto script). Maildrop allows me to organise mail as it arrives at the server rather than putting filtering on at the client, sometimes I like to use webmail which does not have all the filtering that Sylpheed has.
Anyway, heres a reduced version of my header filtering.
$ cat ./.qmail
| maildrop .mailfilter
$ cat ./.mailfilter
if( /^Subject: \(Re: \){0,}test.*/ )
{
to Maildir/.Trash/
}
if( /^Reply-To: mailgroup@yahoogroups.com/ || /^Subject: \[groupname\].*/ )
{
to Maildir/.maillists.operating_systems.linux/
}
if( /^Reply-To: prog@yahoogroups.com/ )
{
to Maildir/.maillists.programming.c/
}
In other news, my time has become too little to keep the openbsd qmr guide updated, I have therefore put it on a wiki page (or should that be, I am in the process of). Feel free to make changes, it's there for everyone. nixwiki, it is my intention to put important articles from this page into the wiki if it covers more than a single entry to cover it. The majority of the work is just converting the files in the qmr directory to wiki pages.