hosted services

Mutt is a highly capable email client. It's strongest feature for me is that it's console based. This means it can run in screen on the vps which I rent from bitfolk without the requirement of an IMAP server running as it's capable of reading local mailboxes just fine.

The most confusing part of mutt is that you have to create your configuration for your mail box from scratch, but once this is done it's smooth sailing with email that's all readable as plain text and editable with your favourite editor.

header_caching

One of the most important features that I have found in mutt is the ability to cache headers. This means fewer mailbox scans and can dramatically improve the performance and load on backend NFS or local disks.

To enable header caching in mutt add the following:

set header_cache=/localdirectory/

Where localdirectory is (preferably) a local directory on your system where you can put header cache data. It's possible to make this a remote directory but you might cost yourself in network resource overhead.

maildir

Another really important thing with mutt is that you can use it with Maildir format mailboxes which makes header caching possible. The following will enable Maildir mailboxes:

set mbox_type=Maildir

setting the envelope from

Another great thing about mutt is that you can define the "envelope from" address field:

set from="user-xyz@s5h.net"
set use_envelope_from=yes

This enables you to configure the address that your mail leaves with (I strongly suspect the -f parameter is defined when calling sendmail). One of the things I do as a qmail user is configure a user-default .qmail file and dish-out user-sitename@s5h.net addresses when using my email address on the web. When replying to this mail I have a script set the from header when replying, based on the Delivered-to header. It is then a very simple matter to filter this mail to defined Maildir boxes, which limits inbox spam.

256 colours

Not all terminals in the world can support 256 colours, but it's nice when they do. See the xterm-256color page for a listing of these. If you have your 256 colour terminal setup then why not make use of one of these themes, you can try them out using the :source command.

To enable the 256 colour set TERM=xterm-256color then make use of the extended colours.

I've put together a colour theme for 256 colour terminals, it's available in the mutt directory, let me know what you think, it's a bit blue.

themes

I've put some themes together in the directory above, if you're after a screen shot of the mutt themes then you can take a preview in the gallery selection below.

index	pager

Please give me your feed back as to the ones which you might prefer or perhaps seriously dislike, but I'd hope they're of some enjoyment to someone.

conditional configuration

Sometimes you want to keep the same configuration files but have then operate differently depending on where it's being invoked (consider running with a NFS home directory mounted on different architectures).

In these instances you might want to make use of the pipe configuration source, consider this .muttrc snippet:

source "~/.mutt/script/editor|"

With this script contents: #!/bin/sh

HOSTNAME=`hostname`;

if [ "$HOSTNAMEx" -eq "fileserverx" ] ; then
        echo "set editor=\"/usr/local/vim-7.0/bin/vim\"";
fi;

In this case we're letting mutt process the output of this script as if it's a config file. If the hostname matches fileserver then we configure a different editor.

Other parameters are worth consideration on a NFS system, such as setting header cache to a local file system which is most likely going to be faster than where ~/.mutt (most likely a NFS), so directing header_cache to the local disks is probably going to improve user experience considerably.

spamcop

A very handy feature for submitting mail to spamcop is to make a key binding for index and pager views. The job I have set configures gpg and the from mail addresses prior to sending and resets it to defaults after.

macro pager \cx ":set from=\"ed-spamcop@example.net\" autoedit=no \
fast_reply=yes editor=\"/bin/true\"\n:unset \
pgp_autosign\n<tag-prefix><forward-message>submit.<your \
id>@spam.spamcop.net\n<send-message>:set autoedit=yes pgp_autosign \
fast_reply=no from=\"ed@example.net\" editor=\"/usr/bin/vim\"\n" \
"Forward mail to SpamCop"

macro index \cx ":set from=\"ed-spamcop@example.net\" autoedit=no \
fast_reply=yes editor=\"/bin/true\"\n:unset \
pgp_autosign\n<tag-prefix><forward-message>submit.<your \
id>@spam.spamcop.net\n<send-message>:set autoedit=yes pgp_autosign \
fast_reply=no from=\"ed@example.net\" editor=\"/usr/bin/vim\"\n" \
"Forward mail to SpamCop"

This is very handy for bulk submissions.

automatic spamcop submissions

It's become annoying for me, I often receive mail from lists which I've not been subscribed. I'm not sure what the best thing to do is, I shouldn't be on these lists, I've never asked, so I've configured [[mailfilter]] to accommodate this with a rule that automatically calls my spamcop submission script.

The script asks spamcop to process my mail attachment which is sent using mutt. I don't create the attachment with MIME myself as I prefer to let mutt store this in my Maildir/sent/ folder rather than me storing this myself, and besides, my mail system may change in the future to use a third-party IMAP or POP mailbox (however, this is incredibly unlikely and we will probably run out of [[ipv6]] addresses first).

Currently, the processed spamcop messages arrive in my Maildir/inbox/ folder, which is ok, it's not as ideal as if the mail was arriving to a .qmail file, which would be excellent as then I'd not have to store state information about the processing.

There are three ways to process the spamcop reports:

1. process with an xfilter rule with maildrop
2. process the Maildir/inbox/ folder with find/xargs/cut/grep
3. process the mail with a .qmail file

Thanks to UNIX being an excellent OS it's very possible to cater for all of these with a single script, just called in different ways.

The script for processing this can read the URLs from STDIN (if received via .qmail), or via command line arguments if called through xargs or just user defined arguments.

For example:

.qmail-sc

|/home/user/services/scripts/spamcop_submit.pl -

Or, if executed from maildrop:

.mailfilter

if( /^From: spamcop@devnull.spamcop.net/ )
{
    xfilter "/home/user/services/scripts/spamcop_submit.pl -"
}

Or from cron:

crontab

*    *    *    *    *    find /home/user/Maildir/.inbox -type f -mtime -1 \
    | xargs cat | grep 'http://www.spamcop.net/sc?id=z' \
    | perl /home/user/services/scripts/spamcop_submit.pl

However, if you'd like to send the job directly from mutt, you can make use of the message pipe, first press |:

Pipe to command: /home/user/services/scripts/spamcop_submit.pl -

Both these scripts are available from the spamcop directory.

last delivered-to header

One thing that I wanted the flexibility with was to be able to reply to any email setting the From and mail from envelope header to the inbound header Delivered-To. The reason behind this was that for a while I sought pleasure in thwarting 419 scammers with actual human replies, just to consume their time.

So each time a 419 message arrived I'd respond using ed-<something unique for them>@s5h.net. This works well, but I needed it to happen automatically.

The resulting script looks something like this:

lastdt=$( awk '
/^$/ {
    if( lastdt == "" ) {
        print "<YOUREMAIL>@DOMAIN.net";
    }
    else {
        print lastdt;
    } 
    exit
} 
/^Delivered-To: /{
    if( lastdt == "" && $2 ~ /@/ ) {
        lastdt=$2
    }
}' )

echo "set from=\"$lastdt\"" > ~/.mutt/myfilter.out

Save it as ~/code/mutt/lastdt.sh, then just add the following macros, (alter the script name if the above doesn't suit).

macro pager,index r '<pipe-message>~/code/mutt/lastdt.sh<enter>\
<enter-command>source ~/.mutt/myfilter.out<enter><reply>'
macro pager,index g '<pipe-message>~/code/mutt/lastdt.sh<enter>\
<enter-command>source ~/.mutt/myfilter.out<enter><group-reply>'

It certainly makes my life easier, especially when replying to mail groups where I have custom .qmail rules for each list.

building on solaris

It took some tweaking to get mutt 1.5.21 to build on solaris, but I tracked the problem down (through some searching of include on linux) to missing ncurses. Since curses was on the system and 1.5.20 built ok, I simply assumed that all was still well.

Undefined                       first referenced
 symbol                             in file
key_defined                         keymap.o
ld: fatal: Symbol referencing errors. No output written to mutt

After putting ncurses in a home directory and building against that all was good.

cd ncurses-5.7
./configure --prefix=$HOME/bin/ncurses --exec-prefix=$HOME/bin/ncurses
gmake && gmake install

Reconfigure with ncurses:

cd mutt-1.5.21
./configure --enable-hcache --prefix=$HOME/bin/mutt-1.5.21 \
--exec-prefix=$HOME/bin/mutt-1.5.21 --disable-iconv \
--with-bdb=/usr/local/db-4.7.25  --with-curses=$HOME/bin/ncurses
gmake && gmake install

All done.

auto copy

One of the other things that I really like about Mutt is that it's possible to run it through screen on another host, but send a message to a copy program running on the system that you're using.

I have to do this on a regular basis since I have to regularly update tickets with email conversation data (yes it would be better to have the ticket program gobble this from a mail relay but life isn't perfect, mutt is nearly perfect though).

Firstly we need to have ssh installed and operational on the desktop. From here onwards we'll call the place mutt runs the "server" and the place where you are working from the "desktop".

Test the following, ssh to the "server" (no X forwarding), then from the server run:

ssh desktop "xeyes"

Do you get the xeyes program begin to run, do you get an error message:

Error: Can't open display:

If so, find out what DISPLAY is set to on your desktop X11 server,

$ export | grep DISPLAY
declare -x DISPLAY=":0.0"

So, then run from the ssh session to the desktop that is running on the server,

$ DISPLAY=":0.0" xeyes

and hopefully you'll get the xeyes program. If not, did you get an auth error message, if so,

$ xhost +localhost

macro

So, once the X11 problems are resolved, you'll need to add the macro:

macro index \cx "<pipe-message>ssh desktop -t 'DISPLAY=:0.0 /usr/bin/xclip -selection p'"

This should then send the message to the xclip copy program when you press ctrl-x, so no more needing to manually select and copy things. Of course if there's only certain parts of the message that you require to copy it would serve you well to grep them in the pipeline.

when mbox is better

There are some times when mbox is a better format than Maildir. However, the only time that I can think of in practical terms is for archival. In the situation of maintaining an archive of historic mail that seldom changes it is better to store in a compressed form. Sadly mutt doesn't include a method of browsing compressed tarballs of Maildir. It does however have the ability of browsing a compressed mbox file.

The following is a simple way to copy mail from Maildir to mbox:

#!/usr/bin/perl

use strict;
use warnings;

# 20131206 (ed)
# to run, change to your maildir and run
#   find .  -type f | MONTHS=1 perl /path/to/maildir_archive

my $formail = $ENV{'FORMAIL'} || "/usr/bin/formail";
my %yyyymm;
my $current_time = time;
my $months_ago = $current_time - int( ( (365.4/12) * ( $ENV{'MONTHS'} || 3 ) ) * 60 * 60 * 24 );

sub epoch_to_string {
    my $time = shift;
    if( defined( $yyyymm{$time} ) ) {
        return( $yyyymm{$time} );
    }
    my @lt = localtime( $time );
    $yyyymm{$time} = sprintf( "%.4d%.2d", $lt[5]+1900, $lt[4]+1 );
    return( $yyyymm{$time} );
}

while( my $file = <> ) {
    chomp( $file );
    next if( ! -f $file );
    my @stat_data = stat($file);
    next if( $stat_data[9] < $months_ago );

    my ($box,$location,$filename) = $file 
        =~ m%^(?:./)?([^/]+)/(cur|new|tmp)/(.*)$%;

    my $date_prefix = epoch_to_string( $stat_data[9] );

    my $flags = "";
    $flags .= "R" if( $filename =~ m%:.*S.*$% );
    $flags .= "A" if( $filename =~ m%:.*R.*$% );
    $flags .= "F" if( $filename =~ m%:.*F.*$% );

    if( ! -d $date_prefix ) {
        mkdir( $date_prefix )
            || die( "cannot create $date_prefix: $!" );
    }

    open( F, "|$formail -I \"Status: "
        . ( $location eq "cur" ? "RO" : "" ) 
        . "\" -I \"X-Status: $flags\" "
        . "| gzip -9 &gt;&gt; ${date_prefix}/${box}.gz" ) || next;

    open( FF, "<$file" ) || next;
    while( <FF> ) {
        print F $_;
    }
    close(FF);
    close(F);
    unlink( $file );
}

Use find to select the mail which you wish to archive. The script above will purge the files that are archived. Note there is no checking to see if files should get deleted ok, it simply assumes all went ok.

You may argue that the gzip compressor does not produce such great compression ratios as bzip2/xz. This is true. However, experimentation has taught me that bzip2 and xz are both incredibly slow compared to gzip. The end result may be ~80% the size of the gzip output. I would consider converting the archives to bz if and only if, space were tight.

mbsync with IMAP servers

Sometimes you need to work with IMAP servers as it may not be possible to get an account on the mailbox store where you mail is located. When you are in this situation you can either instruct mutt to talk directly to the server using the IMAP protocol, cache the headers and bodys and accept a minor delay when using mutt, or you can do a bidirectional sync with the IMAP server and work with a local Maildir. I prefer the latter but will document both solutions.

working with IMAP servers

In this situation it is important to try and cache as much as you possibly can. There will be busy parts of the day then the IMAP servers respond less immediately than others. This can lead to frustration and turn you crazy.

Here are the vital parts of a muttrc to get you going with IMAP servers:

set from="you@example.com"
set realname="Your Name"
set imap_user="you"
set folder="imaps://imap.example.com:993"

mailboxes +INBOX \
    +Sent \
    +Trash \
    +Junk \
    +Drafts

set spoolfile="+INBOX"
set record="=Sent"
set postponed="=Drafts"
set trash="=Trash"
set mail_check=10
set smtp_url="smtp://smtp.example.com:25"
set folder_format="%2C %t %N %d %f”

set header_cache=~/.mutt/headers/
set message_cachedir=~/.mutt/body/

set edit_headers
set mime_forward=yes

Notice the two cache instructions there, you'll need mutt 1.5.21 or better to make use of these, but they're really worth having.

This will get you going, and at a reasonable pace too. However, in my humble opinion, it's not enough, you should really be working on mail locally, even if it's really on an IMAP server. mbsync fills this gap very well.

Just apt-get install isync to retrieve the mbsync program.

Here is an example .mbsync configuration.

SyncState       *
Create          Both

IMAPAccount you-account
        CertificateFile ~/work/imap.example.com.crt
        Host            imap.example.com
        User            you
        Pass            "your-secret-password"
        UseIMAPS        yes
        RequireSSL      yes

IMAPStore you-remote
        Account eneville-account

MaildirStore you-local
        Path    ~/Maildir/
        Inbox   ~/Maildir/inbox

Channel you-inbox
        Master          ":you-remote:INBOX"
        Slave           ":you-local:inbox"
        Patterns        *

Channel you-sent
        Master  ":you-remote:Sent"
        Slave   ":you-local:sent"

Channel you-trash
        Master  ":you-remote:Trash"
        Slave   ":you-local:trash"

Channel you-drafts
        Master  ":you-remote:drafts"
        Slave   ":you-local:drafts"

Group you
        Channel you-inbox
        Channel you-sent
        Channel you-trash
        Channel you-drafts

Just run this from a cron job, every minute should be fine.

I've got the following script to do this and store the progress logs in a gzip file, you can roll your own script if you wish.

#!/bin/sh

D=`/bin/date --iso-8601=date` || exit 1
U=`/usr/bin/whoami` || exit 1

F=`mktemp` || exit 1
F2=`mktemp` || exit 1

/bin/date --iso-8601=sec > "$F"

/usr/bin/mbsync -a 2>&1 >>"$F"

gzip -c -9 "$F" >> "$F2"
cat "$F2" >> /var/tmp/$D.mbsync.$U.gz

rm "$F" "$F2"

Here is the corresponding muttrc to go with this.

set from="you@example.com"
set realname="Your Name"
set mbox_type=Maildir
source ~/.mutt/aliases

set folder="~/Maildir"
mailboxes +inbox \
        +drafts \
        +inbox \
        +sent

set spoolfile="+inbox"
set record="=Sent"
set postponed="=Drafts"
set trash="=Trash"
set mail_check=1

set smtp_url="smtp://smtp.example.com:25"

set folder_format="%2C %t %N %d %f”

set header_cache=~/.mutt/headers/
set message_cachedir=~/.mutt/body/
set edit_headers
set mime_forward=yes
set pager_index_lines=5
set editor='/usr/bin/vim -c "set spell spelllang=en" -c "set tw=72" -c "set
filetype=mail" -c "syntax on" -c "set fo+=aw"'

This is literally all you're going to need to get going, this syncs in both directions.