Using Monaco in an xterm

If you split your work between a mac and linux a lot, you may wish to try to have the same terminal font across the two. I like Monaco that comes by default on the Mac, and today I figured out how to use it on my Linux box at work too. (Disclaimer: you might be violating some kind of usage license on the font. IANAL)
It’s actually quite easy.

  1. Copy Monaco.dfont from /System/Library/Fonts on your Mac to your Linux box in your ~/.fonts directory
  2. run fc-cache to reload the fontconfig cache
  3. run xterm -fa ‘Monaco:pixelsize=10:antialias=no’. The antialias bit forces fontconfig to use the nice looking embedded bitmaps instead of the crappy anti-aliased version at that size.

Et voila. An xterm that looks mostly like Terminal.app.
Unfortunately, bold character’s don’t seem to work. Don’t know why. xterm seems to understand how to doublestrike server-side fonts to fake the bold effect, but maybe that logic isn’t replicated on the Xft rendering path?

No locking = No nothing

So it turns out all kinds of stuff fails when your locking calls fail.
Dovecot can’t create indexes of your mailboxes. It also spits out a IMAP warning about it, and mutt pauses a second to display this warning.
mutt can’t create its header cache without it, making the use of IMAP with large mailboxes pretty unbearable.
I’ve made some changes so that now the things that require locking happen on the local file system, and things seem to be happy now.
BTW, mutt is way faster then wanderlust. I guess I can just get used to vim again.

Is that your final answer?

After a long period of trial and error, I think I’ve finally chanced upon the best email solution for work. It turns out that, the solution follows the principle of using the best component for each task, and building a solution out of all the components — the so-called UNIX way.
Anyways, here’s what it breaks down to:
Exchange server => fetchmail => procmail => Maildir => dovecot ==(imap)==> imap clients (Apple mail.app, wanderlust, thunderbird, etc.)
By component:

  • Exchange server: Well, there’s pretty much no choice here.. this is what our IT forces on us.
  • fetchmail + procmail: Fetchmail is a really nice and simple utility. While not all that functional in itself, it is very good at being the glue between a mail server and a local delivery agent. As for procmail, it’s filtering ability is much more expressive than outlooks, and lets leave it at that.
  • Maildir + dovecot: Maildir seems in general better than mbox. I like the fact that it works over nfs, and I don’t have to worry about locking. The problem with maildir is, when you get lots of messages, it can get really bogged down. Listing a directory over NFS with 20K messages can even take a while. This is where dovecot comes in.. it has a very good caching and indexing solution that essentially overcomes this performance issue of Maildir over (relatively) slow NFS.
  • IMAP: IMAP is the protocol that was built for remote mail-store access. As such, it has all the useful primitives and semantics to make this process efficient. It also allows concurrent access from multiple clients, which was a hard requirement for a new mail setup

It turns out that using Maildir + dovecot + imap client can actually be faster than a bad implementation of a Maildir client trying to access the directory it self. For example, Evolution would take ages to check that it’s own index files were consistent with each Maildir’s contents. kmail was somewhat better, but still seemed flaky. Wanderlust took forever to build its ‘msgdb’ the first time, but subsequent updates were reasonably fast. Through IMAP, since all the clients don’t have to worry about the actual files themselves, they’re all very fast.
The real beauty of the situation is that, each component is tied together using standard interfaces. POP3 to pull off of exchange, a standard MDA protocol for fetchmail to procmail, Maildir for procmail to dovecot, and imap for dovecot to mail clients.
And Phil, Ramesh, and Lisa, thanks for listening to my random rants on the subject.
The last step is to package it as a server appliance so that others at VMware can also use it easily.

Outlook sucks, wanderlust is back in, for now..

So I spent a whole bunch of time writing a VBA script to do mew-style multi-folder re-filing of messages. It works, after a lot of headache and MSDN browsing, but it’s still un-ideal.
The main problem is that, each time you run the damn thing, you get multiple instances of the outlook security warning dialog popping up, indicating that a program (which I wrote myself, nonetheless) is trying to access my mailbox. I got the “Express ClickYes” software which automatically clicks yes to those dialogs when they pop up, but its still visually jarring.
I’ll post the script here soon, just in case anyone wants to see it.
But in other developments.. I tried looking at wanderlust again.. and it turns out it can sort the message order in IMAP mailboxes just as well as it can in local mailboxes. That means I dont get the stupid message ordering problem like I get in mew. Now if only wandlerlust had multi-target refile.. I would be done with this whole mess.. but alas.
My new current plan is to set up a fetchmail + procmail setup, where I can use procmail to do the filtering, and then just use wl to read and write. The only problem with this is now the folders won’t show up in my webmail.. but I might have a secret plan for that.. involving my good old friends courier-imap and IlohaMail.

Is XEmacs dead, or dying?

xemacs related mailing lists seem to mostly be filled with spam.
Feature-wise, it no longer has many advantages over emacs21. It still has a bit more guification, but that seems minor. Gnu emacs21 has all the X-y stuff (variable width fonts, embedded images in buffers, updated look)
Every email post, webpage, or piece of writing that tries to argue for xemacs for gnu emacs is from 2000. Even the ‘updates’ page on xemacs.org is dated 2001.
The Mew author says, while mew may work on xemacs21, it is not officially supported.
Gnu emacs’ fill-paragraph seems smarter than xemacs’.
It seems like a lot of people still stick to xemacs.. but is there any reason besides habit and intertia? Is it still better? does it still have a future?
update: Apparently wl doesn’t even run in xemacs at all. It dies when it tries to connect. Oh well, I guess that’s the end of that.

My first emacs lisp function

So mew has special variable call mew-refile-guess-alist which lets you control how ‘refiling’ works.. i.e. how to guess the target folder of the ‘refile’ operation.
Mew seems to, in general, offer a different way to read mail that comes from many lists… it works better if you have them all come to your inbox, read them there, and then use the refile intelligence to sort them out to folders after the fact. They’re like filters except that you run them after you read the mail.
The way you specify a rule for refiling looks like this:

(setq mew-refile-guess-alist
      '(  ("Sender:"
           ("some_address@some_domain.com" . "%some/folder)))

Unfortunately, at VMware, our mailing list setup does some funny things:

  • If an e-mail is sent to a list, but your address is also in the To: or Cc: fields, then the mail comes directly to you.
  • If an e-mail is sent to a list, but your address is not included in To: or Cc:, then the email comes to you but the Sender: field contains information about the group.

Given this situation, for each mailing list, I have to add three entries like the one above, one for each of the fields. So instead of doing this for every list I’m subscribed to, instead I wrote a function:

(defun mymew-add-vmware-list-rules (liststring)
  "Add rules to filter lists Identified by liststring to mew-refile-guest-alist"
  (setq mew-refile-guess-alist
        (cons (list "To:"
               (cons (concat liststring "@vmware.com")
                     (list (cons t
                                 (concat "%lists/" liststring)))))
              mew-refile-guess-alist))
  (setq mew-refile-guess-alist
        (cons (list "Cc:"
               (cons (concat liststring "@vmware.com")
                     (list (cons t
                                 (concat "%lists/" liststring)))))
              mew-refile-guess-alist))
  (setq mew-refile-guess-alist
        (cons (list "Sender:"
               (cons (concat liststring "-bounces@vmware.com")
                     (list (cons t
                                 (concat "%lists/" liststring)))))
              mew-refile-guess-alist))
  (message (concat "Added rules for list: " liststring))
)

This function takes a single argument, the email alias that identifies the list, and adds three entries to mew-refile-guess-alist, one for each type of filter. The embedded (cons t foo) is a way to tell mew that if something matches, the specified folder should be added to a list of suggested folders, allowing several rules to match simultaneously. This allows, for example, an email that was sent to two lists to get filed such that a copy ends up in each of the folders for each list.
I’m sure there’s tons of better, more concise ways to write this, but hey, it’s a start.
Argh. On an unrelated note, WordPress screws up double quotes (“) in pre regions. You have to say " instead. Lame.

Argh

Why is there never a perfect piece of software.
Wanderlust almost all of what I want, but it has several annoying bugs/misfeatures:

  • It’s single threaded, so if it get’s stuck during an imap sync, the whole damn thing get’s stuck, the whole emacs session is stuck
  • It doesn’t have any way to tell you that there’s a hidden unread message under a collapsed thread
  • It get’s coloring wrong when you’re doing an inline reply.. it seems to get the first block of your text right, but then every other

But then, mew isn’t perfect either:

  • It’s very conservative about threading, so that it misses some threads sometimes
  • You can’t look at imap folders with spaces in the names.. thankfully they’re all of the for “%Public Folders/…” so I don’t care too much
  • Why can’t I highlight unread messages? Isn’t that like a super-basic feature?

And neither can do automatic mail checking too well. Is that so hard to do?
Update
Doh! I know, I know… RTFM. Turns out mew has biff-like functionality and it actually works better than wl’s. The appropriate code being:

(setq mew-use-biff t)
(setq mew-biff-interval 3)

The second bit is to set the check interval (in minutes).
I wish there was a hook into it so that I can make it more obvious (maybe hook it up to growl in the future).

A bit better

For some reason ubuntu had a really old version of wanderlust (2.10 series, which is more than two years old). I guess no one updated the debian repository until very recently, so when ubuntu 5.10 came out, it was already too late to get included. Thankfully, the ubuntu universe pool has a .deb for the newest 2.14 series (which was still released 10 months ago!).
They seem to have fixed a number of bugs, including ldap getting stuck sometimes. The LDAP integration in wl is nicer compared to mew, because it doesn’t require a separate module, and TAB looks up LDAP automatically, where as in mew, you’d have to do a separate sequence for the first lookup.
I’ve also figured out how to turn on autofill mode if anyone’s curious:

(add-hook 'wl-draft-mode-hook
          '(lambda () (auto-fill-mode 1)
             ))

My explanation for what this does (according to my simple, yet recently slightly improved understanding of emacs lisp) is, it calls the add-hook function, to add a function to the wl-draft-mode-hook (which is defined by wanderlust). The quoted list beginning with lambda is like an anonymous function (I’m not sure why the first empty list is necessary), but the remaining items in the lambda list get evaluated one after another, in this case (auto-fill-mode 1).
Update
Though I hardly ever use any Japanese at work, I still spent some time trying to figure out if I could type Japanese emails using wanderlust. Emacs has a surprisingly good framework for input methods, and many of the opensource Japanese input methods have emacs lisp plugins for this framework (the package anthy-el in debian/ubuntu for example) So once I got that going, using the emacs with X11 to type in Japanese was no problem.
Through the terminal, was a whole ‘nother ballgame though. Apparently emacs’ support for display CJK characters in a utf-8 terminal is not complete in the 21.x series (though I’ve read that it’s been improved in the 22.x development series). So the solution is to install the mule-ucs emacs lisp package, and then add the following magic to your .emacs file:

(require 'un-define)
;; character encoding
(set-language-environment "UTF-8")
(set-terminal-coding-system 'utf-8)
(set-keyboard-coding-system 'utf-8)
(prefer-coding-system 'utf-8)
(setq erc-default-coding-system 'utf-8)

So that’s fine, and then to get this to work through Terminal.app on OSX, there’s a bit more magic, namely, in Terminal.app’s Window Settings, under Display, you need to check both of the options that begin with “Wide Glyphs …” and also make sure your encoding is set to UTF-8.
Now that all works fine and dandy. I can use anthy-el to use the emacs mechanism to input Japaense, even when I access through Terminal.app.
The only remaining problem is that the client-side mac OS X input method ‘kotoeri’ doesn’t work. If I hit option-space and switch it to Japanese and start typing, I do get the pre-edit buffer, but as soon as I try to commit, emacs chokes. I guess it can’t handle utf-8 coming in as input, though you would think that the (set-keyboard-coding-system) command would make it handle exactly that case.
Oh well, enough for now I guess.

My ideal work setup (almost)

Nerd alert. You’ve been warned.
So I’m back on emacs. I like emacs’ perforce integration better than I do vim’s.
I’ve also set up “mew” to read my imap mail. It works reasonably well, but one annoying limitation is that it can’t view imap folders with spaces in them!? But I’ve gotten mew to work with mew-ldap as well as the namazu indexing engine, so I can do some interesting stuff with it.
However, I’d still like to get wanderlust working if I can. My main issues with it right now are:

  1. When I do ldap queries, sometimes I get a ‘variable binding depth exceeds blah blah’ error. It doesn’t go away until I restart emacs.
  2. It complains to me about some IMAP folder not existing and asks me to create it, but then fails.
  3. It’s buggy. Mew is simpler and so seems to work better

I’m a little concerned though, since mew seems to move at a slower pace. I tried mutt for a bit, but then gave up right away, since it doesn’t seem to be able to cache IMAP folder information (and I did try using header_cache)
There’s also a few things I need to figure out how to do in emacs:

  • Do different things in .emacs based on command line arguments.. I’ve seen people use things like “–mode foo”
  • Learn how to make it automatically load python-mode for SConsctruct and SConscript and *.sc files
  • Learn to make it ignore my custom-set-faces command when its in terminal mode