Sylpheed Thread View and Summary Cache Problem

Posted: 2013-03-20. Updated: 2013-05-22.

Problem

New emails in my inbox were not shown in "Thread view" in the email program Sylpheed (version 3.3.0) running on Ubuntu Linux:
$ uname -a
Linux test 3.2.0-39-generic #62-Ubuntu SMP Wed Feb 27 22:05:17 UTC 2013 i686 i686 i386 GNU/Linux
Using GTK+ 2.24.10.

For some reason the thread view did not work at all for new email in my inbox in Sylpheed version 3.3.0. Neither did it work in an older version that I tried as well. I just tested it with the newest version found online, in particular with Sylpheed 3.4.0beta2. Update: I also tested it with the newest svn version (revision 3234). It does not work there either.

As far as I can tell, the problem only affects mailboxes on the local disk (MH). When using an IMAP server the thread view works as expected.

I first saw this problem in regular email conversation, but attributed it to badly configured email clients of my correspondents.

Tests

I did some test by sending email back and forth between different email addresses always hitting reply to simulate an email conversation. Even in this simple test the emails were not displayed in a thread in my inbox.

After these tests I began looking at the source code, because I thought there was something broken with Sylpheed's thread view algorithm. Specifically how it treated the "In-Reply-To" or "References" header. This took me quite some time, but did not improve the situation. The new emails were still displayed unthreaded.

Temporary Workaround

A temporary solution turned out to be rather simple. In the Sylpheed window look at the folder view that shows the different mailboxes. Right click on one of the mailboxes that show this problem and click on "Update summary". The new emails should be properly shown in thread view now.

This is only a workaround. It requires the user to click "Update summary" every time new emails have been received. The real problem seems to be in the computation of the summary cache. It seems it is computed correctly only when invoked by clicking "Update summary", not when invoked automatically. In particular it seems to be computed correctly only when "cache_dirty: 1" is set.

You can see below that the summary cache (file ".sylpheed_cache") is recomputed when a new email in received. However, it is only recomputed correctly when invoked by "Update summary". The cache file that is produced differs.

1) Before: 

$ cd /home/test/Mail/inbox/
$ ls -lh .syl*
-rw------- 1 test test 499K Mar 20 17:57 .sylpheed_cache
-rw------- 1 test test  12K Mar 20 17:57 .sylpheed_mark
$ md5sum .syl*
fe1ef3fe0f74080ae0e67be0cec0338f  .sylpheed_cache
31732dbccfcc23bf1f017b89b7077ca7  .sylpheed_mark

2) After receiving mail. Threads are not displayed correctly. 

$ ls -lh .syl*
-rw------- 1 test test 499K Mar 20 17:59 .sylpheed_cache
-rw------- 1 test test  12K Mar 20 17:59 .sylpheed_mark
$ md5sum .syl*
9ae4212791ecbff2e8b13df8302adf37  .sylpheed_cache
4bbf6425ee40e2975cd47c095a85db7b  .sylpheed_mark

3) After clicking "Update summary". Threads are displayed correctly now. Debug output displays "cache_dirty: 1". 

$ ls -lh .syl*
-rw------- 1 test test 500K Mar 20 18:00 .sylpheed_cache
-rw------- 1 test test  12K Mar 20 18:00 .sylpheed_mark
$ md5sum .syl*
e34d54bed9be1925c00121edf9319374  .sylpheed_cache
ec82a65eca515154873cdaa9cbf90855  .sylpheed_mark

Configuration

During the tests the Sylpheed configuration was set as the following.

Original configuration

o Common preferences - Receive - Update all local folders after incorporation:	disabled (not checked)
o Common preferences - Details - Open inbox after receiving new mail:		enabled (checked)

=> Does not work, as described above.

Now I reran the tests (sending email back and forth) with the rest of the possible combinations of these configuration options.

Test configuration 1

o Common preferences - Receive - Update all local folders after incorporation:	enabled (checked)
o Common preferences - Details - Open inbox after receiving new mail:		enabled (checked)

=> Does not work, same behaviour as described above.
(As a side note: the new email in the inbox is now shown/sorted above/behind all older emails. Only after switching to a different mailbox and switching back to the inbox is the summary view of the inbox redrawn correctly. The new email is displayed at the bottom now, as expected.)

Test configuration 2

o Common preferences - Receive - Update all local folders after incorporation:	disabled (not checked)
o Common preferences - Details - Open inbox after receiving new mail:		disabled (not checked)

=> Does not work, exactly as in "Original configuration".

Test configuration 3

o Common preferences - Receive - Update all local folders after incorporation:	enabled (checked)
o Common preferences - Details - Open inbox after receiving new mail:		disabled (not checked)

=> Does not work, exactly as in "Original configuration".

Test from Scratch

The tests above have been carried out on a live installation (many mailboxes, filters configured, different accounts, POP3 and IMAP, other custom configuration). Now I wanted to see if this custom configuration is responsible for the wrong behaviour.

Now I started from scratch (no mailboxes, no configuration at all). I am using Sylpheed 3.3.0 here. Update: I also tested it with the newest svn version (revision 3234). It does not work there either.

Different Mail Clients

Now I wanted to test if the error depends on the mail client that the other end uses.

Client1: Sylpheed, versions as given above.
Client2: mail client as given in the list below.

Does not work:

Back to writings.