Full-text search using FTS3.
Full-text search with a contact constraint.
The global database sqlite file resulting from indexing all of mozilla.dev.apps.thunderbird is about 13M for something like 4500 messages. We’re providing FTS3 with the bodies (but not attachments!) of all the newsgroup messages and the subjects of the messages which initiate new threads. For real usage, we will need to also index the subjects of each message.
Note that the message bodies have not been processed at all by the Thunderbird/gloda code before handing them off to FTS3. So quoted messages get indexed even though it’s a lot of excess data. We’re relying on FTS3 to do all stop-words, etc. FTS3’s Porter stemming/tokenization is in use.
Arthur | 19-Aug-08 at 6:30 am | Permalink
4500 is bit low for daily usage. My mail folders have something like 2 million messages in total. I hope it’ll be still usable with that many mails.
Andrew Sutherland | 19-Aug-08 at 11:44 am | Permalink
Right, that’s not a limitation on a number of messages it can index or easily handle. The example is to provide an idea of ‘bloat’ for disk usage for FTS3, mozilla.dev.apps.thunderbird hopefully being near and dear to many hearts.
On the other hand, 2 million messages is a lot. I expect that would require extra leg-work to be able to handle that optimally.
Are a lot of those messages from mailing lists? More specifically, can they be categorized into different domains, not all of which need to be accessed all simultaneously? We could potentially segment the database along those lines.
Arthur | 19-Aug-08 at 1:11 pm | Permalink
Yes, a lot of my messages won’t ever be searched simultaneously. They’re in different folders and different mail account. The maximum of messages to search at the same time would probably be around 400′000 at the moment. That’s still quite a lot and the current subject/sender search field already takes its time. But I’m certainly looking forward to the new search features! Making the handling of mails easier is wonderful. Thank’s for the great work.
A plot of index disk usage vs. number of mails and search time vs. number of mails would certainly be interesting. Probably interesting enough for an automated regression test like the current performance measurements for FF?
david ascher - » Gearing up | 19-Aug-08 at 8:53 pm | Permalink
[...] end-users, from the mundane (a birthday field in the address book, simpler account setup) to some powerful, platform-style [...]