Warning, /pim/trojita/docs/masters/mobile-imap.tex is written in an unsupported language. File is not indexed.

 % vim: spelllang=en spell textwidth=120
 \documentclass[trojita]{subfiles}
 
 \begin{document}
 
 \chapter{The Mobile IMAP}
 \label{sec:mobile-imap}
 
 Many of the existing IMAP extensions discussed in \secref{sec:imap-extensions} have the potential of improving the
 client's operation tremendously.  At the same time, experience has shown that there is a certain chicken-and-egg problem
 with new proposals where server vendors are not willing to invest their time into promising extensions which no client
 supports yet and clients are not interested in implementing extensions which they cannot test for interoperability.  In
 this chapter, I am trying to provide a concise summary of individual merit of these extensions.
 
 \section{The Lemonade Profile}
 \label{sec:lemonade-comparison}
 
 The Lemonade profile, as defined in RFC 4550 \cite{rfc4550} in 2006 and later updated through RFC 5550 \cite{rfc5550}
 during 2009, provides a list of extensions considered ``critical'' for any mobile IMAP e-mail client.  The set of
 mandatory extensions is rather big, though, and to the best of my knowledge, there is {\em no} server on the market
 implementing all of the compulsory features.  One might therefore wonder what were the reasons for this lack of general
 availability of the Lemonade extension family.
 
 \subsection{Cross-Service Requirements}
 
 One unique feature of Lemonade is the possibility to {\em forward messages without their prior download}.  The three
 ESMTP~\cite{rfc5321} and IMAP extensions, often referred to as the {\em Lemonade trio}, namely the {\tt CATENATE}, {\tt
 URLAUTH} and {\tt BURL}, allow the clients to compose a message using existing parts available from the IMAP mail store,
 provide a way of generating single-purpose ``pawn tickets'' for making the composed messages available to the submission
 server, and replacing the {\tt DATA} SMTP command with a way of downloading the message from the IMAP server,
 respectively.  This feature prevents having to transfer potentially huge data over the network three times --- once when
 the users wants to read it, second time when the message is saved to the sent folder, and finally when delivering via
 SMTP.
 
 Unfortunately, a big problem with said approach is the fact that it mandates collaboration across different services ---
 an explicit trust path between the IMAP and ESMTP servers have to be set up, which is a process prone to errors
 \cite{qmf-fastmail-burl-bug}.  This matter is also complicated by the fact that no open source MTA~\footnote{Mail
 Transfer Agent, typically an SMTP or ESMTP server} ships with official support for {\tt BURL}.~\footnote{Unofficial
 patches exists for Postfix dating back to 2010 \cite{apple-postfix-burl}, but they have not been integrated into the
 mainline version as of July 2012 (the {\tt postfix-2.10-20120715.tar.gz} development snapshot.}  Situation is better on
 the IMAP server front with Cyrus supporting the {\tt URLAUTH} and {\tt CATENATE} extensions out-of-box with Dovecot's
 support scheduled for its upcoming 2.2 release \cite{imap-server-extension-matrix}.
 
 \subsection{Complicated Extensions}
 
 Some of the extensions whose support is mandated by the Lemonade proposal seems to be notoriously hard for the server
 vendors to implement.
 
 A perfect example is the {\tt CONTEXT=SORT} extension \cite{rfc5267}.  As a client developer, I recognize its extreme
 usefulness and appreciate its design.  Availability of such an extension would make it extremely easy to implement
 live-updated sorting in my Trojitá (and Trojitá {\em does} make use of the sort context extension).  That said, given
 that no IMAP server which I am aware of announces its availability, clients have to deal with the status quo in the
 meanwhile.
 
 The {\tt CONVERT} extension \cite{rfc5259} belongs to a similar category --- the features it offers, like the
 server-side downscaling of JPEG images, would be {\em very} handy on a cell phone, yet no IMAP server known to the
 author includes that functionality.
 
 Both of these RFCs were published four and five years ago, respectively, and were designed by engineers working for an
 IMAP server vendor.  One cannot therefore dismiss them altogether as a product of people not having any say in the
 server development.  My opinion is that the allocation of engineering resources required for shipping a particular
 feature in a finished product is based on another criteria than the research activity.
 
 \section{State of Other Client Implementations}
 
 To obtain a better understanding on how the existing solutions available on the market today use IMAP, this section
 takes a look at some of the most popular solutions.
 
 \subsection{Apple iOS}
 
 Apple's devices generally ship with a decent implementation of their IMAP stack, an evaluation shared by independent
 researchers \cite{isode-iphone4}.  The list of extensions supported by the application includes {\tt CONDSTORE} and {\tt
 ESEARCH} for improved mailbox synchronization, {\tt COMPRESS} for transparent deflate compression and {\tt BURL} for the
 forward-without download.
 
 It is, however, surprising that their support of extensions aimed at making mailbox resynchronization more efficient
 does not include the {\tt QRESYNC} extension \cite{rfc5162}, especially given that its implementation does not impose
 much in terms of additional requirements on top the already-supported {\tt CONDSTORE}.
 
 The iOS also notably does not use the {\tt IDLE} command at all.  The reason, according to a message allegedly sent by
 Steve Jobs \cite{jobs-ios-idle}, is that is is {\em ``a power hungry standard''}.  Systematic measurements
 \cite{wcdma-energy} \cite{cridland-fach-dch-measurements} and experience alike~\footnote{Mark Crispin's famous {\em ``I
 have built on-demand networks which shut down until signaled back on, and then happily resumed all the active TCP
 sessions even though the "network connection" had been powered off for days.''} \cite{crispin-no-ifup}.} shows, however,
 that the mere act of having a TCP connection open with an occasional keep alive ``pings'' being transfered have no
 significant impact on battery life on other platforms.
 
 \subsection{Android's Native E-mail Client}
 
 There is not much to be said about Android's native client's IMAP performance --- the stock client does not offer push
 notification through {\tt IDLE} \cite{android-idle} and the list of extension identifiers referenced from the
 application's source~\footnote{File \path{src/com/android/email/mail/store/ImapConnection.java} from Android's
 \path{platform/packages/apps/Email} repository as of the {\tt android-4.1.1\_r3-35-g01c55fd} revision.} only references
 the {\tt NAMESPACE}, {\tt UIDPLUS} and {\tt STARTTLS} capabilities.  None of the extensions which try to improve
 synchronization performance (the {\tt ESEARCH}, {\tt CONDSTORE} and {\tt QRESYNC}) are available.  No provisions for
 Lemonade's family are present at all.  The {\tt LITERAL+}, an extension which takes literally no effort for clients to
 support and removes one network round trip when uploading messages, is not supported.
 
 Further analysis shows that the code is blocking and incapable of issuing or processing requests in parallel.  These
 observations are consistent with what users generally describe as a ``slow'' experience.  This might not come surprising
 given that Google would likely prefer its users to choose Google's own e-mail offering, the GMail, over various private
 IMAP accounts for business reasons.
 
 \subsection{Android's K-9 Mail}
 
 The Android's K-9 mail~\cite{k9mail} is a fork of the original e-mail application from Google.  The developers have
 managed to add support for two extensions, namely the {\tt IDLE} and the {\tt COMPRESS=DEFLATE}.  More advanced features
 like the {\tt ESEARCH} / {\tt CONDSTORE} / {\tt QRESYNC} are however still missing.
 
 Furthermore, comments like {\em ``TODO Need to start keeping track of UIDVALIDITY''}~\footnote{File
 \path{src/com/fsck/k9/mail/store/ImapStore.java}, line 103 as of the Git revision {\tt 3.512-1249-g5ce0e19} of the K-9's
 source code repository.} might make one feel nervous about the safety of the data being accessed.
 
 \subsection{Modest / Tinymail}
 
 The several years old Nokia N900 shipped with a mail client called Modest, an application based on top of the Tinymail
 framework~\cite{tinymail}.  The underlying library supports an efficient mailbox synchronization through {\tt QRESYNC}
 and the sources contain a reference to the {\tt CONVERT} extension --- however, this functionality appears to be limited
 and in an early state.  The ``Lemonade trio'' for forwarding without download is not supported.
 
 The code is written using the synchronous idioms. No support for command pipelining is present.
 
 \subsection{Nokia's Qt Messaging Framework}
 
 Nokia's Qt Messaging Framework~\cite{qmf} powers the e-mail functionality in the MeeGo Harmattan line of phones, the N9
 and the N950.  Support for extensions appears to be rich --- the Lemonade trio is supported in its
 entirety,~\footnote{An interesting detail is that the {\tt BURL} extension is disabled if the account is configured to
 use IMAP over an SSL port instead of the {\tt STARTTLS}; this is caused by a deficiency in the IMAP-URL standard
 \cite{rfc5092}.} as well as are the {\tt COMPRESS=DEFLATE}, {\tt BINARY}, {\tt IDLE}, {\tt CHILDREN} and many others.
 
 The underlying IMAP library exhibits regular ``batched synchronization'' mode of operation; connections to mailboxes are
 typically not left alone for long, but the code makes aggressive use of the {\tt QRESYNC} extension to benefit from
 extremely cheap mailbox synchronization.  Code is written with memory-constrained devices in mind, care is taken to
 prevent data copying if possible.  The general behavior focuses on making the specified subset of messages always
 available on device --- there is no support for server-side threading or sorting, but some form of a server-side search
 {\em is} available.  Live updates through {\tt CONTEXT=SEARCH} are not employed.  Support for the {\tt LIST-STATUS} is
 not available.
 
 An interesting extension which is supported by the QMF is Google's {\tt XLIST} \cite{gmail-xlist} extension.  However,
 the code~\footnote{File \path{src/plugins/messageservices/imap/imapprotocol.cpp} as of QMF's Git revision {\tt
 2011W26\_2-263-gec26531}.} appears to only use the {\tt \\Inbox} mailbox flag for forcing case-insensitive mailbox
 comparison, a mechanism already mandated by the baseline IMAP protocol \cite[p. 17]{rfc3501}.
 
 All in all, the Qt Messaging Framework is a promising library with support for other protocols besides IMAP.  Its IMAP
 implementation feels solid and is reasonably well covered by the unit tests.
 
 \subsection{Trojitá}
 
 Trojitá~\cite{trojita-website} is an advanced IMAP client which includes support for many different extensions from the
 basic ones like {\tt IDLE}, {\tt LITERAL+} or the {\tt ID} extension to the complex ones like the Lemonade trio or the
 {\tt ESEARCH} / {\tt CONDSTORE} / {\tt QRESYNC} triplet.  Trojitá can scale up to company-wide ERP-level e-mail
 processing (Appendix \ref{sec:xtuple}, p. \pageref{sec:xtuple}) while still using the same code base of
 the underlying IMAP library as the desktop and cell phone version.
 
 \section{Evaluating Extensions}
 
 As the previous section shows, the level of support for various extensions and their real-world deployment varies
 wildly.  Many today's clients are using only a limited subset of features which are generally available, leaving
 opportunities for new applications entering the market to offer smoother, more efficient user experience.
 
 Previous efforts aimed at standardizing a set of extensions to serve as a ``mobile profile'' of IMAP have not delivered
 a universally acceptable result.  Lemonade, a specification universally considered to be {\em the} specification to go
 when evaluating clients and servers alike \cite{isode-lemonade}, mandates a set of extensions which --- to the best of
 my knowledge --- is not available on any single IMAP server in its entirety.  Furthermore, the clients do not even
 attempt to use these extensions, probably due to not being able to verify them for interoperability and because of
 concerns about shipping something which could very well be considered a dead code.
 
 The rest of this chapter tries to look at the available extensions through the optics of a client developer who is
 trying to push the IMAP server developers slightly, but not too much --- pushing forward is crucial in attaining a
 sustainable development, yet applying too much pressure often leads to the whole movement getting stuck on a first
 obstacle.  I make a few assumptions about the features which the client support; most notably, the client should have a
 persistent cache to store already downloaded data.  The client is also expected to serve a human user who will work with
 the application for many days across a network which might fail occasionally or even very often.
 
 This humble taxonomy is not an attempt to define a {\em level of support} --- entire working groups dedicated to this
 task have failed to deliver a universally-accepted solution despite spending many years on this goal.  Instead, it is
 intended to serve as a reference suggesting client and server developers alike what extensions might be useful.
 
 Further details about the mentioned extensions are available in chapters three (p.  \pageref{sec:imap-extensions} and
 four (p. \pageref{sec:proposed-extensions}).
 
 \subsection{The Bare Minimum}
 
 The first group of extensions lists those proposals which I consider to be a practical requirement for a well-behaving
 client to implement.  Furthermore, their implementation does not impose an overly significant burden on the servers,
 neither in terms of complicated code and therefore increased development costs, nor in the runtime overhead.
 
 \subsubsection{LITERAL+}
 
 Implementing the {\tt LITERAL+} extension actually {\em reduces} the amount of special code which the client executes
 when transmitting data using literals.  As such, there is no excuse for not implementing this extension.
 
 \subsubsection{IDLE}
 
 Entering the {\tt IDLE} mode enables the server to issue any untagged response, including the {\tt EXPUNGE} and {\tt
 VANISHED}, at any time.  Because the IMAP clients are already required to handle any response at any time, actually
 making use of this feature should not impose any extra requirements on their implementors.  The only acceptable reason
 for not using {\tt IDLE} is, in my opinion, a technical deficiency in the platform's TCP stack and the related layers.
 If the total power consumption of the {\tt IDLE} command exceed periodic polling, clients shall refrain from calling
 {\tt IDLE}.
 
 \subsubsection{ID}
 
 The {\tt ID} command provides useful diagnostics about the other end of the IMAP channel.  Similar features exist in
 other Internet protocols.  Hiding the name of one's implementation is a weak form of security measure --- existing
 efforts \cite{openemailsurvey} are able to recognize an IMAP server by its {\tt CAPABILITY} response, a set of standard
 human-readable texts and through additional criteria and it is conceivable that an alternative for clients is feasible
 as well.
 
 \subsubsection{BINARY}
 
 The {\tt BINARY} extension can delegate decoding of the content-transfer-encoding to the server side.  Given that
 servers compliant with the baseline IMAP protocol already have to include a full MIME decoder, support for the BINARY
 extension is a firm candidate for the basic level of the extension taxonomy in spite of the fact that the most widely
 deployed IMAP server does not support it in its production releases yet.
 
 \subsubsection{UIDPLUS}
 
 Without the {\tt UIDPLUS} extension, it is hard for clients to tell the UID of a message they have just appended to the
 mailbox.  On servers which support persistent storage of UIDs, support for this extension does not add any further
 requirements.
 
 \subsubsection{CHILDREN, LIST-EXTENDED and LIST-STATUS}
 
 With the {\tt CHILDREN} {\tt LIST} extension, GUI clients do not have to explicitly issue one more {\tt LIST} command
 for each mailbox shown in the GUI to tell whether said mailbox shall be drawn with a visual cue indicating its
 potentially expandable state.  The {\tt LIST-EXTENDED} and {\tt LIST-STATUS} allow for bundling of additional metadata
 with the {\tt LIST} responses and invoking the {\tt STATUS} command, respectively.
 
 \subsubsection{ESEARCH}
 
 The {\tt ESEARCH} response does not add any requirements on the IMAP servers besides the fact that they must send the
 whole response at once.  I believe that this is a reasonable trade-off considering the benefits the compact {\tt uidset}
 syntax can provide when working with large mailboxes.
 
 \subsubsection{COMPRESS=DEFLATE}
 
 Given the ubiquitous presence of the ZLib library, clients shall make sure that either the TLS compression is active or
 the {\tt COMPRESS=DEFLATE} command is issued.  IMAP is a textual protocol and as such compresses extremely well.
 
 \subsection{Useful Extensions}
 
 The second group of extensions consists of those which are very valuable for the increased efficiency of the protocol
 exchange, yet their implementation imposes a set of requirements on the server vendors, CPU resources or generally
 requires considerable effort to ``get right''.
 
 \subsubsection{CONDSTORE and QRESYNC}
 
 The {\tt CONDSTORE} extension enables clients to skip downloading of all flags upon each mailbox reconnect.  As such, it
 is highly desirable.  It is not placed in the basic category due to its implications on the server side, though.
 
 With server requirements not significantly different to the {\tt CONDSTORE}, {\tt QRESYNC} reduces the amount of data
 transferred when resynchronizing a mailbox after new messages have arrived and others have been expunged.  Because it
 allows clients to skip the UID map rebuild, significant bandwidth savings can be obtained.  As such, clients are highly
 recommended to support this extension.
 
 \subsubsection{ENABLE}
 
 The {\tt ENABLE} extension is a requirement for QRESYNC.  It does not add any additional requirements on clients or
 servers, so it shall belong to the same category.
 
 \subsubsection{MULTIAPPEND}
 
 The {\tt MULTIAPPEND} command improves performance in clients upload many messages at once --- a scenario common in
 applications which support batched import of existing e-mails.  Clients which do not need this feature can safely ignore
 this extension.  {\tt MULTIAPPEND} also mandates that the whole operation is atomic, that is, either all of the messages
 are appended, or none for them are.  For this reason, this extension is not put into the basic level.
 
 \subsubsection{SENDMAIL}
 
 Servers implementing the {\tt SENDMAIL} extension as proposed in this thesis free their clients from having to speak the
 ESMTP protocol at all, a benefit bringing along streamlined end-user configuration and reducing support calls.  However,
 due to a certain political pressure against the adoption of any extension allowing message submission via IMAP, I have
 decided to put {\tt SENDMAIL} to the middle category.
 
 \subsubsection{CATENATE}
 
 If the client can make use of the {\tt CATENATE} extension, it can be leveraged for tasks like stripping out an unwanted
 attachment from a message otherwise considered valuable.  Its usefulness increases in combination with {\tt SENDMAIL} or
 {\tt BURL} where it allows for full support for the ``forward without download'' functionality and significant bandwidth
 reduction.
 
 \subsection{The Most Advanced Extensions}
 
 The last group contains those extensions which are either something {\em special}, exotic, or generally hard to
 implement correctly.  Clients can benefit from many extensions from this group, yet one cannot expect to have them
 available on many IMAP servers due to their inherent complexity or increased amount of requirements which they put on
 the server.
 
 \subsubsection{SORT, SORT=DISPLAY and THREAD}
 
 These extensions present functionality which is very desirable from a client's point of view, yet requires the server to
 build an index of the whole mailbox contents at once.  In absence of this extension, clients wishing to offer threading
 support or those that need to present the results to their users in a particular order have no choice but to download
 additional data for every single message in the mailbox.
 
 \subsubsection{INCTHREAD, CONTEXT=SEARCH and CONTEXT=SORT}
 
 This group builds upon the functionality provided by {\tt SORT} and {\tt THREAD} by enabling live updates to the results
 the client is interested in.  In their absence, clients either have to repeat the server-side operation once again
 whenever a new message arrive, or defer to a purely client-side sorting or threading.
 
 \subsubsection{SEARCH=FUZZY}
 
 Adding support for ``fuzzy search'', this extension mandates servers to build an equivalent of a fulltext search index
 for the current mailbox, placing a significant burden on the IMAP server.
 
 \subsubsection{URLAUTH and BURL}
 
 This subgroup of extensions is known as the ``Lemonade trio''.  Together, they allow for efficient operations like
 forward-without download, drastically reducing the amount of data to send when working with big messages.  Due to the
 requirement for an explicit trust path between the IMAP server and the ESMTP submission service, clients have to
 anticipate problems when they try to utilize this functionality.
 
 \subsubsection{SPECIAL-USE and CREATE-SPECIAL-USE}
 
 A relatively new extension which allows clients to automatically obtain data about what mailbox serves a ``special
 purpose'' --- for example, containing {\em all messages} from any other mailbox, or being designated as a ``sent
 folder'' by the system or account configuration.  This could become a very useful extension, but it is too early to
 mandate its support now.
 
 \subsubsection{CONVERT}
 
 No servers that I am aware of deploy the {\tt CONVERT} extension despite the time it has been around.  No matter how
 useful I find this extension, this trait alone puts it firmly into the ``speciality'' group.
 
 \subsubsection{NOTIFY and MULTISEARCH}
 
 The {\tt NOTIFY} builds where {\tt IDLE} stopped, adding support for asynchronous notification about state of many
 mailboxes without a need of keeping concurrent connections opened.  The {\tt MULTISEARCH} command adds support for
 searches spanning several mailboxes, improving the traditional model where clients are required to ``hop'' mailboxes
 periodically.  Without much support on the server side, these extensions probably cannot be relied upon in 2012.
 
 \end{document}