Warning, /frameworks/kio/docs/design.txt is written in an unsupported language. File is not indexed.

0001 DESIGN:
0002 =======
0004 libkio uses kioslaves (separate processes) that handle a given protocol.
0005 Launching those slaves is taken care of by the kdeinit/klauncher tandem,
0006 which are notified by DBus.
0008 Connection is the most low-level class, the one that encapsulates the pipe.
0010 SlaveInterface is the main class for transferring anything to the slave
0011 and Slave, which inherits SlaveInterface, is the sub class that Job should handle.
0013 A slave inherits SlaveBase, which is the other half of SlaveInterface.
0015 The scheduling is supposed to be on a two level basis. One is in the daemon
0016 and one is in the application. The daemon one (as opposite to the holy one? :)
0017 will determine how many slaves are ok for this app to be opened and it will
0018 also assign tasks to actually existing slaves.
0019 The application will still have some kind of a scheduler, but it should be
0020 a lot simpler as it doesn't have to decide anything besides which
0021 task goes to which pool of slaves (related to the protocol/host/user/port)
0022 and move tasks around.
0023 Currently a design study to name it cool is in scheduler.cpp but in the
0024 application side. This is just to test other things like recursive jobs
0025 and signals/slots within SlaveInterface. If someone feels brave, the scheduler
0026 is yours!
0027 On a second thought: at the daemon side there is no real scheduler, but a
0028 pool of slaves. So what we need is some kind of load calculation of the
0029 scheduler in the application and load balancing in the daemon. 
0031 A third thought: Maybe the daemon can just take care of a number of 'unused'
0032 slaves. When an application needs a slave, it can request it from the daemon. 
0033 The application will get one, either from the pool of unused slaves, 
0034 or a new one will be created. This keeps things simple at the daemon level.
0035 It is up to the application to give the slaves back to the daemon.
0036 The scheduler in the application must take care not to request too many
0037 slaves and could implement priorities.
0039 Thought on usage:
0040 * Typically a single slave-type is used exclusively in one application. E.g.
0041 http slaves are used in a web-browser. POP3 slaves used in a mail program.
0043 * Sometimes a single program can have multiple roles. E.g. konqueror is
0044 both a web-browser and a file-manager. As a web-browser it primarily uses
0045 http-slaves as a file-manager file-slaves.
0047 * Selecting a link in konqueror: konqueror does a partial download of
0048 the file to check the MIME type (right??) then the application is
0049 started which downloads the complete file. In this case it should 
0050 be able to pass the slave which does the partial download from konqueror
0051 to the application where it can do the complete download.
0053 Do we need to have a hard limit on the number of slaves/host?
0054 It seems so, because some protocols are about to fail if you
0055 have two slaves running in parallel (e.g. POP3)
0056 This has to be implemented in the daemon because only at daemon
0057 level all the slaves are known. As a consequence slaves must 
0058 be returned to the daemon before connecting to another host.
0059 (Returning the slaves back to the daemon after every job is not
0060 strictly needed and only causes extra overhead) 
0062 Instead of actually returning the slave to the daemon, it could 
0063 be enough to ask 'recycling permission' from the daemon: the 
0064 application asks the daemon whether it is ok to use a slave for 
0065 another host. The daemon can then update its administration of
0066 which slave is connected to which host.
0068 The above does of course not apply to hostless protocols (like file).
0069 (They will never change host).
0071 Apart from a 'hard limit' on the number of slaves/host we can have
0072 a 'soft limit'. E.g. upon connection to a HTTP 1.1 server, the web-
0073 server tells the slave the number of parallel connections allowed.
0074 THe simplest solution seems to be to treat 'soft limits' the same
0075 as 'hard limits'. This means that the slave has to communicate the
0076 'soft limit' to the daemon.
0078 Jobs using multiple slaves.
0080 If a job needs multiple slaves in parallel (e.g. copying a file from 
0081 a web-server to a ftp-server or browsing a tar-file on a ftp-site)
0082 we must make sure to request the daemon for all slaves together since 
0083 otherwise there is a risk of deadlock. 
0085 (If two applications both need a 'pop3' and a 'ftp' slave for a single 
0086 job and only a single slave/host is allowed for pop3 and ftp, we must 
0087 prevent giving the single pop3 slave to application #1 and the single 
0088 ftp slave to application #2. Both applications will then wait till the 
0089 end of times till they get the other slave so that they can start the 
0090 job. (This is a quite unlikely situation, but nevertheless possible))
0093 File Operations:
0094 listRecursive is implemented as listDir and finding out if in the result
0095  is a directory. If there is, another listDir job is issued. As listDir
0096  is a readonly operation it fails when a directory isn't readable
0097   .. but the main job goes on and discards the error, because
0098 bIgnoreSubJobsError is true, which is what we want (David)
0100 del is implemented as listRecursive, removing all files and removing all
0101  empty directories. This basically means if one directory isn't readable
0102  we don't remove it as listRecursive didn't find it. But the del will later
0103  on try to remove it's parent directory and fail. But there are cases when
0104  it would be possible to delete the dir in chmod the dir before. On the
0105  other hand del("/") shouldn't list the whole file system and remove all
0106  user owned files just to find out it can't remove everything else (this
0107  basically means we have to take care of things we can remove before we try)
0109  ... Well, rm -rf / refuses to do anything, so we should just do the same:
0110  use a listRecursive with bIgnoreSubJobsError = false. If anything can't
0111  be removed, we just abort. (David)
0113  ... My concern was more that the fact we can list / doesn't mean we can
0114  remove it. So we shouldn't remove everything we could list without checking
0115  we can. But then the question arises how do we check whether we can remove it?
0116  (Stephan)
0118  ... I was wrong, rm -rf /, even as a user, lists everything and removes
0119  everything it can (don't try this at home!). I don't think we can do
0120  better, unless we add a protocol-dependent "canDelete(path)", which is
0121  _really_ not easy to implement, whatever protocol. (David)
0124 Lib docu
0125 ========
0127 mkdir: ...
0129 rmdir: ...
0131 chmod: ...
0133 special: ...
0135 stat: ...
0137 get is implemented as TransferJob. Clients get 'data' signals with the data.
0138 A data block of zero size indicates end of data (EOD)
0140 put is implemented as TransferJob. Clients have to connect to the 
0141 'dataReq' signal. The slave will call you when it needs your data.
0143 mimetype: ...
0145 file_copy: copies a single file, either using CMD_COPY if the slave 
0146            supports that or get & put otherwise.
0148 file_move: moves a single file, either using CMD_RENAME if the slave
0149            supports that, CMD_COPY + del otherwise, or eventually
0150            get & put & del.
0152 file_delete: delete a single file. 
0154 copy: copies a file or directory, recursively if the latter
0156 move: moves a file or directory, recursively if the latter
0158 del: deletes a file or directory, recursively if the latter
0160 Resuming
0161 --------
0162 If a .part file exists, KIO offers to resume the download.
0163 This requires negotiation between the kioslave that reads
0164 (handled by the get job) and the kioslave that writes
0165 (handled by the put job).
0167 Here's how the negotiation goes.
0168 (PJ=put-job, GJ=get-job)
0170 PJ can't resume:
0171 PJ-->app: canResume(0)  (emitted by dataReq)
0172 GJ-->app: data()
0173 PJ-->app: dataReq()
0174 app->PJ: data()
0176 PJ can resume but GJ can't resume:
0177 PJ-->app: canResume(xx)
0178 app->GJ: start job with "resume=xxx" metadata.
0179 GJ-->app: data()
0180 PJ-->app: dataReq()
0181 app->PJ: data()
0183 PJ can resume and GJ can resume:
0184 PJ-->app: canResume(xx)
0185 app->GJ: start job with "resume=xxx" metadata.
0186 GJ-->app: canResume(xx)
0187 GJ-->app: data()
0188 PJ-->app: dataReq()
0189 app->PJ: canResume(xx)
0190 app->PJ: data()
0192 So when the slave supports resume for "put" it has to check after the first 
0193 dataRequest() whether it has got a canResume() back from the app. If it did 
0194 it must resume. Otherwise it must start from 0.
0196 Protocols
0197 =========
0199 Most KIO slaves (but not all) are implementing internet protocols.
0200 In this case, the slave name matches the URI name for the protocol.
0201 A list of such URIs can be found here, as per RFC 4395:
0202 https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml