KDE/KApplication problems on Opensync
When trying to make a KIO plugin for Opensync, I (Eduardo Habkost) have hit some problems related to the behaviour of KApplication. Armin and me have discussed this problem. Part of the log is below.
Note: Refactoring of this page is welcome. Please feel free to change it to a structured text, instead of an IRC discussion.
The problem
10:43:29 <boto> BTW, I've stuck on a problem that may be related on the usage of opensync on kde
10:44:08 <abauer> you are stuck or you struck a problem?
10:44:10 <boto> I was going to write a KIO plugin, to be able to synchronize file-sync with an
arbitrary location
10:44:22 <abauer> ok
10:44:57 <boto> better: I was stuck on a problem when working on the plugin
10:45:41 <abauer> ok. what exactly was it?
10:45:43 <boto> according to the documentation, the Qt QApplication class (and hence KApplication)
is not supposed to work outside the main thread for the application (ouch)
10:46:06 <boto> it is related to the problem of creating a KApplication object twice, but it is worse
10:46:55 <boto> my plugin simply wasn't handling the KIO jobs, even if I create a KApplication object
and call its exec() method
10:47:47 <boto> then I've noticed that I was not supposed to create a KApplication() outside the main
thread. I took the same code from the plugin and called on a standalone program, and it worked.
But it isn't working on the member thread
10:48:26 <abauer> ok.
10:48:38 <boto> I am trying to find a good solution to it
The problem is similar to problems found when trying to implement a Evolution 1.4 plugin: some libraries only work on the main thread.
Possible Solution(s)
Basically, the only solution I can see, that would work on both kde and non-kde applications is being able to run the plugin code on a separated process, if necessary.
In short, proposed steps to a complete solution:
- Make plugin able to use an existing KApplication object, if there is already one (after this step, the plugin would
only work if opensync is being used from a KDE application)
- Make plugin able to run on a separated process (not necessarily with a transparent API) (after this step, the plugin would work even on non-KDE applications)
- Make plugin API transparent to both plugin modes (separated process or not)
Ticket #90 was opened for this task.
10:49:11 <abauer> i see 3 solutions:
10:49:24 <boto> It probably won't cause problems if I use libopensync on a KDE application,
but I really would like to make it work even from a non-kde app
10:49:31 <abauer> 1. fix these programs / libraries
10:49:44 <abauer> 2. let them run on the opensync main thread
10:50:13 <abauer> 3. write a generic plugin that works over IPC and use a new process for
these plugins
10:50:56 <boto> the problem of (2) is that opensync would be very intrusive, if we require that
the user let it run stuff on the main thread. e.g. a glib-based application would need
to allow KApplication handle its events on its main thread
10:52:23 <boto> I was thinking on doing (3), but the Cornelius' message make me remember that we
wouldn't want a new process if we are a kde application (althought it would work, but
wouldn't be so efficient)
10:54:10 <abauer> well maybe we need 3, but it has to be transparent and configurable
10:54:46 <abauer> so that you can say on a plugin configuration osync_memer_enable_process_separation
10:55:06 <boto> then, we could make the new process be started and used transparently only if we don't
have a already-available KApplication object
10:55:57 <abauer> something like this
10:56:04 <abauer> but i dont know how much work this is
10:56:16 <abauer> since we have to marshal EVERYTHING
10:56:42 <abauer> hm....
10:56:47 <boto> implementing it transparently seems to be an interesting task :)
10:56:51 <abauer> maybe we could do shared memory
10:57:31 <boto> shared memory wouldn't keep the pointers working, as it may live on different virtual addresses
10:58:35 <boto> we don't have a lot of stuff to marshall, just the incoming/outgoing changes
10:59:36 <boto> we may start implementing it non-transparently, and then do some kind of "subclassing",
to make it work transparently
11:01:35 <boto> i.e. make OSyncContext an "abstract class", and have two "subclasses" OSyncLocalContext and
OSyncIPCContext, and implement report_change() accordingly on them
11:04:47 <abauer> boto: why wouldnt shared memory work exactly?
11:07:02 <boto> unless we have some mechanism to guarantee that the objects will live on the same
virtual addresses, the pointers on the "other side" would see invalid pointers
11:08:01 <abauer> uhh sorry but i dont understand, what do you mean with virtual adresses?
11:08:08 <abauer> opensync is c code
11:09:35 <boto> I mean, when we use shared memory, not necessarily the shared block of memory will be seen as
living on the same memory address by the two processes
11:11:03 <boto> so, if our struct has a pointer to another object, the pointer wouldn't necessarily be valid
on the "other side"
11:15:00 <boto> and we need to make sure all necessary objects will be accessible using shared memory.
Then we will have the same amount of work to make it work that we would have when marshalling
the objects, won't we?
11:21:21 <abauer> hmmm
11:21:45 <abauer> are you sure that pointers dont work with shared mem? :)
11:23:08 <boto> unless we have a mechanism to make sure the segment will be mapped to the same virtual
address on both sides
11:23:15 <boto> there are two (possible) different problems:
11:23:28 <boto> - make sure the objects will be on the same address on both sides (the virtual address)
11:24:00 <boto> - make sure all objects pointed by our objects will be shared (I think it requires more
work, and I think it require even more work than marshalling the objects)
11:25:13 <abauer> hmm :)
11:25:25 <abauer> but the question is what to marshal
11:25:34 <abauer> since we need info from different objects
11:25:44 <abauer> like slow-sync etc
11:25:48 <boto> the first problem may be solved (man shmop; SHM_REMAP flag), but we still have the
second problem
11:27:36 <boto> I think that the information we need to be available is not so much. Just the change data
being received/sent to the plugin would suffice for all cases I can see
11:30:13 <boto> starting from a non-transparent implementation: we may just implement IPC versions of the
API functions we want to make available, and give the response to these functions using the
message passing mechanism
11:30:19 <boto> I see two approaches here:
11:30:50 <boto> - marshall a lot of objects and then use the same opensync API functions (this would
require lot of work to marshall, it would not be trivial)
11:32:18 <boto> - make IPC versions of the API functions, that send the parameters and get the return values
from the other side. Considering that most (or all) of the API for plugins involve send/recv of changes,
or more primitive data (like getting/setting a boolean flag for a member), I think it would be easier
11:34:45 <boto> later, if we want to make it work transparently, we may just make the normal opensync
API functions "virtual" (i.e. call the right function (either the IPC or the normal version) accordingly)
11:51:40 <abauer> ok. sounds good :)
11:52:18 <abauer> but i dont know if making this stuff virtual would introduce too much overhead...
11:52:56 <abauer> so i would propose this:
11:53:07 <abauer> - marshal the relevant data from the objects we need
11:53:50 <abauer> - unmarshal it on the plugin side and create "fake" objects (objects that only
have the right pointers and the relveant data set) and pass these fake objects to the plugin
11:53:59 <abauer> but i dont know how much work this would be
11:55:01 <boto> we can evaluate the possibility of making it work transparently later. First we need
to make it work with separated IPC implementation of the functions
11:55:53 <abauer> ok
11:56:01 <boto> the question is: at which time we would marshall/unmarshall the objects? this is the main
difference of the two approaches above
11:56:04 <abauer> so another topic:
11:57:16 <abauer> well: opensync send get_changes request -> ipc plugin marshal request as well as
member data etc -> makes ipc call -> other process unmarshalls data -> creates fake objects -> calls
real plugin function
-- EduardoHabkost?, 16 Jun 2005, 11h13 (initial version)
