SyncEvolution 1.3.99.3 released

So far the development cycle for SyncEvolution 1.4 mostly focused on implementing the [“PIM Manager”](http://comments.gmane.org/gmane.comp.mobile.syncevolution/4009) D-Bus API for IVI use cases [1]. The 1.3.99.3 pre-release starts to include more features and bug fixes again for syncing. For example, several ActiveSync improvements from Graham Cobb were included. The remaining goal for 1.4, besides more testing of course, is to work out how to support Google CalDAV and CardDAV. I am in discussion with Google to get SyncEvolution [whitelisted](http://googleblog.blogspot.de/2013/03/a-second-spring-of-cleaning.html) for use with these APIs - fingers crossed… Upgrading from release 1.2.x —————————- The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution –configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the “refresh-from-server” sync mode to avoid getting throttled with 417 ‘retry later’ errors. The same must be added to existing configs manually: syncevolution –configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2 ———————————- Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see “syncevolution –print-configs”) but must be renamed manually to use them again under their original names with older SyncEvolution releases. Changes 1.3.2 -> 1.3.99.3 ========================= * PIM Manager: add ReplaceSearch, always allow it The new ReplaceSearch is more flexible than RefineSearch. It can handle both tightening the search and relaxing it. The downside of it is the more expensive implementation (must check all contacts again, then find minimal set of change signals to update view). Previously, a search which had no filter set at all at the begining could not be refined. This limitation of the implementation gets removed by always using a FilteredView, even if the initial filter is empty. * PIM Manager: introduce CreateConfig() That SetPeer() allows modifying and creating a config leads to race conditions when multiple clients want to create a config. The new CreateConfig() avoids that by atomically checking that a config does not exist yet and creating it. SetPeer() is still available for backwards compatibility. It continues to be used for modifying an existing config in TestContacts.testSync to check the effect of the logging settings. * PIM Manager: fix double entries in filtered search with limit Stressing the FilteredView by using it in tests originally written for the FullView showed that the filling up a view may have used data while it was inconsistent internally, leading to contacts being present multiple times. * PIM Manager and sync: support location = GEO property ([FDO #60373](https://bugs.freedesktop.org/show_bug.cgi?id=60373)) Exposed as “location” -> (lat, long) in the D-Bus bindings. Reading, writing and updating are supported. * PIM Manager: support groups = CATEGORIES ([FDO #60380](https://bugs.freedesktop.org/show_bug.cgi?id=60380)) Allow reading and writing of groups (folks terminology), aka CATEGORIES in vCard. * PIM Manager: intelligent phone search in EDS (part of [FDO #59571](https://bugs.freedesktop.org/show_bug.cgi?id=59571)) If phone number search is enabled in EDS, then the direct search in EDS now uses the more accurate E_BOOK_QUERY_EQUALS_NATIONAL_PHONE_NUMBER comparison, with the E164 formatted caller ID as value to compare against. This gives semantically correct results. The previous solution (now the fallback) had to use substring searches, which did not match if the contact’s phone number was not formatted according to E164 and which may have matched the wrong contacts if the trailing numbers are the same. * PIM Manager : use pre-computed normalized phone numbers from EDS (part of [FDO #59571](https://bugs.freedesktop.org/show_bug.cgi?id=59571)) When available, the pre-computed E164 number from EDS will be used instead of doing one libphonebook parser run for each telephone number while reading. Benchmarking showed that this parsing was the number one hotspot, so this is a considerable improvement. * PIM Manager: fix error messages Ensure and check that no unnecessary ERROR messages are printed. libfolks was used slightly incorrectly, leading to several harmless error messages (glib asserts). libphonenumber printed its error messages to stdout. * PIM Manager: fix memory leaks during writing of contacts Constructing the GValues created additional references instead of taking over ownership as intended. * D-Bus server: fix read-after-free bug when using syslog openlog() expects the string to remain valid. Must ensure that in LoggerSyslog by making a copy. Found with valgrind. * PIM Manager: make implementation of some of the D-Bus methods thread-safe The goal is to make it easier to extend syncevo-dbus-server with other IPC mechanisms, which then can call the native C++ code directly. That code was not prepared to handle calls in threads other than the main one. Now this is checked when entering the methods and work is shifted to the main thread if necessary. In the meantime the calling thread waits for completion. * PIM Manager: check responsiveness (part of [FDO #60851](https://bugs.freedesktop.org/show_bug.cgi?id=60851)) Enhanced the testActive test so that it can detect when the D-Bus server stops responding for too long. One major reason for that was event processing in folks, which got improved as part of https://bugzilla.gnome.org/show_bug.cgi?id=694385 * PIM Manager: adapt to gee 0.8 Changed the code to compile with gee 0.8, as used by folks 0.9.x. Older versions of folks are no longer supported. * PBAP: support Bluez 5 The new Bluez 5 API is the third supported API for doing PBAP transfers. It gets checked first, then the PBAB backend falls back to new-style obexd (file based, similar to Bluez 5, but not quite the same) and finally old-style obexd (data transfer via D-Bus). In contrast to previous APIs, Bluez 5 does not report the reason for a failed PBAP transfer. SyncEvolution then throws a generic “transfer failed” error with “reason unknown” as message. * command line: recover from slow sync with new sync modes The error message for an unexpected slow sync still mentioned the old and obsolete “refresh-from-client/server” sync modes. Better mention “refresh-from-local/remote”. * CalDAV: more workarounds for Google CalDAV + unique IDs Google became even more strict about checking REV. Tests which reused a UID after deleting the original item started to fail sometime since middle of December 2012. * CalDAV: work around Google server regression (undeclared namespace prefix in XML) Google CalDAV for a while (December 2012 till January 2013) sent invalid XML back when asked to include CardDAV properties in a PROPFIND. This got rejected in the XML parser, which prevents syncing calendar data: Neon error code 1: XML parse error at line 55: undeclared namespace prefix In the meantime Google fixed the issue in response to a bug report via email. But the workaround, only asking for the properties which are really needed, still makes sense and thus is kept. * WebDAV: don’t send Basic Auth via http proactively ([FDO #57248](https://bugs.freedesktop.org/show_bug.cgi?id=57248)) Sending basic authentication headers via http is insecure. Only do it proactively when the connection is encrypted and thus protects the information or when the server explicitly asks for it. * Nokia: always add TYPE=INTERNET to EMAIL ([FDO #61784](https://bugs.freedesktop.org/show_bug.cgi?id=61784)) Without the explicit TYPE=INTERNET, email addresses sent to a Nokia e51 were not shown by the phone and even got lost eventually (when syncing back). This commit ensures that the type is set for all emails sent to any Nokia phone, because there may be other phones which need it and phones which don’t, shouldn’t mind. This was spot-checked with a N97 mini, which works fine with and without the INTERNET type. This behavior can be disabled again for specific Nokia phones by adding a remote rule which sets the addInternetEmail session variable to FALSE again. Non-Nokia phones can enable the feature in a similar way, by setting the variable to TRUE. * SyncML: config option for broken peers Some peers have problems with meta data (CtCap, old Nokia phones) and the sync mode extensions required for advertising the restart capability (Oracle Beehive). The default in SyncEvolution is to advertise the capability, so manual configuration is necessary when working with a peer that fails in that mode. Because the problem occurs when SyncEvolution contacts the peers before it gets the device information from the peer, dynamic rules based on the peer identifiers cannot be used. Instead the local config must already disable these extra features in advance. The “SyncMLVersion” property gets extended for this. Instead of just “SyncMLVersion = 1.0” (as before) it now becomes possible to say “SyncMLVersion = 1.0, noctcap, norestart”. “noctcap” disables sending CtCap. “norestart” disables the sync mode extensions and thus doing multiple sync cycles in the same session (used between SyncEvolution instances in some cases to get client and server into sync in one session). Both keywords are case-insensitive. There’s no error checking for typos, so beware! The “SyncMLVersion” property was chosen because it was already in use for configuring SyncML compatibility aspects and adding a new property would have been harder. * ActiveSync: added support for specifying folder names Previously, the database field was interpreted as a Collection ID. This adds logic to allow the database to be interpreted as a folder path. The logic is: 1) If the database is an empty string, pass it through (this is the most common case as it is interpreted as “use the default folder for the source type”). 2) If the database matches a Collection ID, use the ID (this is the same as the previous behaviour). 3) If the database matches a folder path name, with an optional leading “/”, use the Collection ID for the matching folder. 4) Otherwise, force a FolderSync to get the latest folder changes from the server and repeat steps 2 and 3 5) If still no match, throw an error. * ActiveSync: support for listing databases Now –print-databases scans folders on the ActiveSync server and shows suitable folders for the ActiveSync backends instead of the previous, hard-coded help text. Invoking –print-databases can be used as a workaround for “SyncFolder error: Invalid synchronization key” errors. A better solution would be to do that automatically, but there was no time to implement that. See [FDO #61869](https://bugs.freedesktop.org/show_bug.cgi?id=61869) and “[SyncEvolution] Activesync server losing state” http://thread.gmane.org/gmane.comp.mobile.syncevolution/4295 * command line: show backend error when listing databases fails The command line swallowed errors thrown by the backend while listing databases. Instead it just showed “: backend failed”. The goal was to not distract users who accidentally access a non-functional backend. But the result is that operations like –configure or –print-databases could fail without giving the user any hint about the root cause of the issue. Now the error explanation in all its gory details is included. For example, not having activesyncd running leads to: INFO] eas_contact: backend failed: fetching folder list: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.meego.activesyncd was not provided by any .service files And running activesyncd without the necessary gconf keys shows up as: [INFO] eas_contact: backend failed: fetching folder list: GDBus.Error:org.meego.activesyncd.Error.AccountNotFound: Failed to find account [syncevolution@lists.intel.com] * Minor memory leak fix when using GDBus GIO: GDBusMethodInfo Also depends on a glib fix, see https://bugzilla.gnome.org/show_bug.cgi?id=695376 * build fixes Avoid -lrt in make dependencies. Add missing pcre libs to syncevo-dbus-server. sqlite backend needs “#include <stdio.h>” (patch from Mario Kicherer). 1.3.99.1 => 1.3.99.2, 13.12.2012 ================================ * PIM Manager searches for a caller ID (‘phone’ search) in EDS directly while folks still starts up. No unification is done of these results. Intermediate results are replaced by the final ones from folks once those are ready. * PIM Manager: allow configuration of session directories (part of [FDO #55921](https://bugs.freedesktop.org/show_bug.cgi?id=55921)) Useful for moving the session directories to a temporary file system. They are essentially just useful for debugging when used as part of PIM Manager. - “logdir” - a directory in which directories are created with debug information about sync session - “maxsessions” - number of sessions that are allowed to exist after a sync (>= 0): 0 is special and means unlimited, 1 for just the latest, etc.; old sessions are pruned heuristically (for example, keep sessions where something changed instead of some where nothing changed), so there is no hard guarantee that the last n sessions are present. * PIM Manager: write less data to disk (part of [FDO #55921](https://bugs.freedesktop.org/show_bug.cgi?id=55921)) Avoid writing config file changes to disk by enabling a new “ephemeral” mode for syncing via the PIM Manager. In this mode, config file changes are not flushed resp. discarded directly. This prevents writing to .ini files in ~/.config. The “synthesis” binfile client files are still written, but they get redirected into the session directory, which can (and should) be set to a temp file system and get deleted again quickly. Data dumps are turned off now in the configs created by the PIM Manager. * syncevo-dbus-server: use syslog instead of standard output by default * syncevo-dbus-server: command line options for controlling output and startup -d, –duration=seconds/’unlimited’ Shut down automatically when idle for this duration (default 300 seconds) -v, –verbosity=level Choose amount of output, 0 = no output, 1 = errors, 2 = info, 3 = debug; default is 1. -o, –stdout Enable printing to stdout (result of operations) and stderr (errors/info/debug). -s, –no-syslog Disable printing to syslog. -p, –start-pim Activate the PIM Manager (= unified address book) immediately. * PIM Manager: store set of active address books persistently ([FDO #56334](https://bugs.freedesktop.org/show_bug.cgi?id=56334)) Together with storing the sort order persistently, this allows restarting the daemon and have it create the same unified address book again. * PIM Manager: remove colon from valid peer UID character set ([FDO #56436](https://bugs.freedesktop.org/show_bug.cgi?id=56436)) Using the UID as part of file names gets more problematic when allowing colons. Remove that character from the API and enforce the format in the source code. * PIM Manager API: introduce contact ID and use it for reading This makes it easier for a client to fully polulate its view with contact data. Previously it could happen that due to concurrent changes in the server, a client was returned data for the same contact multiple times. A client had to detect that and re-issue read requests. * PIM Manager API: optional ViewAgent.Quiescent() ([FDO #56428](https://bugs.freedesktop.org/show_bug.cgi?id=56428)) The callback is guaranteed to be invoked once when a search has finished sending its initial results, and not sooner. This makes it possible to check whether the current data contains some contact or not. * PIM Manager: limit number of search results ([FDO #56142](https://bugs.freedesktop.org/show_bug.cgi?id=56142)) A ‘limit’ search term with a number as parameter (formatted as string) can be added to a ‘phone’ or ‘any-contains’ search term to truncate the search results after a certain number of contacts. Example: Search([[‘any-contains’, ‘Joe’], [‘limit’, ‘10’]]) => return the first 10 Joes. As with any other search, the resulting view will be updated if contact data changes. The limit must not be changed in a RefineSearch(). A ‘limit’ term may (but doesn’t have to) be given. If it is given, its value must match the value set when creating the search. This limitation simplifies the implementation and its testing. The limitation could be removed if there is sufficient demand. * PIM Manager: fix refining a search Due to not mapping the local index in the view to the parent’s index, refining only worked in views where parent and child had the same index for the contacts in the search view. * PIM Manager: fix starting when done via search When the unified address book (= FullView) was not running yet at the time when a client wanted to search it, the unified address book was not started and thus the search never returned results. * PIM Manager: fix writing contact, support photo and notes folks and EDS do not support writing properties in parallel (https://bugzilla.gnome.org/show_bug.cgi?id=652659). Must serialize setting of modified properties. * PIM Manager: fix incorrect contact removal signals in filtered view The filtered view did not check whether a parent’s removed contact was really part of the view before sending a removal signal for it. * D-Bus: missing out parameters in D-Bus introspection XML ([FDO #57292](https://bugs.freedesktop.org/show_bug.cgi?id=57292)) The problem was in the C++ D-Bus binding. If the method that gets bound to D-Bus returns a value, that value was ignored in the signature: int foo() => no out parameter It works when the method was declared as having a retval: void foo (int &result) => integer out parameter This problem existed for both the libdbus and the GIO D-Bus bindings. In SyncEvolution it affected methods like GetVersions(). * PIM Manager performance: pre-compute normalized telephone numbers Looking up by phone number spends most of its cycles in normalizing of the phone numbers in the unified address book. Instead of doing that work over and over again during the search, do it once while loading. Looking up a phone number only once does not gain from this change, it even gets slower (more memory intensive, less cache locality). Only searching multiple times becomes faster. Ultimately it would be best to store the normalized strings together with the telephone number inside EDS when the contact gets created. Work on that is in progress. * PIM Manager: improve performance of FullView sorting This fixes the hotspot during populating the FullView content: moving contacts around required copying IndividualData and thus copying complex C++ structs and strings. Storing pointers and moving those avoids that, with no lack of convenience thanks to boost::ptr_vector. Reordering also becomes faster, because the intermediate copy only needs to be of the pointers instead of the full content. * PIM Manager example: add benchmarking The new “checkpoints” split up the whole script run into pieces which are timed separately, with duration printed to stdout. In addition, tools like “perf” can be started for the duration of one phase. * EDS: fix creating databases –create-database was broken in combination with the final code in EDS 3.6 because it passed NULL for the UID to e_source_new_with_uid(), which is considered an error by the implementation of that method. Must use e_source_new() if we don’t have a UID. * fixed some memory leaks, extended tests to cover new features and bugs SyncEvolution 1.3.99.1, 25.10.2012 ================================== * workarounds for warnings from g++ 4.5 * engine: : local cache sync mode This patch introduces support for true one-way syncing (“caching”): the local datastore is meant to be an exact copy of the data on the remote side. The assumption is that no modifications are ever made locally outside of syncing. This is different from one-way sync modes, which allows local changes and only temporarily disables sending them to the remote side. Another goal of the new mode is to avoid data writes as much as possible. This new mode only works on the server side of a sync, where the engine has enough control over the data flow. Setting “sync” to: - “local-cache-incremental” will do an incremental sync (if possible) or a slow sync (otherwise). This is usually the right mode to use, and thus has “local-cache” as alias. - “local-cache-slow” will always do a slow sync. Useful for debugging or after (accidentally) making changes on the local side. An incremental sync will ignore such changes because they are not meant to happen, aren’t checked for to improve performance and thus will leave client and server out-of-sync! Both modes are recorded in the sync report of the local side. The target side is the client and records the normal “two-way” or “slow” sync modes. With the current SyncEvolution contact field list, first, middle and last name are used to find matches for contacts. For events, tasks and memos, time, summary and description are used. * HTTP proxy: useProxy=0 overrides http_* env variables Previously, if http_proxy was set, a proxy was used even if explicitly disabled. This prevented disabling the use of a proxy which only made sense in some cases, like accessing something that runs locally. Explicitly telling SyncEvolution to ignore http_proxy is necessary because it doesn’t support no_proxy. * WebDAV: auto-discovery fix With Google Contact + CardDAV the auto-discovery failed after finding the default address book, without reporting that result. * command line: implement –create/remove-database Creating a database is only possible with a chosen name. The UID is chosen automatically by the storage. Only implemented in the EDS backend. * file backend: sub-second mod time stamps Change tracking in the file backend used to be based on the modification time in seconds. When running many syncs quickly (as in testing), that can lead to changes not being detected when they happen within a second. Now the file backend also includes the sub-second part of the modification time stamp, if available. This change is relevant when upgrading SyncEvolution: most of the items will be considered “updated” once during the first sync after the upgrade (or a downgrade) because the revision strings get calculated differently. * D-Bus server: avoid progress outside of 0-100% range For example in the new TestLocalCache.testItemDelete100, the percentage value in the ProgressChanged signal become larger than 100 and then revert to 100 at the end of the sync. Seems the underlying calculation is faulty or simply inaccurate. This is not fixed. Instead the result is just clipped to the valid range. * code cleanup + improvements in testing Source, Installation, Further information ========================================= http://syncevolution.org/blogs/pohly/2013/syncevolution-13993-released Source code bundles for users are available in http://downloads.syncevolution.org/syncevolution/sources and the original source is [the git repositories](http://cgit.freedesktop.org/SyncEvolution/). i386, lpia and amd64 binaries for Debian-based distributions are available via the “unstable” syncevolution.org repository. Add the following entry to your /apt/source.list:

  deb http://downloads.syncevolution.org/apt unstable main Then install “syncevolution-evolution”, “syncevolution-kde” and/or “syncevolution-activesync”. These binaries include the “sync-ui” GTK GUI and were compiled for Ubuntu 10.04 LTS (Lucid), except for “syncevolution-activesync” which depends on libraries in Debian Squeeze, for example EDS 3.4. Older distributions like Debian 4.0 (Etch) can no longer be supported with precompiled binaries because of missing libraries, but the source still compiles when not enabling the GUI (the default). The same binaries are also available as .tar.gz and .rpm archives in [the download directories](http://downloads.syncevolution.org/syncevolution/). In contrast to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the content must be moved to /usr, because several files would not be found otherwise. After installation, follow the [getting started](/documentation/getting-started) steps. More specific [HOWTOs](/wiki/howto) can be found in the Wiki.

SyncEvolution 1.3.2 released   SyncEvolution 1.3.99.4 released