What's this all about? Well, having worked with csync2 combined with 10M+ files, with 100+ updates/second I can guess I have seen the limits of csync2, and they are well below 10 concurrent updates/second in a 2 way cluster. So the need arised to fix it. Fixes have resulted in patches:
  • sqlite3 support (is in upstream)
  • limit support to limit the amount of files handled per batch (needed for database locking)
  • Alternate temp directory option for csync
  • csync2id daemon that uses inotify to keep stuff hinted to csync2.
  • splitting of the database in a per table seperate database. It overcomes a lot, but it also introduces deadlocks ;-).
(You can find the remainder of the patches here: http://projects.kwaak.net/twiki/pub/CSync2/WebHome/csync2-r401-experimental-patches.patch ) But none of them really fixes the big issue: concurrent database access on an sqlite database sucks bad, really really bad... So I am about to say that the only solution is to make the database access part a single process, in other words: a client server architecture. But which way to go? Just using an sqlite wrapper like uSQLiteserver would that solve stuff? Well probably. It actually doesn't really matter: we are now at a point that we must describe the states and interaction with other hosts in order to survive the current locking madness.

SomeDiagrams

Topic attachments
I Attachment Action Size Date Who Comment
elsepatch csync2-r401-experimental-patches.patch manage 12.8 K 16 Sep 2009 - 17:45 ArdVanBreemen unsubmitted patches to csync2
elsepatch csync2-trunk409-to-local-4.patch manage 69.6 K 25 Nov 2009 - 14:49 ArdVanBreemen Latest version of patches
elsepatch local-4-to-local-5.patch manage 0.9 K 25 Nov 2009 - 16:28 ArdVanBreemen Allow daemonless usage of csync2
elsebz2 new-csdb.tar.bz2 manage 499.6 K 24 Nov 2009 - 13:54 ArdVanBreemen  
Topic revision: r13 - 12 Mar 2010 - 07:35:31 - TWikiGuest
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback