clsync/man/man1/clsync.1

.\" Sorry for my English 
.\" --Dmitry Yu Okunev <dyokunev@ut.mephi.ru> 0x8E30679C
.\"
.\" Thanks to oldlaptop [https://github.com/oldlaptop] for help with spelling
.\"
.TH CLSYNC 1 "JULY 2013" Linux "User Manuals"
.SH NAME
clsync \- live sync tool, written in GNU C
.SH SYNOPSIS
.B clsync [ options ] 
.I watch\-dir
.I sync\-handler
.B [
.I rules\-file
.B ]
.B [
.I dest-dir
.B ]
.SH DESCRIPTION
.B clsync
executes
.I sync\-handler
with appropriate arguments on FS events in directory
.I watch\-dir
using the
.BR inotify (7)
Linux kernel subsystem.

Extended regex\-rules to filter what files and
directories to sync may be placed in
.I rules\-file


.SH OPTIONS

.B \-b, \-\-background
.RS 8
Daemonize, forcing clsync to fork() on start.

.PP
.RE

.B \-H, \-\-config\-file
.I config\-file\-path
.RS 8
Use configuration from file
.IR config\-file\-path .
(see 
.BR "CONFIGURATION FILE" )

.PP
.RE

.B \-K, \-\-config\-block
.I config\-block\-name
.RS 8
Use configuration block with name
.IR config\-block\-name .
(see 
.BR "CONFIGURATION FILE" )

.PP
.RE

.B \-z, \-\-pid\-file
.I path\-to\-pidfile
.RS 8
Writes pid to file by path
.IR path\-to\-pidfile .

.PP
.RE

.B \-u, \-\-uid
.I uid
.RS 8
Drop user privileges to uid
.I uid
with
.BR setuid (2)
.PP
.RE

.B \-g, \-\-gid
.I gid
.RS 8
Drop group privileges to gid
.I gid
with
.BR setgid (2)
.PP
.RE

.B \-p, \-\-pthread
.RS 8
Use
.BR pthreads (7)
to parallelize syncing processes.

If you're running
.B clsync
with option
.B \-\-pthread
in conjunction with
.B rsync
with option
.BR \-\-backup ,
you may catch a bug due to nonatomicity of rsync's file replace operation.
(see
.BR DIAGNOSTICS )
.RE

.B \-Y, \-\-syslog
.RS 8
Use syslog instead of stderr for logging
.RE

.PP
.B \-c, \-\-cluster\-iface
.I interface\-ip
.RS 8
.B Not implemented, yet.

.B DANGEROUS OPTION. This functionality wasn't tested well. You can lost your data.

Enables inter-node notifing subsystem to prevent sync looping between nodes.
This's very useful features that provides ability of birectional sync of the
same directory between two or more nodes.
.I interface-ip
is an IP-address already assigned to the interface that will be used for
multicast notifing.

Not enabled by default.

To find out the IP-address on interface "eth0", you can use for example next
command:

ip a s eth0 | awk '{if($1=="inet") {gsub("/.*", "", $2); print $2}}'
.RE

.PP
.B \-m, \-\-cluster\-ip
.I multicast\-ip
.RS 8
.B Not implemented yet.

Sets IP-address for multicast group.

This option can be used only in conjunction with
.BR \-\-cluster\-interface .

Use IP-addresses from 224.0.0.0/4 for this option.

Default value is "227.108.115.121". [(128+"c")."l"."s"."y"]
.RE

.PP
.B \-P, \-\-cluster\-port
.I multicast\-port
.RS 8
.B Not implemented yet.

Sets UDP-port number for multicast messages.

This option can be used only in conjunction with
.BR \-\-cluster\-interface .

.I multicast\-port
should be greater than 0 and less than 65535.

Default value is "40079". [("n" << 8) + "c"]
.RE

.PP
.B \-W, \-\-cluster\-timeout
.I cluster\-timeout
.RS 8
.B Not implemented yet.

Sets timeout (in milliseconds) of waiting answer from another nodes of the
cluster. If there's no answer from some node, it will be excluded.

Default value is "1000". [1 second]
.RE

.PP
.B \-n, \-\-cluster\-node\-name
.I cluster\-node\-name
.RS 8
.B Not implemented yet.

Sets the name of current node in the cluster. It will be used in action
scripts of another nodes (see 
.BR "SYNC HANDLER" ).

Default value is $(uname \-n).
.RE

.PP
.B \-o, \-\-cluster\-hash\-dl\-min
.I hash\-dirlevel\-min
.RS 8
Sets minimal directory level for ctime hashing (see
.BR CLUSTERING ).

Default value is "1".
.RE

.PP
.B \-O, \-\-cluster\-hash\-dl\-max
.I hash\-dirlevel\-max
.RS 8
.B Not implemented yet.

Sets maximal directory level for ctime hashing (see
.BR CLUSTERING ).

Default value is "16".
.RE

.PP
.B \-O, \-\-cluster\-scan\-dl\-max
.I scan\-dirlevel\-max
.RS 8
.B Not implemented yet.

Sets maximal directory level for ctime scanning (see
.BR CLUSTERING ).

Default value is "32".
.RE

.PP
.B \-k, \-\-timeout\-sync
.I sync-timeout
.RS 8
Sets timeout for syncing processes.
.B clsync
will die if syncing process alive more than
.I sync-timeout
seconds.

Set "0" to disable the timeout.

Default value is "86400" ["24 hours"].
.RE

.PP
.B \-w, \-\-delay\-sync
.I additional\-delay
.RS 8
Sets the minimal delay (in seconds) between syncs.

Default value is "30".
.RE

.PP
.B \-t, \-\-delay\-collect
.I ordinary\-delay
.RS 8
Sets the delay (in seconds) to collect events about ordinary files and
directories.

Default value is "30".
.RE

.PP
.B \-T, \-\-delay\-collect\-bigfile
.I bigfiles\-delay
.RS 8
Sets the delay (in seconds) to collect events about "big files" (see
.IR \-\-threshold\-bigfile ).

Default value is "1800".
.RE

.PP
.B \-B, \-\-threshold\-bigfile
.I filesize\-threshold
.RS 8
Sets file size threshold (in bytes) that separates ordinary files from
"big files". Events about "big files" are processed in another queue with a
separate collecting delay. This is supposed to be used as a means of unloading
IO resources.

Default value is "134217728" ["128 MiB"].
.RE

.PP
.B \-d, \-\-dir\-lists
.I tmpdir\-path
.RS 8
Sets directory path to output temporary events\-lists files.

If this option is enabled,
.B clsync
will execute
.I sync\-handler
once for each aggregated event list, passing the path to a file containing
this list (actions "synclist" and "rsynclist").
Otherwise,
.B clsync
will execute
.I sync\-handler
for every file in the aggregated event list (action "sync").

Cannot be used in conjunction with
.IR \-\-synchandler\-so\-module
.

See
.BR "SYNC HANDLER" .

Is not set by default.
.RE

.PP
.B \-S, \-\-initialsync\-enable
.RS 8
Use action "initialsync" instead of "synclist" for directories that were just marked (see
.B "SYNC HANDLER"
case
.BR c ).

Is not set by default.
.RE

.PP
.B \-Z, \-\-synclist\-simplify
.RS 8
Removes the first 3 parameters in list files of action "synclist" (see
.B "SYNC HANDLER"
case
.BR b ).

Is not set by default.
.RE

.PP
.B \-A, \-\-auto\-add\-rules\-w
.RS 8
Forces clsync to create a "w\-rule" for every non-"w-rule" (see
.BR RULES ).

Not recommended to use in conjunction with
.I \-\-rsync
option.

Is not set by default.
.RE

.PP
.B \-M, \-\-synchandler\-so\-module
.RS 8
.B clsync
will load
.I sync-handler
as a shared library with
.BR dlopen (3)
and run syncing by calling function "clsyncapi_sync()" from
this shared object file (see 
.B "SYNC HANDLER"
case
.BR f ).

Cannot be used in conjunction with
.I \-\-rsync
and/or
.IR \-\-dir\-lists .

Is not set by default.
.RE

.PP
.B \-R, \-\-rsync
.RS 8
Forces
.B clsync
to use "rsynclist" action instead of "initialsync" and "synclist" (see 
.B "SYNC HANDLER" 
case 
.BR d ).
This option can be used only with
.I \-\-dir\-lists
option.

Recommended option.

You can use "\-R2" to run "rsync" directly, bypassing intermediate script
(see
.B "SYNC HANDLER"
case
.BR e ).

Cannot be used in conjunction with 
.IR \-\-synchandler\-so\-module .

But you cannot use 
.I \-\-cluster\-*
options in 
.I \-R2
mode.

Is not set by default.
.RE

.PP
.B \-L, \-\-rsync\-inclimit
.I rsync\-includes\-line\-limit
.RS 8
Sets soft limit for lines count in files by path
.IR rsync\-listpath .
Unfortunately, rsync works very slowly with huge "\-\-include\-from"
files. So,
.B clsync
splits that list with approximately
.I rsync\-includes\-line\-limit
lines per list if it's too big, and executes by one rsync instance per list
part. Use value "0" to disable the limit.

Default value is "20000".
.RE

.PP
.B \-I, \-\-rsync\-prefer\-include
.RS 8
Forces
.B clsync
to prefer a "lot of includes" method instead of a "excludes+includes" for
rsync on recursive syncing.

See
.B case d
of
.BR "SYNC HANDLER" .

This option is not recommended.

Is not set by default.
.RE

.PP
.B \-x, \-\-ignore\-exitcode
.I exitcode
.RS 8
Forces
.B clsync
to do not process exitcode
.I exitcode
of
.I sync\-handler
as an error. You can set multiple ignores by passing this option multiple
times.

Recommended values for rsync case are "23" and "24".
["\-x 23 \-x 24"]
.RE

.PP
.B \-U, \-\-dont\-unlink\-lists
.RS 8
Do not delete list\-files after
.I sync\-handler
has finished.

This may be used for debugging purposes.

Is not set by default.
.RE

.PP
.B \-F, \-\-full\-initialsync
.RS 8
Ignore filter rules from
.I rules-file
on initial sync.

This may be useful for quick start or e.g. if it's required to sync 
"/var/log/" tree but not sync every change from there.

Is not set by default.
.RE

.PP
.B \-v, \-\-verbose
.RS 8
This option is supposed to increase verbosity. But at the moment there's no
"verbose output" in the code, so the option does nothing. :)
.RE

.PP
.B \-D, \-\-debug
.RS 8
Increases debugging output. This may be supplied multiple times for more
debugging information, up to a maximum of three "D" flags (more will do 
nothing), for example "\-D \-D \-D" or "\-D3" (equivalent cases)
.RE

.PP
.B \-q, \-\-quiet
.RS 8
Suppresses error messages.
.RE

.PP
.B \-f, \-\-fanotify
.RS 8
.B Don't use this option!

Switches monitor subsystem to "fanotify" [it's described for
future\-compatibility].
.RE

.PP
.B \-i, \-\-inotify
.RS 8
Switches monitor subsystem to "inotify".

Is set by default.
.RE

.PP
.B \-l, \-\-label
.I label
.RS 8
Sets a label for this instance of clsync. The
.I label
will be passed to
.I sync\-handler
every execution.

Default value is "nolabel".
.RE

.PP
.B \-h, \-\-help
.RS 8
Outputs options list and exits with exitcode "0".
.RE

.PP
.B \-V, \-\-version
.RS 8
Outputs clsync version and exits with exitcode "0".
.RE

.SH SYNC HANDLER
.B clsync
executes
.I sync\-handler
that supposed to take care of the actual syncing process. Therefore
.B clsync
is only a convenient way to run a syncing script.

.B clsync
can run
.I sync\-handler
in six ways. Which way will be used depends on specified options (see
.I \-\-dir\-lists
,
.I \-\-rsync
and 
.IR \-\-synchandler\-so\-module ).

case
.B a
.RS
.I sync\-handler
sync
.I label evmask path [nodes]

In this case,
.I sync\-handler
is supposed to non\-recursively sync file or directory by
.IR path .
With
.I evmask
it's passed bitmask of events with the file or directory (see 
"/usr/include/linux/inotify.h").

This case will be used, if there is no
.I \-\-dir\-lists
and
.I \-\-synchandler\-so\-module
options are set.
.RE

case
.B b
.RS
.I sync\-handler
synclist
.I label listpath [nodes]

In this case,
.I sync\-handler
is supposed to non\-recursively sync files and directories from list in a file by
path
.IR listpath (see
below). With
.I evmask
it's passed bitmask of events with the file or directory (see 
"/usr/include/linux/inotify.h").

This case will be used if option
.I \-\-dir\-lists
is set, but options
.I \-\-rsync
and
.I \-\-synchandler\-so\-module
are not set.
.RE

case
.B c
.RS
.I sync\-handler
initialsync
.I label dirpath [nodes]

In this case,
.I sync\-handler
is supposed to recursively sync data from directory by path
.I dirpath
with manual excluding extra files.

This case will be used for initial sync, if options
.I \-\-dir\-lists
and
.I \-\-initialsync\-enable
are set, but option
.I \-\-synchandler\-so\-module
is not set.
.RE

case
.B d
.RS
.I sync\-handler
rsynclist
.I label rsync\-listpath [nodes] [rsync\-exclude\-listpath]

In this case,
.I sync\-handler
is supposed to run "rsync" application with parameters: 

\-aH \-\-delete\-before \-\-include\-from
.I rsync\-listpath
\-\-exclude '*'

if option
.I \-\-rsync\-prefer\-include
is enabled.

And with parameters:

\-aH \-\-delete\-before \-\-exclude\-from
.I rsync\-exclude\-listpath
\-\-include\-from
.I rsync\-listpath
\-\-exclude '*'

if option
.I \-\-rsync\-prefer\-include
is disabled.

This case will be used, if option
.I \-\-rsync
is set once.

Recommended case.
.RE

case
.B e
.RS
.I sync\-handler
\-\-inplace \-aH \-\-delete\-before [\-\-exclude\-from
.I rsync\-exclude\-listpath
]
\-\-include\-from
.I rsync\-listpath
\-\-exclude '*'
.I watch-dir/ dest-dir/

In this case,
.I sync\-handler
is supposed to be a path to
.B rsync
binary.

Error code "24" from
.I sync\-handler
will be ignored in this case.

This case will be used, if option
.I \-\-rsync
is set twice (or equivalently "\-\-rsync=2"). This case is supposed to be used
only as a proof of concept.
.RE

case
.B f
.RS
In this case there's no direct exec*() calling. In this case
.B clsync
loads
.I sync-handler
as a shared library with
.BR dlopen (3)
and calls function "int clsyncapi_sync(int n, api_eventinfo_t *ei)" from it
for every sync.
.B n
is number of elements of
.BR ei .
.B ei
is an array of structures with information about what and how to sync (see
below).

api_eventinfo_t is a structure:
.RS
struct api_eventinfo {
.br
        uint32_t    evmask;	// event bitmask for file/dir by path
.BR path .
.br
        uint32_t    flags;	// flags of "how to sync" the file/dir
.br
        size_t      path_len;	// strlen(path)
.br
        const char *path;	// the path to file/dir need to be
synced
.br
};
.br
typedef struct api_eventinfo api_eventinfo_t;
.RE

The event bitmask (evmask) values can be learned from
"/usr/include/linux/inotify.h".

There may be next flags' values (flags):
.RS
enum eventinfo_flags {
.br
        EVIF_RECURSIVELY = 0x00000001	// sync the file/dir recursively
.br
};
.RE
.br
Flag "EVIF_RECURSIVELY" may be used if option
.I\-\-initialsync\-enable
is set.

Also may be defined functions "int clsyncapi_init(options_t *, indexes_t *)"
and "int clsyncapi_deinit()" to initialize and deinitialize the syncing
process by this shared object.

See example file "clsync-synchandler-so.c".

This case will be used, if option
.I \-\-synchandler\-so\-module
is set.

Recommended case.
.RE

As can be noticed, in the first four cases clsync's
.I label
(see 
.I \-\-label
) and
.I nodes
are passed (or can be passed).
.I nodes
is comma-separated list of cluster nodes names where to sync to (see
.I \-\-cluster-node-name
)

The listfile by path
.I listpath
contains lines separated by NL (without CR) of next format:
.RS
sync
.I label evmask path
.RS
if option
.I \-\-synclist\-simplify
is not set
.RE
.I path
.RS
if option
.I \-\-synclist\-simplify
is set
.RE

Every lines is supposed to be proceed by external syncer to sync file or
directory by path
.IR path .
With
.I evmask
it's passed bitmask of events with the file or directory (see
"/usr/include/linux/inotify.h").

.RE

.SH RULES
Filter riles can be placed into
.I rules\-file
with one rule per line.

Rule format:
.I [+\-][fdw*]regexp

.I +
\- means include;
.I \-
\- means exclude;
.I f
\- means file;
.I d
\- means directory;
.I w
\- means walking to directory;
.I *
\- means all.

For example: \-*^/[Tt]est

It's not recommended to use
.I w
rules for
.I \-\-rsync
cases.
.BR rsync (1)
allows to set syncing and walking only together in "\-\-include" rules
("\-\-files\-from" is not appropriate due to problem with syncing files
deletions). So there may be problems with clsync's
.I w
rules in this cases.

More examples:

Syncing pwdb files and sshd_config (non-rsync case):
.RS
	+f^/passwd$
.br
	+f^/group$
.br
	+f^/shadow$
.br
	+f^/ssh/sshd_config$
.br
	+w^$
.br
	+w^/ssh$
.br
	-*
.RE

Syncing pwdb files and sshd_config (non-rsync case with option
.IR \-\-auto\-add\-rules\-w ):
.RS
	+f^/passwd$
.br
	+f^/group$
.br
	+f^/shadow$
.br
	+f^/ssh/sshd_config$
.br
	-*
.RE

Syncing pwdb files and sshd_config (rsync case):
.RS
	+f^/passwd$
.br
	+f^/group$
.br
	+f^/shadow$
.br
	+f^/ssh/sshd_config$
.br
	+d^$
.br
	+d^/ssh$
.br
	-*
.RE

Syncing /srv/lxc tree (rsync case):
.RS
	-d/sess(ion)?s?$
.br
	-f/tmp/
.br
	+*
.RE

.SH SIGNALS
1  \- to reread filter rules

10 \- runs threads' GC function

12 \- runs full resync

16 \- interrupts sleep()/select() and wait() [for debugging and internal uses]


.SH DIAGNOSTICS

Initial rsync process works very slow on clsync start
.RS
Probably there's too huge exclude list is passed to rsync. This can happened
if you're excluding with regex in clsync's rules a lot of thousands files.
They will be passed to rsync's exclude list one by one.

To diagnose it, you can use "\-U" option and look into 
.I rsync\-exclude\-listpath
file (see
.B "SYNC HANDLER"
case 
.BR d )

To prevent this, it's recommended to write such rules for rsync directly 
(not via clsync).

For example, often problem is with PHP's session files. You shouldn't exclude
them in clsync's rules with "\-f/sess_.*", but you should exclude it in rsync
directly (e.g with «\-\-exclude "sess_*"»).
.RE

The following diagnostics may be issued on stderr:

Error: Cannot inotify_add_watch() on [...]
.RS
Not enough inotify watching descriptors is allowed. It can be fixed
by increasing value of "sysctl fs.inotify.max_user_watches"
.RE

Error: Got non-zero exitcode
.I exitcode
[...]
.RS
.I sync\-handler
returned non-zero exitcode. Probably, you should process exitcodes in it or
your syncer process didn't worked well. I case of using rsync, you can find
the exitcodes meanings in
.BR "man 1 rsync" .

If
.I exitcode
equals to 23 and you're using
.B clsync
in conjunction with
.BR rsync ,
this may happend, for example in next cases:

.RS

\- Not enough space on destination.

\- You're running clsync with 
.B \-\-pthread
and rsync with
.BR \-\-backup .
See bugreport by URL:
.IR https://bugzilla.samba.org/show_bug.cgi?id=10081 .

.RE

To confirm the problem, you can try to add "return 0" or "exit 0" into
your
.IR sync\-handler .

.RE

.SH CONFIGURATION FILE

.B clsync
supports configuration file.

By default
.B clsync
tries to read next files (in specified order):
.RS
~/.clsync.conf
.br
/etc/clsync/clsync.conf
.RE

This may be overrided with option
.IR \-\-config\-file .

.B clsync
reads only one configuration file. In other words, if option
.I \-\-config\-file
is not set and file
.B ~/.clsync.conf
is accessable and parsable,
.B clsync
will not try to open
.BR /etc/clsync/clsync.conf .
Command line options have precedence over config file options.

Configuration file is parsed with glib's g_key_file_* API. That means,
that config should consits from groups (blocks) of key-value lines as in the
example:
.RS
[default]
.br
background=1
.br
rsync=1
.br
debug=0
.br
syslog=1
.br
pid-file=/var/run/clsync.pid
.br

.br
[test]
.br
rsync=2
.br
debug=3
.RE

In this example there's 2 blocks are set - "default" and "test".

By default
.B clsync
uses block with name "default". Block name can be set by option
.IR \-\-config\-block .

.SH CLUSTERING

Not implemented yet.
.B Don't try to use cluster functionality.

Not described yet.

.SH EXAMPLES
Working example you can try out in "example/" or 
"/usr/share/doc/clsync/example/" directory. Copy this directory somewhere
(e.g. into "/tmp"). And try to run "clsync-start.sh" in there. Any
files/directories modifications in "example/testdir/from" will be synced to
"example/testdir/to" with few seconds delay.
.RE
.SH AUTHOR
Dmitry Yu Okunev <dyokunev@ut.mephi.ru> 0x8E30679C
.SH "SEE ALSO"
.BR rsync (1),
.BR pthreads (7),
.BR inotify (7)