Realtime event backup to cloud storage - Revision history

Kernelkurtz at 20:27, 22 February 2026

2026-02-22T20:27:16Z

← Older revision		Revision as of 16:27, 22 February 2026
Line 98:		Line 98:
	''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.		''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.

	''exclude = { '.jpg' }'' These are generally used by the Zoneminder GUI but are not really needed as standalone files in my case (though some people may save .jpgs on purpose, in which case you will want to remove this line). I'm only interested in having backups of the videos, so not syncing these reduces the number of uploaded files significantly. You can break them out by type 'snapshot.jpg' , 'alarm.jpg' , 'capture*.jpg' etc., or just exclude them all as shown. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync, so you can avoid that with these filters.		''exclude = { '.jpg' }'' These are generally used by the Zoneminder GUI but are not really needed as standalone files in my case (though some people may save .jpgs on purpose, in which case you will want to remove this line). I'm only interested in having backups of the videos, so not syncing these reduces the number of uploaded files significantly. You can break them out by type 'snapshot.jpg' , 'alarm.jpg' , 'capture*.jpg' etc., or just exclude them all as shown. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync, so you can avoid that with these filters. The log snippet below did not include this filter.

	''archive = true ''This preserves ownership and permissions on the copied files.		''archive = true ''This preserves ownership and permissions on the copied files.

Kernelkurtz: Add exclude all jpegs

2026-02-22T20:26:15Z

Add exclude all jpegs

← Older revision		Revision as of 16:26, 22 February 2026
Line 64:		Line 64:
	init = false,		init = false,

	exclude = { '~~snapshot~~*~~.jpg' , 'alarm~~.jpg' },		exclude = { '*.jpg' },

	rsync = {		rsync = {
Line 98:		Line 98:
	''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.		''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.

	''exclude = { '~~snapshot~~~~.jpg' , 'alarm~~.jpg' }'' These are used by the Zoneminder GUI but are ~~less useful~~ as standalone files. I'm only interested in having backups of the videos, so not syncing these ~~means uploading a single file per event rather than three~~, ~~but this directive is of course optional~~. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync~~. If~~ you ~~just want to exclude~~ these dynamically generated files, use 'snapshot-.jpg' instead of 'snapshot*.jpg' which will only catch the ones with dimensions in the filename eg.'snapshot-712x400.jpg'. The log snippet below did not include this option.		''exclude = { '.jpg' }'' These are generally used by the Zoneminder GUI but are not really needed as standalone files in my case (though some people may save .jpgs on purpose, in which case you will want to remove this line). I'm only interested in having backups of the videos, so not syncing these reduces the number of uploaded files significantly. You can break them out by type 'snapshot.jpg' , 'alarm.jpg' , 'capture*.jpg' etc., or just exclude them all as shown. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync, so you can avoid that with these filters.

	''archive = true ''This preserves ownership and permissions on the copied files.		''archive = true ''This preserves ownership and permissions on the copied files.

Kernelkurtz at 17:05, 20 February 2026

2026-02-20T17:05:20Z

← Older revision		Revision as of 13:05, 20 February 2026
Line 86:		Line 86:
	''maxProcesses = 1'' is the default. If you are syncing multiple sources or targets you may want to increase this.		''maxProcesses = 1'' is the default. If you are syncing multiple sources or targets you may want to increase this.

	''insist = true'' allows the service to start even if the target is not ready. ~~I got errors without this which went away when I added~~ it. ~~YMMV.~~		''insist = true'' allows the service to start even if the target is not ready. According to the docs "In production mode it is recommended to have insist on."

	''source = ''The directory where ZM events are stored.		''source = ''The directory where ZM events are stored.

Kernelkurtz: Reduce systemd logging

2026-02-19T19:54:57Z

Reduce systemd logging

← Older revision		Revision as of 15:54, 19 February 2026
Line 82:		Line 82:
	}</nowiki>		}</nowiki>

	''logfile = '' is pretty self-explanatory. Be sure to set up log rotation after you are done setting lsyncd up, if required.		''logfile = '' is pretty self-explanatory. Be sure to set up log rotation after you are done setting lsyncd up, if required. By default the packaged lsyncd service logs to the systemd journal, which also by default echoes to /var/log/messages. Since the application logs are sufficient I've added the 'StandardOutput=null' option to the lsyncd service's unit file to prevent this over-duplication, but I've left StandardError with the existing default logging to the journal.

	''maxProcesses = 1'' is the default. If you are syncing multiple sources or targets you may want to increase this.		''maxProcesses = 1'' is the default. If you are syncing multiple sources or targets you may want to increase this.
Line 115:		Line 115:
	''init = false'' causes the lsyncd to not do a complete initial synchronization between source and target on service startup. On S3 this is expensive in terms of API calls and may take a while. I know I'm going to miss some events during system maintenance/OS updates etc anyway so I don't consider this worth the cost, but you can set to true if you want it. Be warned it will make your target an exact mirror of your source.		''init = false'' causes the lsyncd to not do a complete initial synchronization between source and target on service startup. On S3 this is expensive in terms of API calls and may take a while. I know I'm going to miss some events during system maintenance/OS updates etc anyway so I don't consider this worth the cost, but you can set to true if you want it. Be warned it will make your target an exact mirror of your source.

	Having delete and init set to false also should make this config fairly safe to use against an existing target if you just run it on your machine without heeding my previous warning. Don't change or remove these unless you are absolutely sure you know what you are doing. There are also options besides true and false so read the docs for more info. It is far too much to cover here but the simple config above is all I need for my purposes.		Having delete and init set to false also should make this config fairly safe to use against an existing target if you just run it on your machine without heeding my previous warning. ''Don't change or remove these unless you are absolutely sure you know what you are doing.'' There are also options besides true and false so read the docs for more info. It is far too much to cover here but the simple config above is all I need for my purposes.

	Once the service is configured you can control it with systemctl like any other systemd service. By default it will be disabled so you will need to enable it to run automatically after suitable testing. You can check the logs for ~~startup or running~~ errors with journalctl. If syncs are working correctly you can watch them as uploads are logged very shortly after the events are saved to local disk;		Once the service is configured you can control it with systemctl like any other systemd service. By default it will be disabled so you will need to enable it to run automatically after suitable testing. You can check the logs for errors with journalctl (see note about logging above). If syncs are working correctly you can watch them as uploads are logged very shortly after the events are saved to local disk;

	<nowiki>[root@nvr system]# tail -50 /var/log/lsyncd/lsyncd.log		<nowiki>[root@nvr system]# tail -50 /var/log/lsyncd/lsyncd.log

Kernelkurtz at 05:11, 18 February 2026

2026-02-18T05:11:28Z

← Older revision		Revision as of 01:11, 18 February 2026
Line 98:		Line 98:
	''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.		''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.

	''exclude = { 'snapshot.jpg' , 'alarm.jpg' }'' These are used by the Zoneminder GUI but are less useful as standalone files. I'm only interested in having backups of the videos, so not syncing these means uploading a single file per event rather than three, but this directive is of course optional. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync. If you just want to exclude these dynamically generated files, use 'snapshot-.jpg' instead of 'snapshot*.jpg' which will only catch the ones with dimensions in the filename eg.'snapshot-712x400.jpg'.		''exclude = { 'snapshot.jpg' , 'alarm.jpg' }'' These are used by the Zoneminder GUI but are less useful as standalone files. I'm only interested in having backups of the videos, so not syncing these means uploading a single file per event rather than three, but this directive is of course optional. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync. If you just want to exclude these dynamically generated files, use 'snapshot-.jpg' instead of 'snapshot*.jpg' which will only catch the ones with dimensions in the filename eg.'snapshot-712x400.jpg'. The log snippet below did not include this option.

	''archive = true ''This preserves ownership and permissions on the copied files.		''archive = true ''This preserves ownership and permissions on the copied files.

Kernelkurtz at 05:05, 18 February 2026

2026-02-18T05:05:33Z

← Older revision		Revision as of 01:05, 18 February 2026
Line 98:		Line 98:
	''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.		''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.

	''exclude = { 'snapshot.jpg' , 'alarm.jpg'}'' These are used by the Zoneminder GUI but are less useful as standalone files. I'm only interested in having backups of the videos, so not syncing these means uploading a single file per event rather than three, but this directive is of course optional. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync. If you just want to exclude these dynamically generated files, use 'snapshot-.jpg' instead of 'snapshot*.jpg' which will only catch the ones with dimensions in the filename eg.'snapshot-712x400.jpg'.		''exclude = { 'snapshot.jpg' , 'alarm.jpg' }'' These are used by the Zoneminder GUI but are less useful as standalone files. I'm only interested in having backups of the videos, so not syncing these means uploading a single file per event rather than three, but this directive is of course optional. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync. If you just want to exclude these dynamically generated files, use 'snapshot-.jpg' instead of 'snapshot*.jpg' which will only catch the ones with dimensions in the filename eg.'snapshot-712x400.jpg'.

	''archive = true ''This preserves ownership and permissions on the copied files.		''archive = true ''This preserves ownership and permissions on the copied files.

Kernelkurtz at 05:04, 18 February 2026

2026-02-18T05:04:46Z

← Older revision		Revision as of 01:04, 18 February 2026
Line 64:		Line 64:
	init = false,		init = false,

	exclude = { 'snapshot*.jpg' , 'alarm.jpg'},		exclude = { 'snapshot*.jpg' , 'alarm.jpg' },

	rsync = {		rsync = {

Kernelkurtz: Added exclude snapshots

2026-02-18T05:04:11Z

Added exclude snapshots

← Older revision		Revision as of 01:04, 18 February 2026
Line 63:		Line 63:

	init = false,		init = false,

			exclude = { 'snapshot*.jpg' , 'alarm.jpg'},

	rsync = {		rsync = {
Line 94:		Line 96:
	''delay = ''This is how often rsync will run if there are any filesystem changes. Default is 15 seconds. For many users this will be close enough to real-time for the purpose. This can be combined with (or substituted) for another value in the settings field;		''delay = ''This is how often rsync will run if there are any filesystem changes. Default is 15 seconds. For many users this will be close enough to real-time for the purpose. This can be combined with (or substituted) for another value in the settings field;

	''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.		''maxDelay = # '' (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.

			''exclude = { 'snapshot.jpg' , 'alarm.jpg'}'' These are used by the Zoneminder GUI but are less useful as standalone files. I'm only interested in having backups of the videos, so not syncing these means uploading a single file per event rather than three, but this directive is of course optional. Functions like zmNinja's 24 Hour Review may create hundreds of snapshots all at once, which lsyncd will immediately sync. If you just want to exclude these dynamically generated files, use 'snapshot-.jpg' instead of 'snapshot*.jpg' which will only catch the ones with dimensions in the filename eg.'snapshot-712x400.jpg'.

	''archive = true ''This preserves ownership and permissions on the copied files.		''archive = true ''This preserves ownership and permissions on the copied files.
Line 111:		Line 115:
	''init = false'' causes the lsyncd to not do a complete initial synchronization between source and target on service startup. On S3 this is expensive in terms of API calls and may take a while. I know I'm going to miss some events during system maintenance/OS updates etc anyway so I don't consider this worth the cost, but you can set to true if you want it. Be warned it will make your target an exact mirror of your source.		''init = false'' causes the lsyncd to not do a complete initial synchronization between source and target on service startup. On S3 this is expensive in terms of API calls and may take a while. I know I'm going to miss some events during system maintenance/OS updates etc anyway so I don't consider this worth the cost, but you can set to true if you want it. Be warned it will make your target an exact mirror of your source.

	Having delete and init set to false also should make this config fairly safe to use against an existing target if you just run it on your machine without heeding my previous warning. Don't change or remove these unless you are absolutely sure you know what you are doing. There are also options besides true and false so read the docs for more info. ~~You can also add options to be added directly to the rsync calls. You can sync multiple directories to multiple targets with multiple options. There are exclude options which you may find useful.~~ It is far too much to cover here but the simple config above is all I need for my purposes.		Having delete and init set to false also should make this config fairly safe to use against an existing target if you just run it on your machine without heeding my previous warning. Don't change or remove these unless you are absolutely sure you know what you are doing. There are also options besides true and false so read the docs for more info. It is far too much to cover here but the simple config above is all I need for my purposes.

	Once the service is configured you can control it with systemctl like any other systemd service. By default it will be disabled so you will need to enable it to run automatically after suitable testing. You can check the logs for startup or running errors with journalctl. If syncs are working correctly you can watch them as uploads are logged very shortly after the events are saved to local disk;		Once the service is configured you can control it with systemctl like any other systemd service. By default it will be disabled so you will need to enable it to run automatically after suitable testing. You can check the logs for startup or running errors with journalctl. If syncs are working correctly you can watch them as uploads are logged very shortly after the events are saved to local disk;

Kernelkurtz: Remove unnecessary option compress

2026-01-15T03:25:29Z

Remove unnecessary option compress

← Older revision		Revision as of 23:25, 14 January 2026
Line 44:		Line 44:
	maxProcesses = 1,		maxProcesses = 1,

	insist = true,		insist = true

	}		}
Line 74:		Line 74:
	whole_file = true,		whole_file = true,

	prune_empty_dirs = true,		prune_empty_dirs = true

	~~compress = false~~

	}		}

Kernelkurtz at 04:57, 14 January 2026

2026-01-14T04:57:45Z

← Older revision		Revision as of 00:57, 14 January 2026
Line 104:		Line 104:
	''inplace = true'' and ''whole_file = true'' Sets these rsync options and are a further optimization for S3 API use efficiency. This makes rsync's functionality less "sync" and more straight "copy", again because we can safely assume only newly created files will ever be transferred. These options used together make this method of backing up events very close in terms of total traffic cost to natively writing to S3 straight from Zoneminder. Once again if your provider does not charge for API calls this will not matter to you.		''inplace = true'' and ''whole_file = true'' Sets these rsync options and are a further optimization for S3 API use efficiency. This makes rsync's functionality less "sync" and more straight "copy", again because we can safely assume only newly created files will ever be transferred. These options used together make this method of backing up events very close in terms of total traffic cost to natively writing to S3 straight from Zoneminder. Once again if your provider does not charge for API calls this will not matter to you.

	''prune_empty_dirs = true'' Because they are watched by inotify, rsync will create empty remote directories even for events Zonmeminder deletes locally. If you are not syncing deletes you will want to use this flag, see below.		''prune_empty_dirs = true'' Because they are watched by inotify, rsync will create empty remote directories even for events Zonmeminder deletes locally. If you are not syncing deletes you will want to use this flag, see ''delete'' below.