Difference between revisions of "Realtime event backup to cloud storage"
Kernelkurtz (talk | contribs) m |
Kernelkurtz (talk | contribs) (Added prune empty dirs flag) |
||
| Line 73: | Line 73: | ||
whole_file = true, | whole_file = true, | ||
prune_empty_dirs = true, | |||
compress = false | compress = false | ||
| Line 101: | Line 103: | ||
''inplace = true'' and ''whole_file = true'' Sets these rsync options and are a further optimization for S3 API use efficiency. This makes rsync's functionality less "sync" and more straight "copy", again because we can safely assume only newly created files will ever be transferred. These options used together make this method of backing up events very close in terms of total traffic cost to natively writing to S3 straight from Zoneminder. Once again if your provider does not charge for API calls this will not matter to you. | ''inplace = true'' and ''whole_file = true'' Sets these rsync options and are a further optimization for S3 API use efficiency. This makes rsync's functionality less "sync" and more straight "copy", again because we can safely assume only newly created files will ever be transferred. These options used together make this method of backing up events very close in terms of total traffic cost to natively writing to S3 straight from Zoneminder. Once again if your provider does not charge for API calls this will not matter to you. | ||
''prune_empty_dirs = true'' Because they are watched by inotify, rsync will create empty remote directories even for events Zonmeminder deletes locally. If you are not syncing deletes you will want to use this flag, see below. | |||
Revision as of 23:55, 13 January 2026
It's possible to use lsync daemon, which leverages the inotify https://en.wikipedia.org/wiki/Inotify kernel subsystem to run triggers based on events in a directory (create, modify, delete). Triggers can be varied (you can run any arbitrary command) but in the default use case, is as a filesystem backup / clone using the rsync command.
You may also want to consider other options. Inotify has command line tools, and there is an incrond, which uses cron syntax to run commands based on inotify.
About
After recently migrating from another popular NVR software on Windows to Zoneminder on Linux I started looking for a way to do near-real-time backups of my events to offsite storage, simply as a way to have a copy should the local ZM server be compromised by break-in, fire, etc. Software such as Dropbox, Google Drive, etc. make it relatively easy to do this on Windows but the setup on Linux takes bit more effort.
I wanted to keep it as simple as possible and don't need my backup solution to interface with the ZM database. After experimenting with several other options, I have settled on an application called lsyncd. It runs as a service and leverages inotify and rsync to watch a directory or directories and mirror only new changes to another storage location. Lsyncd should be available as a package for most Linux distributions. I used the installation instructions here for my Rocky Linux system;
https://docs.rockylinux.org/10/guides/backup/mirroring_lsyncd/
Note the version of lsyncd included in EPEL for Rocky 8 (2.2.2-9) has a noticeable memory leak. Building from source solved this issue.
At this point it is worthwhile to at least skim through the docs.
https://lsyncd.github.io/lsyncd/manual/config/file/
https://linux.die.net/man/1/rsync
Both lsyncd and rsync are highly configurable. For many people the use case will be very simple like my own which I will describe here - watch my Zoneminder storage location and copy new files to a remote location, but much more complex configurations are certainly possible. In my case I'll be writing to an Amazon S3 bucket which I have already setup using S3FS-FUSE according to the documentation here;
https://zoneminder.readthedocs.io/en/latest/userguide/options/options_storage.html
You can also use an Rclone mount point as a destination. I've tested this with Google Drive but it should work with any cloud provider supported by Rclone;
Follow the instructions here to set up your cloud storage as a local mountpoint;
https://rclone.org/commands/rclone_mount/
Setup
Once lsyncd is installed you will need to create a configuration file in /etc/lsyncd.conf. I've included mine as an example below. Note this file uses Lua syntax, not Bash. Be VERY CAREFUL if you are testing remote write/sync/copy operations to a destination that already has important data on it (see links below). I highly recommend initial testing with an empty bucket or target filesystem. You have been warned.
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd-status.log",
statusInterval = 10,
maxProcesses = 1,
insist = true,
}
sync {
default.rsync,
source = "/home/Cameras",
target = "/media/aws",
delete = false,
delay = 10,
init = false,
rsync = {
archive = true,
ignore_times = true,
inplace = true,
whole_file = true,
prune_empty_dirs = true,
compress = false
}
}
logfile = is pretty self-explanatory. Be sure to set up log rotation after you are done setting lsyncd up, if required.
maxProcesses = 1 is the default. If you are syncing multiple sources or targets you may want to increase this.
insist = true allows the service to start even if the target is not ready. I got errors without this which went away when I added it. YMMV.
source = The directory where ZM events are stored.
target = The location to copy events to. This will be the mount point for the cloud storage configured previously.
targetdir = "/somedirectory" (not shown, part of sync block) subdirectory to use on the target. I'm just using the root directory in a dedicated bucket. You may not want this and should specify something here.
delay = This is how often rsync will run if there are any filesystem changes. Default is 15 seconds. For many users this will be close enough to real-time for the purpose. This can be combined with (or substituted) for another value in the settings field;
maxDelay = # (not shown, part of settings block) will queue this # of file changes before calling rsync. Between delay and/or maxDelay it is possible to tune the timing of your file copies to your exact preference. inotify waits for write_close to add files to the transfer list so there are no issues with rsync trying to upload uncompleted files. I typically only record short 10 second clips. If you use your cameras to continuously record be aware that files in the process of being written will not be copied until they are closed. It may be further possible to remove this limitation with the inotifyMode directive, though I have not tested this.
archive = true This preserves ownership and permissions on the copied files.
ignore_times = true This sets rsync's --ignore-times directive. For the purpose it is being used here it is safe to assume you will never have a newer copy of the files you are uploading already existing in your destination. Removing this check reduces API calls and is a noticeable optimization for AWS. If your cloud provider does not meter traffic you probably won't notice any difference with this either way.
inplace = true and whole_file = true Sets these rsync options and are a further optimization for S3 API use efficiency. This makes rsync's functionality less "sync" and more straight "copy", again because we can safely assume only newly created files will ever be transferred. These options used together make this method of backing up events very close in terms of total traffic cost to natively writing to S3 straight from Zoneminder. Once again if your provider does not charge for API calls this will not matter to you.
prune_empty_dirs = true Because they are watched by inotify, rsync will create empty remote directories even for events Zonmeminder deletes locally. If you are not syncing deletes you will want to use this flag, see below.
Two other very important variables in this file are;
delete = false this is also pretty self explanatory, file deletes will not be synced from source to target. In the case of my S3 bucket I have lifecycle rules that handle this. If you prefer for your remote storage location to synchronize the deletions made by Zoneminder you can set this to true. This also means if you accidentally delete events from your local storage they will be deleted from the cloud as well. Use with caution.
init = false causes the lsyncd to not do a complete initial synchronization between source and target on service startup. On S3 this is expensive in terms of API calls and may take a while. I know I'm going to miss some events during system maintenance/OS updates etc anyway so I don't consider this worth the cost, but you can set to true if you want it. Be warned it will make your target an exact mirror of your source.
Having delete and init set to false also should make this config fairly safe to use against an existing target if you just run it on your machine without heeding my previous warning. Don't change or remove these unless you are absolutely sure you know what you are doing. There are also options besides true and false so read the docs for more info. You can also add options to be added directly to the rsync calls. You can sync multiple directories to multiple targets with multiple options. There are exclude options which you may find useful. It is far too much to cover here but the simple config above is all I need for my purposes.
Once the service is configured you can control it with systemctl like any other systemd service. By default it will be disabled so you will need to enable it to run automatically after suitable testing. You can check the logs for startup or running errors with journalctl. If syncs are working correctly you can watch them as uploads are logged very shortly after the events are saved to local disk;
[root@nvr system]# tail -50 /var/log/lsyncd/lsyncd.log /1/2025-12-11/8707/8707-video.mp4 /1/2025-12-11/8707/snapshot.jpg /1/2025-12-11/8707/alarm.jpg /6/2025-12-11/8708/ /6/2025-12-11/ /6/ /6/2025-12-11/8708/8708-video.mp4 /6/2025-12-11/8708/snapshot.jpg /6/2025-12-11/8708/alarm.jpg /8/2025-12-11/8706/8706-video.mp4 /8/2025-12-11/8706/ /8/2025-12-11/ /8/ Thu Dec 11 14:30:25 2025 Normal: Finished a list after exitcode: 0 Thu Dec 11 14:31:50 2025 Normal: Calling rsync with filter-list of new/modified files/dirs /2/2025-12-11/8709/ /2/2025-12-11/ /2/ / /2/2025-12-11/8709/8709-video.mp4 /2/2025-12-11/8709/snapshot.jpg /2/2025-12-11/8709/alarm.jpg Thu Dec 11 14:32:00 2025 Normal: Finished a list after exitcode: 0 Thu Dec 11 14:32:04 2025 Normal: Calling rsync with filter-list of new/modified files/dirs /2/2025-12-11/8710/ /2/2025-12-11/ /2/ / /2/2025-12-11/8710/8710-video.mp4 /2/2025-12-11/8710/snapshot.jpg /2/2025-12-11/8710/alarm.jpg Thu Dec 11 14:32:11 2025 Normal: Finished a list after exitcode: 0 Thu Dec 11 14:32:45 2025 Normal: Calling rsync with filter-list of new/modified files/dirs /2/2025-12-11/8711/ /2/2025-12-11/ /2/ / /2/2025-12-11/8711/8711-video.mp4 /2/2025-12-11/8711/snapshot.jpg /2/2025-12-11/8711/alarm.jpg Thu Dec 11 14:32:51 2025 Normal: Finished a list after exitcode: 0 Thu Dec 11 14:37:36 2025 Normal: Calling rsync with filter-list of new/modified files/dirs /3/2025-12-11/8712/ /3/2025-12-11/ /3/ / /3/2025-12-11/8712/8712-video.mp4 /3/2025-12-11/8712/snapshot.jpg /3/2025-12-11/8712/alarm.jpg Thu Dec 11 14:37:41 2025 Normal: Finished a list after exitcode: 0 [root@nvr system]#
Note: If you receive errors similar to "maximum number of watches reached" in the logs you may need to increase the default limits on your system for /proc/sys/fs/inotify/max_user_instances and /proc/sys/fs/inotify/max_user_watches. See here for more details;
https://support.scc.suse.com/s/kb/360054835111?language=en_US
I'm happy to again have offsite copies of all my events now. While they are not tracked in the Zoneminder database, they are still saved in folders by monitor id and date for easy retrieval should the ZM database become unavailable. Props to the ZM devs for their great software, and I hope this writeup proves useful to some other users in the future.
See Also
- https://github.com/lsyncd/lsyncd/issues/668 - Beware that lsync will do rsync with deletes on by default. With some casual testing, you might inadvertedly delete your home directory or root filesystem. From the github: "I was told many times delete by default was a bad idea and was a result of the usual use case in the sense of replicating the target exactly like the source. I suppose the people telling so have a point."
- https://docs.rockylinux.org/10/books/learning_rsync/06_rsync_inotify/
- https://linuxvox.com/blog/what-is-the-proper-way-to-use-inotify/ - Using inotify-tools command line interface
- https://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/ - Probably easier than lsyncd. (Was not available in default Debian Bullseye (have to use backports https://qa.debian.org/madison.php?package=incron&table=archived&a=&c=&s=#) but is in most other Debian releases.).
- https://wiki.alpinelinux.org/wiki/Inotifyd
- https://wiki.archlinux.org/title/Incron
