What's new
PGBlitz.com

Register Now! Find useful tips, Interact /w Community Members and join the part the Best Community on the Internet!

Discussion An alternative, replacement or additional backup solution to Plexguide backup

plex_noob

Senior Member
Staff
Donor
PG Version
6.087
Server Type
Remote - Dedicated Server
Although being quite satisfied by the standard backup tool in Plexguide, I was looking for additional functionalities currently not available.

This backup process could be combined with the traditional Plexguide one in order to guarantee a full recovery in case of trouble.

Among other added values of this solution, here are the one I was interested in:
- Real backup software with advanced options such as backup resuming, encryption, backup reporting, ...
- Backup reporting and logging
- Backup deduplication which should greatly reduce the time required to upload the data to Google drive. This is especially true for the Plex and Radarr volumes

This process uses the excellent job of Steffen Bleul with his volumerize docker project (https://github.com/blacklabelops/volumerize).

- Create the folders structure
- Obtain Google drive API "Client ID" and "Client Secret" (https://console.developers.google.com./)
- Get authorisation code from Google to let the program access your Google Drive

-************************************************
Folder structure creation:
-************************************************
mkdir -p /opt/appdata/volumerize/cache
mkdir -p /opt/appdata/volumerize/log
mkdir -p /mnt/gdrive/volumerize/credentials

-************************************************
Google Authorisation:
-************************************************

Bash:
docker run -it --rm \
-v /opt/appdata/volumerize/cache:/volumerize-cache \
-v /mnt/gdrive/volumerize/credentials:/credentials \
-v /opt/appdata/netdata:/source:ro \
-e "VOLUMERIZE_SOURCE=/source" \
-e "VOLUMERIZE_TARGET=gdocs://[email protected]/volumerize" \
-e "GOOGLE_DRIVE_ID=<your google drive client id>" \
-e "GOOGLE_DRIVE_SECRET=<your google drive client secret>" \
blacklabelops/volumerize backup
The purpose of the previous command aims to request access Google to access your drive:
Copy/Paste the URL found in your terminal in your browser
Copy/Paste the result in the terminal window
Press enter
This process should create 2 additional files in the credentials folder on Google drive:
- cred.file
- googledrive.cred

!! The cache folder is linked to the credentials created during the previous command. If for whatever reason you need to recreate new credentials files you need to clean the cache folder
rm /opt/appdata/volumerize/cache/*

Run the following docker command to create/start the real Volumerize docker
-************************************************
Creating the container
-************************************************

Bash:
docker run -d \
--name volumerize \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/appdata/volumerize/cache:/volumerize-cache \
-v /mnt/gdrive/volumerize/credentials:/credentials \
-v /opt/appdata/volumerize/log:/logs \
-v /opt/appdata/container1:/source/container1:ro \
-v /opt/appdata/container2:/source/container2:ro \
-v /opt/appdata/container3:/source/container3:ro \
-e "VOLUMERIZE_SOURCE=/source" \
-e "VOLUMERIZE_TARGET=gdocs://[email protected]/volumerize" \
-e "VOLUMERIZE_CONTAINERS=container1 container2 container3" \
-e "VOLUMERIZE_FULL_IF_OLDER_THAN=7D" \
-e "VOLUMERIZE_JOBBER_TIME=0 0 8 * * *" \
-e "TZ=EUROPE/BRUSSELS" \
-e "GOOGLE_DRIVE_ID=<your google drive client id>" \
-e "GOOGLE_DRIVE_SECRET=<your google drive client secret>" \
-e "VOLUMERIZE_DUPLICITY_OPTIONS=--log-file /logs/bakup.log" \
blacklabelops/volumerize
Explanation:
"-v /var/run/docker.sock:/var/run/docker.sock": Allows Volumerize to stop/start the containers
"-v /opt/appdata/volumerize/cache:/volumerize-cache": Create a local docker volume for the backup processing before upload
"-v /mnt/gdrive/[email protected]/credentials":/credentials: Refer to the authorization file previously created
"-v /opt/appdata/volumerize/log:/logs": (Create a local folder to store the backup logs)
"-v /opt/appdata/MyContainerName1:/source/MyContainerName1:ro"
"-v /opt/appdata/MyContainerName2:/source/MyContainerName2:ro":
"-v /opt/appdata/MyContainerName2:/source/MyContainerName2:ro": Mounts each container volume corresponding to each container to backup in read-only mode. One container per line
-e "VOLUMERIZE_SOURCE=/source": This is a required variable.
-e "VOLUMERIZE_TARGET=gdocs://[email protected]/volumerize": Specifies the target for the backups
-e "VOLUMERIZE_CONTAINERS=container1 container2 container3": Lists all the containers associated with the container volumes referenced earlier. Each container name separated by space.
-e "VOLUMERIZE_FULL_IF_OLDER_THAN=7D": Forces a full backup each 7 days
-e "VOLUMERIZE_JOBBER_TIME=0 0 8 * * *": Perform a daily backup @ 8 am
-e "TZ=EUROPE/BRUSSELS": Specifies the container timezone
-e "VOLUMERIZE_DUPLICITY_OPTIONS=--log-file /logs/bakup.log": Here we can specify additional options for duplicity. See here for additional information.

-************************************************
Testing the backup process manually
-************************************************
- To manually test/run a backup
Code:
docker exec volumerize backup
This will run the backup once out of the predefined schedule

-************************************************
Checking the backup logs
-************************************************
cat /opt/appdata/volumerize/log/bakup.log

-************************************************
List of all of your previous executed backups
-************************************************
- To list the backups
Code:
docker exec volumerize list
-************************************************
Restoring a container
-************************************************
- To restore a docker volume (e.g: ombi)
Code:
docker run --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/appdata/ombi:/source/ombi \
-v  /mnt/gdrive/volumerize/credentials:/credentials \
-v /opt/appdata/volumerize/log:/logs \
-e "VOLUMERIZE_SOURCE=/source" \
-e "VOLUMERIZE_TARGET=gdocs://[email protected]/volumerize" \
-e "VOLUMERIZE_DUPLICITY_OPTIONS=--log-file /logs/restore.log" \
-e "VOLUMERIZE_CONTAINERS=ombi" \
blacklabelops/volumerize restore

Some comments:
- If you encounter errors or if you restart the install procedure you need to clean the credentials and cache folder otherwise you will get error messages and it won't work.
- If you plan to backup more than 1 server which is my case, you simply have to create a sub-folder in all the references to the GDRive. This means mainly the reference for the credential folder (/mnt/gdrive/credentials => /mnt/gdrive/MyServerName/credentials) and the reference to the backup folder (VOLUMERIZE_TARGET=gdocs://Gmail_Email_Address/volumerize => VOLUMERIZE_TARGET=gdocs://Gmail_Email_Address/volumerize/MyServerName)
-
 
Last edited:

plex_noob

Senior Member
Staff
Donor
Yes, for sure I use it.

---- Automatically Merged Double Post ----

If help needed don't hesitate to ask.

Denis
 
Last edited:

LckyLuciano

Junior Member
Thanks for this. I've been looking for a more elegant backup solution. I am having some trouble with the google authorization step and hope you can help. I get an "invalid reference format" which I think is a syntax error. Am I missing something? Screenshot below of my entry with some redactions.

volumerize setup.png
 

plex_noob

Senior Member
Staff
Donor
Hi,

Your second volume should look something like this:
-e /mnt/pgdrive/volumerize/credentials:/credentials

I think this is the problem
 

LckyLuciano

Junior Member
Hi,

Your second volume should look something like this:
-e /mnt/pgdrive/volumerize/credentials:/credentials

I think this is the problem
Thank you. That makes sense. I'll give it another shot!

Edit: That was the problem, thanks!
 
Last edited:

bubbadk

Legendary Member
Staff
and if it could have a webui so you can see if it's stuck or works. that would be great :)

This system is awesome. but it needs deduplication so we shouldn't backup so large files everytime.
 

plex_noob

Senior Member
Staff
Donor
Deduplication is the main advantage in comparison with the PG Backup. Especially for the Plex Database which size reaches several GBs.
 

timekills

Legendary Member
Staff
Donor
-************************************************
Google Authorization:
-************************************************
Run this command from the command line:

docker run -it --rm \
-v /opt/appdata/volumerize/cache:/volumerize-cache \
-v /mnt/gdrive/<your backup folder on gdrive> + /credentials:/credentials \
-v /opt.appdata/netdata:/source:ro \
-e "VOLUMERIZE_SOURCE=/source" \
-e "VOLUMERIZE_TARGET=gdocs://[email protected]/<your backup folder on gdrive>" \
-e "GOOGLE_DRIVE_ID=<your google drive client id>" \
-e "GOOGLE_DRIVE_SECRET=<your google drive client secret>" \
blacklabelops/volumerize backup


This command does not backup anything. It simply request the authorization code from Google to access your drive.
Copy/Paste the URL in your browser and Copy/Paste the result in the terminal window and press enter.
Normally you should have a 2 additional files in the credentials folder on Google drive.
Recommend using code tags so the spacing is correct in the post. i.e.

Code:
docker run -it --rm \
    -v /opt/appdata/volumerize/cache:/volumerize-cache \
    -v /mnt/gdrive/volumerize/credentials:/credentials \
    -v /opt/appdata:/source:ro \
    -e "VOLUMERIZE_SOURCE=/source" \
    -e "VOLUMERIZE_TARGET=gdocs://[email protected]/<your backup folder on gdrive>" \
    -e "GOOGLE_DRIVE_ID=<your google drive client id>" \
    -e "GOOGLE_DRIVE_SECRET=<your google drive client secret>" \
    blacklabelops/volumerize backup
Also, I believe the line "-v /opt.appdata/netdata:/source:ro \" should be
-v /opt/appdata:/source:ro \

1) replace opt.apdata with opt\appdata
2) The appdata files all get mounted in the container at \source\AppName in the next portion. So the directory mounted to source should be \opt\appdata

Also, I think a backslash is required at the end of each line in the Docker creation portion.

Even after this, I'm getting a significant amount of errors, python type, when trying to run the backup or list. But I'm not certain I have the scripts wired tight yet.
 
Last edited:

LckyLuciano

Junior Member
Recommend using code tags so the spacing is correct in the post. i.e.

Code:
docker run -it --rm \
    -v /opt/appdata/volumerize/cache:/volumerize-cache \
    -v /mnt/gdrive/volumerize/credentials:/credentials \
    -v /opt/appdata:/source:ro \
    -e "VOLUMERIZE_SOURCE=/source" \
    -e "VOLUMERIZE_TARGET=gdocs://[email protected]/<your backup folder on gdrive>" \
    -e "GOOGLE_DRIVE_ID=<your google drive client id>" \
    -e "GOOGLE_DRIVE_SECRET=<your google drive client secret>" \
    blacklabelops/volumerize backup
Also, I believe the line "-v /opt.appdata/netdata:/source:ro \" should be
-v /opt/appdata:/source:ro \

1) replace opt.apdata with opt\appdata
2) The appdata files all get mounted in the container at \source\AppName in the next portion. So the directory mounted to source should be \opt\appdata

Also, I think a backslash is required at the end of each line in the Docker creation portion.

Even after this, I'm getting a significant amount of errors, python type, when trying to run the backup or list. But I'm not certain I have the scripts wired tight yet.

Here is what I was ultimately able to get working:

To get the credential files created, I used this. In line 4, I think you can pick any existing appdata folder, so I used Deluge for mine and it seemed to work.

Code:
docker run -it --rm \
-v /opt/appdata/volumerize/cache:/volumerize-cache \
-v /mnt/gdrive/volumerize/credentials:/credentials \
-v /opt/appdata/deluge:/source:ro \
-e "VOLUMERIZE_SOURCE=/source" \
-e "VOLUMERIZE_TARGET=gdocs://<removed>/volumerize" \
-e "GOOGLE_DRIVE_ID=<removed>" \
-e "GOOGLE_DRIVE_SECRET=<removed>" \
blacklabelops/volumerize backup
I checked my GDrive drive folder and it did create two files in the volumerize/credentials directory after running the part above so I moved on.

For the next part, I had to create some folders for the cache and log directories.

Code:
sudo mkdir -p /opt/appdata/volumerize/cache
sudo mkdir -p /opt/appdata/volumerize/log
I changed the owner of the top-level folder to my plexguide user that runs most of my services, but I'm not sure if this is necessary.

Code:
sudo chown -R plexguide:plexguide /opt/appdata/volumerize
Then I ran the part to create the docker container. This required a good bit of editing to include all my application folders and container names.

Code:
docker run -d \
--name volumerize \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/appdata/volumerize/cache:/volumerize-cache \
-v /mnt/gdrive/volumerize/credentials:/credentials \
-v /opt/appdata/volumerize/log:/logs \
-v /etc/localtime:/etc/localtime:ro \
-v /opt/appdata/deluge:/source/deluge:ro \
-v /opt/appdata/jackett:/source/jackett:ro \
-v /opt/appdata/lazylibrarian:/source/lazylibrarian:ro \
-v /opt/appdata/monitorr:/source/monitorr:ro \
-v /opt/appdata/nzbget:/source/nzbget:ro \
-v /opt/appdata/nzbhydra2:/source/nzbhydra2:ro \
-v /opt/appdata/ombi:/source/ombi:ro \
-v /opt/appdata/organizrv2:/source/organizrv2:ro \
-v /opt/appdata/plex:/source/plex:ro \
-v /opt/appdata/portainer:/source/portainer:ro \
-v /opt/appdata/radarr:/source/radarr:ro \
-v /opt/appdata/sonarr:/source/sonarr:ro \
-v /opt/appdata/synclounge:/source/synclounge:ro \
-v /opt/appdata/tautulli:/source/tautulli:ro \
-v /opt/appdata/traefik:/source/traefik:ro \
-e "VOLUMERIZE_SOURCE=/source" \
-e "VOLUMERIZE_TARGET=gdocs://<removed>/volumerize" \
-e "VOLUMERIZE_CONTAINERS=deluge jackett lazylibrarian monitorr nzbget nzbhydra2 ombi organizrv2 plex portainer radarr sonarr synclounge tautulli traefik" \
-e "VOLUMERIZE_FULL_IF_OLDER_THAN=7D" \
-e "VOLUMERIZE_JOBBER_TIME=0 0 5 * * *" \
-e "TZ=AMERICA/CHICAGO" \
-e "GOOGLE_DRIVE_ID=<removed>" \
-e "GOOGLE_DRIVE_SECRET=<removed>" \
-e "VOLUMERIZE_DUPLICITY_OPTIONS=--log-file /logs/backup.log" \
blacklabelops/volumerize
Then I tested it with sudo docker exec volumerize backup. So far so good. It ran this morning at 5:00 AM on it's own and created a small incremental backup. I hope this helps.
 

timekills

Legendary Member
Staff
Donor
I was wondering if we should stop containers manually first so there are no errors with active database backup, but I see that the containers get stopped when the backup script runs.

They also appear to be restarted once complete.

1) Is it better to manually stop/start them, or just let the script do it's thing?
2) Any tips on the deduplication commands that we should try placing in the opt/appdata/volumerize/logs/bakup.log"
(As mentioned in @plex_noob OP, specify additional options for duplicity (http://duplicity.nongnu.org/duplicity.1.html#sect5))
 
Last edited:

LckyLuciano

Junior Member
I was wondering if we should stop containers manually first so there are no errors with active database backup, but I see that the containers get stopped when the backup script runs.

They also appear to be restarted once complete.

1) Is it better to manually stop/start them, or just let the script do it's thing?
2) Any tips on the deduplication commands that we should try placing in the opt/appdata/volumerize/logs/bakup.log"
(As mentioned in @plex_noob OP, specify additional options for duplicity (http://duplicity.nongnu.org/duplicity.1.html#sect5))
1) Anything you have in this environment variable will be stopped before the backups are run, and will start up again when all the backups are complete. I don't see any reason to stop them manually.

-e "VOLUMERIZE_CONTAINERS=deluge jackett lazylibrarian...

2) I think deduplication is already a component of this by default. For the partial backups that run daily, only the files that were changed since my last backup were uploaded. When the backup finished, a small report was provided with a tally of the differences between the current and previous backup. I didn't change any of the duplicity commands other than the log file name.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads


Development Donations

 

Top NZB NewsGroups!

Members - Up To a 58% Discount!

Trending

Online statistics

Members online
4
Guests online
113
Total visitors
117
Top