PG Blitz v2

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
PG Blitz v2
NOTE:  When updating for a newer version of blitz beta, you must redeploy it for the changes to go into effect!
V 2.7 (Merged into 8.0.15)

Added clean folders execution for /{hd}/pgblitz/upload after upload is done to catch more folders (outside of the cron job)
Added bandwidth controls for PGBlitz

Been requested feature for awhile




V2.6

Fixed the cron for properly clean command. Just run PG and it will work automatically

V2.5

UnionFS only used /mnt/move.  Fix made is for 2nd HD users.  When deploying pgblitz, it will mount /(hdpath)/move so that is a new fix.

 V2.4

Added Clone Clean. Cleans empty folders every hour
Sped up PG Moves sleep time

V2.3

BlitzUI removed; does not work with new version
Remove pglog command

replace with ‘blitz’ command which shows current transfer as shown in picture below


files are now...
Continue reading...
 
Last edited:

thepj

Blitz Sergeant
Staff
106
53
I can validate this is making the uploads not only go faster, but nzbget +sonarr/radarr seems to be moving faster as well, and not waiting as long for the files. Looks like it is doing more in parallel as well.
 
  • Like
Reactions: Admin9705

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
I can validate this is making the uploads not only go faster, but nzbget +sonarr/radarr seems to be moving faster as well, and not waiting as long for the files. Looks like it is doing more in parallel as well.
Good feedback. I'm working it some more. Keep reporting. Goal is to replace the other one soon as possible.
 

bubbadk

Blitz Samurai
Staff
347
75
How much will I have to start over with if I switch to v8. All I need is plex and tautulli to work. The rest can I setup again ?
 
  • Like
Reactions: Admin9705

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
so bubbadk, you can backup plex and restore. in update 8.0.11, i enable the unionfs path, so your plex will work. tautulli... that shouldnt' be an issue at all. i just started. i took my sonarr and did mass path change from /unionfs/tv to /mnt/unionfs/tv and that was easy.
 

Simpleton

Blitz 2nd Class
Donor
46
6
observations on pgblitz v2.3 beta:

move of files from /mnt/move to /mnt/pgblitz/upload seems very very slow.

When uploads starts, speed looks very good. Maxing out my line. Looks like it transfers 16 files in parallel, but I only see one rclone process. Smooth.

However, uploads being suspended while moving chunks to /mnt/pgblitz/upload could be a significant performace impact. Especially while moves are so slow (they don't seem to get moved by "mv"?

Good job guys! Keep up the good work!
 

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
observations on pgblitz v2.3 beta:

move of files from /mnt/move to /mnt/pgblitz/upload seems very very slow.

When uploads starts, speed looks very good. Maxing out my line. Looks like it transfers 16 files in parallel, but I only see one rclone process. Smooth.

However, uploads being suspended while moving chunks to /mnt/pgblitz/upload could be a significant performace impact. Especially while moves are so slow (they don't seem to get moved by "mv"?

Good job guys! Keep up the good work!
ah ok, so confused on the last part. thanks for the update :D

ok think I know, so what happens is files are being moved, then uploaded, then moved, then uploaded. this is to ensure there are no lost files and etc, including better logging. so technically, nothing is suspended. it's just the order it works.
 
Last edited:

Simpleton

Blitz 2nd Class
Donor
46
6
Further observations on pgblitz 2.3 beta:
local move to /mnt/pgblitz/upload was 112GB (I guess that it tries to hit around 100G and not 100G exactly).
Then upload started.
When upload was finished "blitz" command reported:
2019/01/08 13:55:13 INFO :
Transferred: 68.905G / 68.905 GBytes, 100%, 30.683 MBytes/s, ETA 0s
Errors: 0
Checks: 175 / 175, 100%
Transferred: 175 / 175, 100%
Elapsed time: 38m19.6s
Some strange things:
1. /pgblitz/upload still contains a lot of (now empty" directories)
in /mnt/tcrypt/ the same directories exists, but they are also empty on upload side :/ (not good!) looks like they are all empty (unfortunately du -sk * does not seem to work on /mnt/tcrypt)
2. no "new" 100G chunk is moved to upload, so upload stops after the first 100G (which is actually good, as it looks like data integrity is NOT ok)

EDIT: Turns out a new move DID start. It was just so slow and the wait for it to start was so long, that I thought it had stopped.
EDIT: also, it it not strange that the first upload chunk was 112G but the blitz report said 68.905G and then stopped?
 
Last edited:

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
Further observations on pgblitz 2.3 beta:
local move to /mnt/pgblitz/upload was 112GB (I guess that it tries to hit around 100G and not 100G exactly).
Then upload started.
When upload was finished "blitz" command reported:
2019/01/08 13:55:13 INFO :
Transferred: 68.905G / 68.905 GBytes, 100%, 30.683 MBytes/s, ETA 0s
Errors: 0
Checks: 175 / 175, 100%
Transferred: 175 / 175, 100%
Elapsed time: 38m19.6s
Some strange things:
1. /pgblitz/upload still contains a lot of (now empty" directories)
in /mnt/tcrypt/ the same directories exists, but they are also empty on upload side :/ (not good!) looks like they are all empty (unfortunately du -sk * does not seem to work on /mnt/tcrypt)
2. no "new" 100G chunk is moved to upload, so upload stops after the first 100G (which is actually good, as it looks like data integrity is NOT ok)
ya i haven't put a piece for the empty directories, that easy to fix (was put in originally for /mnt/move) i'll patch later. keep typing blitz and you'll notice the size shift. i don't understand why rclone doesnt push it all out, breaks it up.
 

Simpleton

Blitz 2nd Class
Donor
46
6
what is the safest way to stop pgblitz, hopefully without loosing any further data?
will systemctl stop pgblitz do it? Will I then loose any ongoing transfers?
 

Simpleton

Blitz 2nd Class
Donor
46
6
pgblitz local move to /mnt/pgblitz/upload this time was 115G.
Then It started upload.
blitz status showed 65.732G to transfer (exactly half of what was moved to upload?
an IMMIDIATE du -sh * on /mnt/pgblitz/upload now shows 65G remaining (where do the other 65G go while they are uploading? They are still ongoing. )
 

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
no nothing is lost. because it just moves it. you can do systemctl stop pgblitz, check /mnt/move for the rest of the data. no commands are place to delete any data
 

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
@Simpleton try

df -h /mnt

see if your disk size looks right by what you have. the rclone logs are not putting out all the correct information, but yet all my contents are uploaded.
 

Simpleton

Blitz 2nd Class
Donor
46
6
Looks like it might be down to missing logging as you say.

Can't go by df -h /mnt as my /mnt/move is a mounted 13TB NFS filesystem from the old server.
I stopped pgblitz with systemctl stop bgblitz, but files are still turning up in /mnt/tdrive/ and there is still an rclone process running.
Guess I am struggeling with understanding the workflow.

As I understood it, it was like this:
/mnt/move contains files to be uploaded
100G chunks (approx) are taken from /mnt/move by pgblitz and moved (but how? with rclone?) locally to /mnt/pgblitz/upload
rclone upload from /mnt/pgblitz/upload is then started (but how? because it looks like the files are gone in an instant, but not all of them. Approx 65-69GB chunks.)
then when upload is done, an new cycle is started.
 

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
it attempts to move 100gb into /mnt/pgblitz/upload, then uploads. why the logs kept shifting, don't know, but I'll look into. but if you let it run, it does all show up. I have to dig into what Rclone's output does
 
Last edited:
  • Like
Reactions: Simpleton

sparkyowns

Blitz 3rd Class
3
0
with blitz 2.4 this started happening. can't find the files on any of the directories to manually delete them. has cause NZBGet to stop working because of a full drive.blitz 2.4.jpg
 

skijbal

Blitz 1st Class
Patron
51
11
How often does blitz run? What’s the threshold for uploading? Does it upload immediately after download? I had 139GB waiting to upload but it wouldn’t take off.
 

Admin9705

Administrator
Original poster
Project Manager
5,297
1,800
every second when idol and not uploading. no after hit's /mnt/move. which version? if using regular, use the beta edition. fixes those stuck issues. beta v2.6 blitz out.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads