What's new
PGBlitz.com

Register Now! Find useful tips, Interact /w Community Members and join the part the Best Community on the Internet!

Discussion PG9 Feedback Forum

vFlagR

Respected Member
Staff
Hey,

Been playing around with PG9 some more and found the following couple of issues:
1. .Partial~ files aren't being ignored in the move process, as soon as the file starts being copied in to /pg/transfer the tranfer/pgblitz service is copying it straight away
2. Movies are being copied in to the Shared Drive with an extra folder that matches the name of the file (screenshot below)
3. The problems I had previously with deploying clone have happened another few times. I think the solution is to manually install pip3 -- I'm going to wipe and retry again to confirm this is definitely the fix.

1567441747721.png

I appreciate I'm finding a lot of problems, I'm also trying to find solutions but It's taking a while to get up to speed with how everything links together :) Be expecting more pull requests from me as I figure things out.

Cheers!
 

sdf911

Junior Member
I can't seem to get the installer to run on ubuntu 18.04. I have root access and installed Git and Ansible first but the installer gives me a load of permission errors, any help please ( I can install 8.7.5 without any problems )
 

vFlagR

Respected Member
Staff
I can't seem to get the installer to run on ubuntu 18.04. I have root access and installed Git and Ansible first but the installer gives me a load of permission errors, any help please ( I can install 8.7.5 without any problems )
can you post the output or screenshots of the errors?
 

gadgethome

Senior Member
Moderator
Donor
I just used the same command as the normal install but substituted the version number to that of 9 and it worked for me.
 

sdf911

Junior Member
I just used the same command as the normal install but substituted the version number to that of 9 and it worked for me.
Well, i don't know what to say, after spending all day trying different methods, including the above, i've just tried again and the installer has gone through without any errors, strange.

Thanks for the replies, time for some testing :)
 

zerith

Junior Member
Donor
So I got a better understanding of the flow and seems like everything works as expected after the initial installation teething problems.

I have however noted even though I've got my throttle set to either 100MB (got 1GBPs vps) or even 1000MB, I seem to be limited at 80Mb's on uploading and 800Mb's on downloading. Speedtest shows upload on the box is what's to be expected.

I've also got 12 keys deployed just in case. Never know and not that much of an hassle to clean them.

Anyone else experiencing such an issue?
 

vFlagR

Respected Member
Staff
I've updated BlitzGCE for V9 to use the new filesystem. This isn't a code cleanup just a straight port from v8.6 -> v9. If you want to help with testing this follow the steps below:

Run the pg command then use the following numbers to navigate the menu.
  • 8
  • 1
  • 3
  • 1
  • User: vFlagR
  • Branch: v9
  • 2
Start Testing!

 

zerith

Junior Member
Donor
yeah definitely limited somewhere in the system:

2019/09/03 01:39:36 INFO : Starting bandwidth limiter at 9MBytes/s
2019/09/03 01:39:36 INFO : Starting HTTP transaction limiter: max 12 transactions/s with burst 1

2019/09/03 01:43:49 INFO :
Transferred: 619.699M / 697.739 MBytes, 89%, 8.365 MBytes/s, ETA 9s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 1m14s

I only see one connection limited at that speed.

ACTIVELY DEPLOYED: [SDrive Unencrypted]

[1] Google Account Login [********]
[2] Project Name [plexguide-251613]
[3] Client ID & Secret [ACTIVE]
[4] SDrive Label [pg-media]
[5] SDrive OAuth [ACTIVE]
[6] GDrive OAuth [ACTIVE]
[7] Key Management [12] Built
[8] SDrive (E-Mail Share Generator)

[A] Deploy SDrive Unencrypted
Throttle [1000 MB]
[C] Options
[Z] Exit

Is there something I'm missing?
 

zerith

Junior Member
Donor
I see that in the transfer.sh its standard to limit bw on the connections but rather to open multiple connections at the limited speed to ensure ban does not happen. However I cannot see why it does not open multiple connections even though there are multiple files in /pg/logs/.temp_list but /pg/logs/.transfer_list only shows 1 being transferred at a time.
 

zerith

Junior Member
Donor
Ahh, I think the two settings got moved around:

[email protected] /pg/pgclone/source # cat /pg/var/blitz.bw
1000
[email protected] /pg/pgclone/source # cat /pg/var/move.bw
9

bwg="$(cat /pg/var/blitz.bw)"
bws="$(cat /pg/var/move.bw)"

if [[ "$uread" == "sc" ]]; then encryptbit="C"; fi

rclone move "$uploadfile" "${readykey}${encryptbit}:/$truepath" \
--config /pg/rclone/blitz.conf \
--log-file=/pg/logs/pgblitz.log \
--log-level INFO --stats 5s --stats-file-name-length 0 \
--tpslimit 12 \
--checkers=20 \
--min-age=30s \
--transfers=16 \
--bwlimit="$bws"M \
--user-agent="$useragent" \
--drive-chunk-size={{dcs}} \
--exclude="**_HIDDEN~" --exclude=".unionfs/**" \
--exclude="**partial~" --exclude=".unionfs-fuse/**" \
--exclude=".fuse_hidden**" --exclude="**.grab/**"
fi

As per the code snippet, the pg shared location is referencing speed limit of 9 instead of the 1000 per configuration. The inverse is true for the gdrive.
 

vFlagR

Respected Member
Staff
Ahh, I think the two settings got moved around:

[email protected] /pg/pgclone/source # cat /pg/var/blitz.bw
1000
[email protected] /pg/pgclone/source # cat /pg/var/move.bw
9

bwg="$(cat /pg/var/blitz.bw)"
bws="$(cat /pg/var/move.bw)"

if [[ "$uread" == "sc" ]]; then encryptbit="C"; fi

rclone move "$uploadfile" "${readykey}${encryptbit}:/$truepath" \
--config /pg/rclone/blitz.conf \
--log-file=/pg/logs/pgblitz.log \
--log-level INFO --stats 5s --stats-file-name-length 0 \
--tpslimit 12 \
--checkers=20 \
--min-age=30s \
--transfers=16 \
--bwlimit="$bws"M \
--user-agent="$useragent" \
--drive-chunk-size={{dcs}} \
--exclude="**_HIDDEN~" --exclude=".unionfs/**" \
--exclude="**partial~" --exclude=".unionfs-fuse/**" \
--exclude=".fuse_hidden**" --exclude="**.grab/**"
fi

As per the code snippet, the pg shared location is referencing speed limit of 9 instead of the 1000 per configuration. The inverse is true for the gdrive.
I've opened a pull request to fix this. Until it's merged you can either manually change what I've done in my commit (literally 2 small changes in one file) or you can deploy a personal fork using my branch (username: vflagr // branch: fix_bw_for_sd_sc)

Needs tested from Gdrive users, I've only tested with Shared Drives on PG9

 

zerith

Junior Member
Donor
vFlagR do you possibly know what the cause could be that its only moving 1 file at a time? Not really sure where to start looking. I did however see someone on v8.6base having the same issue.
 

vFlagR

Respected Member
Staff
vFlagR do you possibly know what the cause could be that its only moving 1 file at a time? Not really sure where to start looking. I did however see someone on v8.6base having the same issue.
I don't off the top of my head but I have noticed sort of similar behaviour. I have a feeling it's the files being moved in to the move location incrementally rather than a problem with Rclone. (If you move 6 movies at the same time in to /pg/transfer I think all 6 will start transferring to Google at the same time like they should).

I've been able to replicate this behaviour by moving 5x 5gb files in to /pg/transfer. Rclone is only moving 1 at a time, I did not experience this on v8.6.

I'm trying to track down & fix another Rclone related bug just now so I'll have a look and see if I can find anything about files moving 1 at a time.
 

zerith

Junior Member
Donor
I'm currently looking and I have a feeling its got to do with

head -n +1 /pg/logs/.temp_list >> /pg/logs/.transfer_list
uploadfile=$(head -n +1 /pg/logs/.temp_list)

The transfer list only get 1 file from the temp upload list at a time. Still looking around it and how it would process if multiple files were to be added with say head -n +12 or something
 

vFlagR

Respected Member
Staff
I'm currently looking and I have a feeling its got to do with

head -n +1 /pg/logs/.temp_list >> /pg/logs/.transfer_list
uploadfile=$(head -n +1 /pg/logs/.temp_list)

The transfer list only get 1 file from the temp upload list at a time. Still looking around it and how it would process if multiple files were to be added with say head -n +12 or something
That looks like it could be the problem, without looking at the rest of the code yet I would guess that +1 should be set to the user limit of concurrent transfers. Going for a smoke break then I'll mess with some code and make another PR if that's the issue.
 
Assists Greatly with Development Costs

zerith

Junior Member
Donor
From what I could see your best bet with that would be to look at that segment and possibly also look at the while at line 41. That's the only looping I could pick up at a glance that incorporates the if statements for the rclone function. It might be work a try to move the done to line 99. I'll look at making a pull request myself if I find the time. Currently having some issues with patches in real life where the vendor made changes to the core functions in the system which is now breaking client specific code.
 

vFlagR

Respected Member
Staff
I spent a few hours on this and came out more confused than when I went in :D I played around with head +2/3 etc but I found that when trying that I just got a bunch of "Directory not found" errors in pgblitz.log. I'm also pretty confused about why there needs to be _.temp_list & .transfer_list as they seem to feed in to each other.

Finally, can anyone shed some light on what this snippet of code is doing? I've been through it over and over and can't quite figure it out exactly. It could be part of the problem/solution.

Bash:
while read p; do
  sed -i "/^$p\b/Id" /pg/logs/.temp_list
done </pg/logs/.transfer_list
 

zerith

Junior Member
Donor
I've also noticed a major problem with moving and having partial files.

'One Piece - S19E86 - Dark King'\''s Direct Precepts! The Battle Against Katakuri Turns Around HDTV-720p.mkv'
'One Piece - S19E86 - Dark King'\''s Direct Precepts! The Battle Against Katakuri Turns Around HDTV-720p.mkv.partial~'
'One Piece - S19E87 - He Finally Returns! Sanji, the Man Who Stops the Yonko HDTV-720p.mkv'
'One Piece - S19E87 - He Finally Returns! Sanji, the Man Who Stops the Yonko HDTV-720p.mkv.partial~'
'One Piece - S19E88 - Lurking in the Darkness! Assassin Attacks Luffy! HDTV-720p.mkv'
'One Piece - S19E88 - Lurking in the Darkness! Assassin Attacks Luffy! HDTV-720p.mkv.partial~'
'One Piece - S19E89 - A Man'\''s Resolution! Katakuri'\''s Life-Risking Great Match HDTV-720p.mkv'
'One Piece - S19E89 - A Man'\''s Resolution! Katakuri'\''s Life-Risking Great Match HDTV-720p.mkv.partial~'
'One Piece - S19E90 - Wake Up! To Cross over the Strongest Kenbunshoku HDTV-720p.mkv'
'One Piece - S19E90 - Wake Up! To Cross over the Strongest Kenbunshoku HDTV-720p.mkv.partial~'
'One Piece - S19E91 - A God Speed Fist! New Gear 4 Activation! HDTV-720p.mkv.partial~'
'One Piece - S19E92 - Finally, It'\''s Over! The Climax of the Intense Fight against Katakuri! HDTV-720p.mkv'
'One Piece - S19E93 - A Desperate Situation! The Iron-Tight Entrapment of Luffy! HDTV-720p.mkv'
'One Piece - S19E93 - A Desperate Situation! The Iron-Tight Entrapment of Luffy! HDTV-720p.mkv.partial~'
'One Piece - S19E94 - Pulling Back from the Brink! The Formidable Reinforcements – Germa! HDTV-720p.mkv'
'One Piece - S19E95 - The Last Hope! The Sun Pirates Emerge! HDTV-720p.mkv'
'One Piece - S19E96 - A Captivating Flavor! Sanji'\''s Cake of Happiness! HDTV-720p.mkv'
'One Piece - S19E96 - A Captivating Flavor! Sanji'\''s Cake of Happiness! HDTV-720p.mkv.partial~'
'One Piece - S19E97 - The Man of Humanity and Justice! Jinbe, a Desperate Massive Ocean Current HDTV-720p.mkv.partial~'
'One Piece - S19E98 - Time for Farewell! Pudding’s One Last Request! HDTV-720p.mkv'


Even though pgui has nothing to move. They do all seem empty when I look, might be a cleanup issue?
 
Assists Greatly with Development Costs

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Thread starter Similar threads Forum Replies Date
Admin9705 PG Script Questions & Bugs 1
Escondido Blitz Apps 1
Whatdafuq Ideas & Suggestions 2

Similar threads


Development Donations

 

Top NZB NewsGroups!

Members - Up To a 58% Discount!

Trending

Online statistics

Members online
12
Guests online
103
Total visitors
115
Top