What's new
PGBlitz.com

Register Now! Find useful tips, Interact /w Community Members and join the part the Best Community on the Internet!

IOwait Issues

R

ribso

Guest
Basically I just want to say i'm not blaming anything or anyone, i'm just creating this thread for a discussion and some ideas to try and nut out a cause and maybe find a solution.

The TLDR of my issue is that post media download, be it via usenet or torrent, there is a massive iowait level occuring (see below)
3488

My thought process is something along the lines of the following;
Media downloads to incomplete folder > media moves to complete folder > sonarr/radarr handles the media renaming accordingly > rclone(?) then copies(?) the files to /mnt/move/blah > rclone uploads file to gdrive/tdrive/gcrypt/tcrypt > rclone then deletes the local copy of the file.

From what I can tell there is alot of unnecessary(in my opinion) file movements, double handling, copies etc happening that would be causing stress on the the hard drive.

A little about my setup;
Host Setup
ESXi 6.7
HP ML350 G9
48GB DDR4 ECC
7 x HP 10K 450GB SAS Drives in raid 1+0 with a hot spare (datastore)
1 x Samsung Evo (something) 500GB SSD (datastore)
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz

PGblitz VM
Ubuntu 18.04.2 LTS
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz (10 vCPU)
16GB Ram Assigned
489GB allocated to the SSD datastore

Basically I'm just after some advice with what to possible change or test.

Over the weekend I will likely try to remove the use of the incomplete folders thus removing one of the file movements.

Anyway, appreciate any ideas or thoughts.

Thanks
 
R

ribso

Guest
During some testing, It's directly related to the importing that sonarr/radarr performs it would appear.
Renders the entire system unusable.
 
T

TheShadow

Guest
During some testing, It's directly related to the importing that sonarr/radarr performs it would appear.
Renders the entire system unusable.
It shouldn't, if it is then your downloader mappings are not correct for sonarr/radarr and it has to copy files.

Additionally, rclone moves the imported sonarr/radarr files, it doesn't copy them. moving from /mnt/incomplete to /mnt/downloads is just hardlinks, so no IO there either.

The files from /mnt/downloads forward are hardlinked, which is basically no IO needed.

Using a HDD instead of an SSD will have more IO wait. Practically everything about pgblitz is IO heavy. Downloading, unpacking, scanning, streaming, uploading all use storage device.

How long does radarr/sonarr take importing items (easily check this w/ a manual import). it should take no more than 5s or so.

Let's say you manually import a 60gb remux. You should see the sizes rapidly change.

Check before import: du -h /mnt/downloads/sabnzbd
then manual import file
then run du -h /mnt/downloads/movies and keep running.

You should see the 60gb file move, not copy. If it copies, imports take a long time. The only case where it will copy and not move, is torrents. Depending on the torrent app and the status of the torrent, it will copy the file, not move it. This is deliberate part of arr so you can keep seeding.

On top of that you are using an encrypted mount, which requires 2x the IO of unencrypted mount. If your downloader mappings are correct (see wiki pages), and you verified arr is moving the files.

You are most likely just bottlenecked by your HDD. You are downloading + encrypting + uploading at minimum after a download. If a scan or streams are playing then that HDD needle has to dance around your HDD for all the different apps/services which makes IO wait longer.
 
T

TheShadow

Guest
Make sure you set up the remote mappings for each downloader in sonarr/radarr.

 
R

ribso

Guest
So after performing the above steps I have the following occurring (all after a full reboot);
3492

Below are my remote path mapping settings in Sonarr/Radarr respectively;
3493
3494

In an effort to try and make clear how the file structure works I've placed a file into each folder 'remote' and 'local' to try and determine if my setup is still incorrect. Also there are other files in these folders that I figured it would be best to not share and draw attention for obvious reasons.

Bash:
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/unionfs/sabnzbd/example.file
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/unionfs/qbittorrent/example2.file
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/downloads/sabnzbd/example3.file
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/downloads/qbittorrent/example4.file


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/unionfs/sabnzbd/ |grep example
-rwxrwxr-x  1 risbo risbo     0 Jun 14 08:59 example3.file*
-rwxrwxr-x  1 risbo risbo     0 Jun 14 08:58 example.file*


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/unionfs/qbittorrent/ |grep example
-rwxrwxr-x 1 risbo risbo     0 Jun 14 08:58 example2.file*
-rwxrwxr-x 1 risbo risbo     0 Jun 14 09:00 example4.file*


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/downloads/sabnzbd/ |grep example
-rw-rw-r--  1 risbo risbo     0 Jun 14 08:59 example3.file


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/downloads/qbittorrent/ |grep example
-rw-rw-r-- 1 risbo risbo     0 Jun 14 09:00 example4.file
Based on the above my understanding is that anything placed into /mnt/downloads/* is instantly linked to /mnt/unionfs/* and not the other way around.

My SAB is configured in a more or less vanilla setup out of the box from PGBlitz with very minimal adjustment (as below)
3495

I'm questioning if the incomplete folder may be causing issues now?

I've also tried disabling direct unpack within SAB with no improvement.

Appreciated your time on Pgblitz yesterday @TheShadow

EDIT: Obviously if it ends up being a hardware issue then so be it, I just expected better performance from an SSD even if its somewhat old.
 

Oohdang

Junior Member
I'm having a very similar issue where my server just slows to an absolute crawl any time Sonarr/Radarr/Lidarr are actively downloading/processing/etc. My CPU/RAM usage remains low during these slowdowns. I too have verified my remote path mappings are set exactly the way the wiki instructs in all 3 download programs (I actually went in and direct copy/pasted from the wiki just to be certain).

I'm using a single SSD on a local server to do everything and was thinking maybe I just need to get a second SSD and point Sonarr/Radarr/Lidarr to it and stop using the system drive. Importing files for me is definitely not instant - depending on how large it has taken up to several hours to import a batch. I actually just ordered a second SSD but reading this thread and @TheShadow stating that imports should take seconds has me wondering if that won't actually fix my problem...
 

Oohdang

Junior Member
@risbo I use nzbget with a vanilla pgblitz setup and had a similar problem - I had /mnt/downloads/nzbget set as my destination directory and /mnt/incomplete/nzbget set as my intermediate directory. This is the blurb about what this option does from nzbget:
If this option is set (not empty) the files are downloaded into this directory first. After successful download of nzb-file (possibly after par-repair) the files are moved to destination directory (option DestDir). If download or unpack fail the files remain in intermediate directory.

Using of intermediate directory can significantly improve unpack performance if you can put intermediate directory (option InterDir) and destination directory (option DestDir) on separate physical hard drives.

NOTE: If the option InterDir is set to empty value the downloaded files are put directly to destination directory (option DestDir).
So I deleted the /mnt/incomplete/nzbget from the InterDir and it seems to have fixed my problem. No more extreme server slow down. Now my only problem is my 250GB SSD keeping up with my internet connection. Maybe trying this in SABnzb will work for you.
 
R

ribso

Guest
So I deleted the /mnt/incomplete/nzbget from the InterDir and it seems to have fixed my problem. No more extreme server slow down. Now my only problem is my 250GB SSD keeping up with my internet connection. Maybe trying this in SABnzb will work for you.
So I tried this, and sabnzbd wouldnt accept a blank value, so it would appear no matter what that a file is moving. What I dont understand is why its adding load to the IO as a move 'should' be instantaneous.
 

nachobel

Senior Member
Staff
Donor
OP - the touch experiment you did shows everything is working. When you touch a file into /mnt/unionfs/* you'll find it appears in /mnt/move as it's queued for upload.

As for the IO wait times, I'm running a two drive setup where things download to a SSD and then unpack to an HDD where they are imported in arr and queue for upload. I've got about a 40% IOWait lag constantly when downloading a lot of files.

This doesn't slow the process down though, and things unpack/upload fast enough not to create limit problems (I've got a 500 gig SSD running the OS, so there's about 300 gigs free normally, and then a 2 TB HDD for processing).

I see that huge spike up to 70% or so - perhaps there was an error unpacking? If possible, perhaps try to get a second drive loaded and then see if that helps alleviate your issue?

Also, you'll find that once a file downloads and begins unpacking, it's mirrored into the /mnt/unionfs/sabnzbd/ folder, and then once arr imports it, it'll go into the /mnt/downloads/Plex and /mnt/move/Plex folders (really the same folder, since unionfs has merged them). It's not so much copying back and forth but it can look like it based on how the drive structure is set up.

Anyway, hope you figure out what's wrong (if anything is) or circumvent it somehow. My setup has been running for a few months now with, essentially, zero hang ups.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads


Top NZB NewsGroups!

Members - Up To a 58% Discount!

Development Donations

 

Online statistics

Members online
7
Guests online
110
Total visitors
117
Top