IOwait Issues

Assists Greatly with Development Costs

risbo

Blitz 2nd Class
Original poster
32
4
Basically I just want to say i'm not blaming anything or anyone, i'm just creating this thread for a discussion and some ideas to try and nut out a cause and maybe find a solution.

The TLDR of my issue is that post media download, be it via usenet or torrent, there is a massive iowait level occuring (see below)
3488

My thought process is something along the lines of the following;
Media downloads to incomplete folder > media moves to complete folder > sonarr/radarr handles the media renaming accordingly > rclone(?) then copies(?) the files to /mnt/move/blah > rclone uploads file to gdrive/tdrive/gcrypt/tcrypt > rclone then deletes the local copy of the file.

From what I can tell there is alot of unnecessary(in my opinion) file movements, double handling, copies etc happening that would be causing stress on the the hard drive.

A little about my setup;
Host Setup
ESXi 6.7
HP ML350 G9
48GB DDR4 ECC
7 x HP 10K 450GB SAS Drives in raid 1+0 with a hot spare (datastore)
1 x Samsung Evo (something) 500GB SSD (datastore)
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz

PGblitz VM
Ubuntu 18.04.2 LTS
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz (10 vCPU)
16GB Ram Assigned
489GB allocated to the SSD datastore

Basically I'm just after some advice with what to possible change or test.

Over the weekend I will likely try to remove the use of the incomplete folders thus removing one of the file movements.

Anyway, appreciate any ideas or thoughts.

Thanks
 

risbo

Blitz 2nd Class
Original poster
32
4
During some testing, It's directly related to the importing that sonarr/radarr performs it would appear.
Renders the entire system unusable.
 

TheShadow

Blitz 2nd Class
Staff
42
14
During some testing, It's directly related to the importing that sonarr/radarr performs it would appear.
Renders the entire system unusable.
It shouldn't, if it is then your downloader mappings are not correct for sonarr/radarr and it has to copy files.

Additionally, rclone moves the imported sonarr/radarr files, it doesn't copy them. moving from /mnt/incomplete to /mnt/downloads is just hardlinks, so no IO there either.

The files from /mnt/downloads forward are hardlinked, which is basically no IO needed.

Using a HDD instead of an SSD will have more IO wait. Practically everything about pgblitz is IO heavy. Downloading, unpacking, scanning, streaming, uploading all use storage device.

How long does radarr/sonarr take importing items (easily check this w/ a manual import). it should take no more than 5s or so.

Let's say you manually import a 60gb remux. You should see the sizes rapidly change.

Check before import: du -h /mnt/downloads/sabnzbd
then manual import file
then run du -h /mnt/downloads/movies and keep running.

You should see the 60gb file move, not copy. If it copies, imports take a long time. The only case where it will copy and not move, is torrents. Depending on the torrent app and the status of the torrent, it will copy the file, not move it. This is deliberate part of arr so you can keep seeding.

On top of that you are using an encrypted mount, which requires 2x the IO of unencrypted mount. If your downloader mappings are correct (see wiki pages), and you verified arr is moving the files.

You are most likely just bottlenecked by your HDD. You are downloading + encrypting + uploading at minimum after a download. If a scan or streams are playing then that HDD needle has to dance around your HDD for all the different apps/services which makes IO wait longer.
 

TheShadow

Blitz 2nd Class
Staff
42
14
Make sure you set up the remote mappings for each downloader in sonarr/radarr.

 
  • Like
Reactions: timekills

risbo

Blitz 2nd Class
Original poster
32
4
So after performing the above steps I have the following occurring (all after a full reboot);
3492

Below are my remote path mapping settings in Sonarr/Radarr respectively;
3493
3494

In an effort to try and make clear how the file structure works I've placed a file into each folder 'remote' and 'local' to try and determine if my setup is still incorrect. Also there are other files in these folders that I figured it would be best to not share and draw attention for obvious reasons.

Bash:
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/unionfs/sabnzbd/example.file
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/unionfs/qbittorrent/example2.file
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/downloads/sabnzbd/example3.file
[email protected]:/mnt/downloads/sabnzbd$ touch /mnt/downloads/qbittorrent/example4.file


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/unionfs/sabnzbd/ |grep example
-rwxrwxr-x  1 risbo risbo     0 Jun 14 08:59 example3.file*
-rwxrwxr-x  1 risbo risbo     0 Jun 14 08:58 example.file*


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/unionfs/qbittorrent/ |grep example
-rwxrwxr-x 1 risbo risbo     0 Jun 14 08:58 example2.file*
-rwxrwxr-x 1 risbo risbo     0 Jun 14 09:00 example4.file*


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/downloads/sabnzbd/ |grep example
-rw-rw-r--  1 risbo risbo     0 Jun 14 08:59 example3.file


[email protected]:/mnt/downloads/sabnzbd$ ll /mnt/downloads/qbittorrent/ |grep example
-rw-rw-r-- 1 risbo risbo     0 Jun 14 09:00 example4.file
Based on the above my understanding is that anything placed into /mnt/downloads/* is instantly linked to /mnt/unionfs/* and not the other way around.

My SAB is configured in a more or less vanilla setup out of the box from PGBlitz with very minimal adjustment (as below)
3495

I'm questioning if the incomplete folder may be causing issues now?

I've also tried disabling direct unpack within SAB with no improvement.

Appreciated your time on discord yesterday @TheShadow

EDIT: Obviously if it ends up being a hardware issue then so be it, I just expected better performance from an SSD even if its somewhat old.
 

Oohdang

Blitz 3rd Class
4
2
I'm having a very similar issue where my server just slows to an absolute crawl any time Sonarr/Radarr/Lidarr are actively downloading/processing/etc. My CPU/RAM usage remains low during these slowdowns. I too have verified my remote path mappings are set exactly the way the wiki instructs in all 3 download programs (I actually went in and direct copy/pasted from the wiki just to be certain).

I'm using a single SSD on a local server to do everything and was thinking maybe I just need to get a second SSD and point Sonarr/Radarr/Lidarr to it and stop using the system drive. Importing files for me is definitely not instant - depending on how large it has taken up to several hours to import a batch. I actually just ordered a second SSD but reading this thread and @TheShadow stating that imports should take seconds has me wondering if that won't actually fix my problem...
 

Oohdang

Blitz 3rd Class
4
2
@risbo I use nzbget with a vanilla pgblitz setup and had a similar problem - I had /mnt/downloads/nzbget set as my destination directory and /mnt/incomplete/nzbget set as my intermediate directory. This is the blurb about what this option does from nzbget:
If this option is set (not empty) the files are downloaded into this directory first. After successful download of nzb-file (possibly after par-repair) the files are moved to destination directory (option DestDir). If download or unpack fail the files remain in intermediate directory.

Using of intermediate directory can significantly improve unpack performance if you can put intermediate directory (option InterDir) and destination directory (option DestDir) on separate physical hard drives.

NOTE: If the option InterDir is set to empty value the downloaded files are put directly to destination directory (option DestDir).
So I deleted the /mnt/incomplete/nzbget from the InterDir and it seems to have fixed my problem. No more extreme server slow down. Now my only problem is my 250GB SSD keeping up with my internet connection. Maybe trying this in SABnzb will work for you.
 

risbo

Blitz 2nd Class
Original poster
32
4
So I deleted the /mnt/incomplete/nzbget from the InterDir and it seems to have fixed my problem. No more extreme server slow down. Now my only problem is my 250GB SSD keeping up with my internet connection. Maybe trying this in SABnzb will work for you.
So I tried this, and sabnzbd wouldnt accept a blank value, so it would appear no matter what that a file is moving. What I dont understand is why its adding load to the IO as a move 'should' be instantaneous.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads


Development Donations

 

Trending