What's new
PGBlitz.com

Register Now! Find useful tips, Interact /w Community Members and join the part the Best Community on the Internet!

Massive amount of data downloaded during library update

astroshare

Blitz Sergeant
Donor
Donor
Just wondering if anyone else has noticed. I'm running a test server in HyperV at home, and noticed that while Plex was doing a library update, it pulled nearly 100GB of data.
I only have one library with about 40 x 4K movies setup on this VM. The drive on the VM is only 50 gig and it never went full, so I'm not sure what all the data transfer was.
Is this normal?
 

toliman

Blitz 3rd Class
Just wondering if anyone else has noticed. I'm running a test server in HyperV at home, and noticed that while Plex was doing a library update, it pulled nearly 100GB of data.
I only have one library with about 40 x 4K movies setup on this VM. The drive on the VM is only 50 gig and it never went full, so I'm not sure what all the data transfer was.
Is this normal?
Yes.

Most likely, you have a "thumbnail" creation process enabled, either as a part of adding the files under the "Library" settings, or it's part of the maintenance under "Scheduled tasks".

you should check the PG Wiki on the recommended plex settings.

If you have a system logger like netdata with hourly stats, you can visually see what plex does when content is being 'created'; then analysed/processed, then thumbnails processed. YMMV, as plex tends to be "slow".

Plex defaults are not great for PGBlitz type applications or low-powered NAS systems. Few people notice, because plex tends to be fire and forget once set up, and background tasks tend not to be noticed unless you have an eye on the server at the time, or check the "status" / "Alerts" pages on the plex web client.

Plex has (at least) 3 optional, but default processes, chapter-thumbs, preview-thumbs and media-info analysis that occur when you add files that uses large amounts of CPU, and network/storage bandwidth.

You can (and should) turn these off on low-end Plex Servers, but also remote plex servers.

If you have 4k HEVC libraries,... chapter thumbs/preview thumbs can drag your server down to a crawl locally, and it will search through 40-90gb movies to create 5-10mb of preview images and take 4-5 hours to do it (depending on your transcode settings I believe). If you have hundreds of 4k Movies (one day...) it's not wise.

So yes, the 50gb of data will be processed at least twice, initially to parse the video/audio files. But also a second time to process incomplete, missing or damaged thumbnails during a maintenance update/library scan.

The next slowest thing is Audio processing, but that's a different topic.

With cloud storage / UnionFS, the file is stored locally until it's moved away, Thumb creation will work while the file is still present so that analysis can take place, but you don't NEED thumbnails. If you add too much content, and plex hasn't finished, it doesn't know that the file was moved, and that can cause excess transfers of moving/caching files.

There's not a simple way to handle this, so it's better not to add thumbnails, chapter-thumbs especially.

Once a video file has been added to the library, it won't need to re-analyse the file unless the data is missing, or you force a re-check. Typically.

Without Thumbnail Generation, Plex will read the first few MB of a video/audio file (settings are in the Library tab) when you add new content, enough data to "analyse" the file content when you add new content, which is the initial few MB of the video or audio file to determine the content, resolution, subtitles etc. It won't need to read the entire file, just enough to read the metadata.
 

astroshare

Blitz Sergeant
Donor
Donor
Thank you for the reply. I've never had to worry about bandwidth since I've always used a dedicated server with unlimited bandwidth. Now after reading your post and some others on the web, I've turned off these default settings and it's much better. My whole library is about 8TB, so plex scanning all of that would've killed me, thankfully I was watching my router and noticed the spike.
Thanks again
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.


Development Donations

 

Top NZB NewsGroups!

Members - Up To a 58% Discount!

Trending

Top