Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Large file deletions VERY slow
03-24-2017, 01:29 AM (This post was last modified: 03-24-2017 01:41 AM by rosscarlson-1.)
Post: #1
Large file deletions VERY slow
Two main issues, first is as the subject says - deleting large files is VERY slow. I have the following drive setup and am using a Windows 10 PC. All updates and firmware up to date. Drobo and PC are connected to a gigabit switch, PC is gigabit.

2 x 4 TB
3 x 3 TB

So a total of 12TB and then 9 or so after RAID. I have about 8 TB of that in use, maybe a bit more (less than 10% free in case that matters). A large chunk of this is movies that range from 1GB - 15GB. Deleting a single large file, in this range, is extreeeeeeeemely slow. I've tried from Windows explorer with just delete and with shift delete, no difference. It estimates forever, shift or not. I've tried a single file at a command line, just "del file.txt" and "del /q /f file.txt" Also to toss in if I delete a directory with say 5-20 small files and folders that deletes very quickly, just as I'd expect, no issues at all. Moving large/huge files (to another folder on the Drobo) is basically instantaneous. I've seen some suggestions about using Cygwin but don't currently have that installed.

The secondary problem is when this happens it almost freezes the drobo, it doesn't stop responding to ping but if I'm watching one of the movies it always freezes or otherwise crashes. Say 30 seconds later when the file is finally deleted (say a 10GB file) everything goes back to normal.

Thoughts? I can't imagine what I'm doing wrong as this is really simple stuff.

Thanks,

-Ross
Find all posts by this user
Quote this message in a reply
03-24-2017, 08:10 PM (This post was last modified: 03-24-2017 08:10 PM by Paul.)
Post: #2
RE: Large file deletions VERY slow
hi ross, can i check if you are using single (SDR) drive redundancy, or dual (DDR) mode currently?

for files in general, while small files always seem to add some overhead to file transfers etc, even the larger files can also need some time to carry out space reclamation and to free-up blocks (and as far as i understand it, large files will need more time to free up the blocks because large files are using up and spanning across more blocks)....

with this is mind though, i would have thought the deletion operation to be fairly quick by comparison, and then for the space reclamation process to run more slowly in the background. (though i must admit i havent created or deleted huge files for ages, and so far all file deletions have been on das units)

if you could help confirm which mode you are using at the moment, and possibly some screenshots of your dashboard windows/tabs (such as the 5 drive bays) and the volumes/capacity screens, and to upload them to imgur or similar when you get a moment, that could be useful (though please do remember to rub out any sensitive / serial numbers before uploading to play safe)

at the moment though, something about low space, or a drive encountering errors could also be a cause.... can you see any messages if you click on each drive bay such as Healed or Warning at all?

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
03-24-2017, 10:57 PM
Post: #3
RE: Large file deletions VERY slow
Thanks for getting back to me - the machine that has Drobo Dashboard installed is offline for a bit, if needed I can absolutely get you screenshots but I think for now I can answer most questions.

Single drive redundancy is what's setup. I am somewhat low on space, meaning I get an orange light next to the top drive (which ironically is one of the 4TB that can't use all it's space due to the other 3's). I'm at about 10% or so free, maybe a touch below. There are 3 shares, one for apps (unused), one generic one (don't recall the name but empty) and my primary share under just "public". Everything is basically wide open in terms of security, while on my primary network that network is firewalled from my main network that then connects to the internet. To confirm there isn't, or shouldn't be any network issue here the machine attempting to delete the file is on the same local network as the Drobo, both in the same gigabit switch (port 1 and 2). I've juggled the ports around just to check, if I do a copy from the Drobo to the machine it's definitely getting gigabit speed, I don't recall the exact numbers but file transfers are *very* fast.

The files in question are in the 8-20GB range per file, so rather large. When I delete a small regular file, say a simple text file WHAM, it's gone instantly. I use SHIFT+DEL on the keyboard on one of the huge files and it's minutes before it even starts to delete, it just says that it's calculating (I assume how long deletion will take). If I do the same at the command prompt it still takes the same huge amount of time. Then when it finally "starts" to delete it takes say 1-4 seconds for it to start then the bar progress to done and the window finally closes. That whole process takes say 30-600 seconds - yeah, a huge range but that's what it does.

My assumption is this is something Windows is doing but as it's a single file I'm not really sure what to change or what to check. Is there any other information I can provide? Would you still like to see the screenshots you requested or any other logs, etc? My gut says we won't find a fix for this but I wanted to ask before I continue forward - I have about 100 or so of those huge files to delete and wanted to see what we found before I went for it. I'll likely move them all into one directory and schedule a task to kill them as when Windows "freezes" it stops doing thinks like streaming video, etc. and since I get all my entertainment from my Drobo I'll schedule that for late hours (don't want my gf to get all pissed that I "turned the TV off")

Thanks for your help Paul, really do appreciate it!

~Ross
Find all posts by this user
Quote this message in a reply
03-25-2017, 09:14 AM
Post: #4
RE: Large file deletions VERY slow
Did a little testing and wanted to share a bit more information. Here's an example of the very slow deletions.

1 Folder contains
22 Files, 500MB each
10.7GB Total

I then browse to that folder using a command prompt (inside the folder where the 22 files reside)
Execute a simple "del *.*" and type a "y" to signify all files (I know this can be scripted but wanted simple)
I then let this run and timed it:

1 minute in video streaming from the Drobo freezes
(streaming a file from SMB to local Windows Media Player on separate laptop over WiFi)

1:50 to delete all 22 files, 10.7 GB worth

Then when the file delete finishes everything performance wise goes right back to normal. If I do another folder just like this the same thing happens with the same times (within a 5% or so margin). So it's not just single huge files that are very slow it seems that anything I'd consider large, 500MB+ is also experiencing the exact same behavior.
Find all posts by this user
Quote this message in a reply
03-27-2017, 05:39 PM (This post was last modified: 03-27-2017 05:41 PM by Paul.)
Post: #5
RE: Large file deletions VERY slow
thanks for the clarification, and more info ross,
from your test and posts, a few things come to mind...

- as any devices or drives get full, they can slow down considerably (but i always thought this was some kind of magic barrier whenever drives or devices reached 95% full.... and often remembered windows defrag advising not to run a defrag unless more than 5% of the drive was free) Smile

would you be able to try freeing up a bit more space (admitedly even if this process will start off slowly), just so that you can see 16% free space in dashboard overall? am wondering if there will be a speed increase once the drobo comes out of its yellow alert (which i imagine it is showing a solid yellow light, maybe orangy in colour though not red?)

- can i also check, if all of the times you have been trying this deleting, always have been during playback and streaming?
(just if that was the case, i was wondering what happens if you were to try deleting something similar, but when not accessing or streaming anything?) - this may be good to try even before freeing up some space above...

- do you have any other programs running, or configured, such as an aggressive antivirus which tries to fully scan the contents of any file being deleted, or content indexing like search indexing? (in case you have that too, it might be worth trying a test without it enabled just to see)

- if the problems still seem to be the same, and if you get a moment, what happens if you try connecting the drobo directly to your computer (without any other networks or hubs or switches etc) just as a test?

(for the part about scheduling the deletions though, it's understandable to try to do those when the slowdown will not interfere with tv time, but hey, maybe if the tv time happened to need to be slowed or halted for a bit, it could always open up a chance for some other activities with the gf to take place) - i didnt say anything Smile
(activities like cooking or reading of course) Big Grin

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
03-27-2017, 08:26 PM
Post: #6
RE: Large file deletions VERY slow
Again Paul many thanks for your help and input here, really appreciate it. Especially for a rather old product. Answers below in-line

(03-27-2017 05:39 PM)Paul Wrote:  ...would you be able to try freeing up a bit more space (admitedly even if this process will start off slowly), just so that you can see 16% free space in dashboard overall?

I could try that but it would be rather time consuming given the amount and size of the files we're talking about. Let me do some math later and see what I could move for the moment, honestly I don't think there is much - at least not much I want to risk losing.

(03-27-2017 05:39 PM)Paul Wrote:  ...the times you have been trying this deleting, always have been during playback and streaming?

No playback at time of issue, machine is basically idling doing next to nothing. CPU/RAM/NET are all running rather flat with very little going on. Don't think it's some kind of conflict with multiple apps reading the same file.

(03-27-2017 05:39 PM)Paul Wrote:  ...any other programs running, or configured, such as an aggressive antivirus which tries to fully scan

Nope, not at all. Indexing is disabled, Plex only updates it's media library when I start it manually (every few hours is the default) and I don't even run with antivirus installed. It's a very isolated machine, I work in IT/security so easy enough for me to keep it clean. Hell I rebuild it every 3-6 months away ;-)

(03-27-2017 05:39 PM)Paul Wrote:  ...what happens if you try connecting the drobo directly to your computer (without any other networks or hubs or switches etc) just as a test?

My gut says I'd get the same but as John Cusack said "sometimes my guts have sh!t for brains!". I'll give that a try soon'ish, it's not trivial to get behind my Drobo and machine.

(03-27-2017 05:39 PM)Paul Wrote:  ...scheduling the deletions though, it's understandable to try to do those when the slowdown will not interfere with tv time...

I almost never schedule jobs that delete data. Just too much could go wrong if I'm not watching it, hell I'd miss one little switch or add the wrong one and WHAM, I'd wake up to all media gone. That would be a bad day.

If you have any other thoughts let me know. Seems my to-do list is to connect the Drobo direct to the PC and/or try to purge the array down to 16% to see what that does.

~R
Find all posts by this user
Quote this message in a reply
03-29-2017, 03:05 PM (This post was last modified: 03-29-2017 03:05 PM by Paul.)
Post: #7
RE: Large file deletions VERY slow
thanks ross,
ok sure, please let us know how things go, (and yes you are right, i meant to say manual scheduling... automatic scripts or deletions are risky in any method - especially powerful scripts like i think recently happened with amazon s3)

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
04-02-2017, 09:19 AM
Post: #8
RE: Large file deletions VERY slow
Did this work? I'm having the same exact problem. It's taking about 5-10 seconds to delete each file. I have tens of thousands to delete. The drive is very full. Only about 250GB left of 10TB.
Find all posts by this user
Quote this message in a reply
04-03-2017, 01:39 PM
Post: #9
RE: Large file deletions VERY slow
hi jimerb,
can i check what your values and % are showing in dashboard for used and free space?
is it only 2 or 3% free?

there is usually a main slowdown once the 95% used threshold is reached, and unless ive done the maths wrong, i think you might be at about 97 or 98% used, and with 1 light solid red in colour? if that is the case, once you are able to delete another couple of percent's worth of data (or thereabouts) it should become much more responsive and go back into the 1 solid yellow light, though it may take some more slower deleting in order to get there...

...am not sure if you are deleting via some kind of recycling bin or trash can, but it might be a bit quicker overall (at least on windows) if some files are selected for deletion, and then as long as you are happy with the selected files, you could try a shift delete, to bypass any recycling bins, as that has saved a bit of time for me in the past.

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: