Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
ground control to msata (idea)
12-17-2012, 08:22 AM (This post was last modified: 12-17-2012 08:47 AM by Paul.)
Post: #1
Lightbulb ground control to msata (idea)
hi i was thinking would it be useful, (or possible) for something like this?

eg where you could rightclick on a folder (or select it via dashboard) and select an option called "preload to msata".

the data would already be on the drobo 5d, but doing this preloading would try to "buffer?" or store as much of that folder (and subolder if needed), onto the msata, possibly wiping anything already stored within it or just using unused space) so that you can access and work with those files as quickly as possible.

(maybe when you finish, you can "flush and use last known msata data before last flush" command to set it back to normal internal msata usage/optimisation)

The scenario's i think this might help (in theory) are for mini-projects.
eg, if you use a drobo normally (eg doing what we do without much thought, it might start to recognise usage pattens and pre buffer things.

but, imagine you start doing some mini projects, such as,
catalogs of family photos,
image manipulation in batches
small audio video clips etc

this way you could tell the drobo that you effectively would like it to "make that Specific data accessible and usable in the most optimal way", and then you go have dinner and when you come back you can work on your project with more efficiency, and only for the sake of a couple of clicks Smile

(i need to read up more about how it decides what to do regarding data aware tiering etc and maybe erik pounds youtube clips will cover the "logic" tree behind it but was an idea so far) Smile

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
12-17-2012, 06:33 PM (This post was last modified: 12-17-2012 06:34 PM by mgriffin34.)
Post: #2
RE: ground control to msata (idea)
Might be helpful if you read up on what Data Tiering is...
http://en.wikipedia.org/wiki/Automated_Tiered_Storage

It moves blocks of data, not files. That's the whole point. If you are working on a project, the Volume Manager (CoreStorage, ZFS, etc.) will detect that and move it to the fastest storage pool with no interaction from the user. Why would you want to introduce an I/O error?

i7 Mac Mini Server, 10.10
Drobo (Gen2), USB, 4x WD 4TB RED

Drobo (Gen3), USB3, 4X WD 4TB RED

Drobo 5D, TB, 5x WD 4TB RED, 120GB Intel 525 mSATA SSD
Find all posts by this user
Quote this message in a reply
12-17-2012, 09:45 PM
Post: #3
RE: ground control to msata (idea)
You're assuming that the Drobo implementation is similar to such systems. This is much more of a cache than data tiering - data isn't stored permanently on the SSD, only a copy of it.

Drobo 5N | 250GB Samsung 850 EVO mSATA | 2 x 4TB Seagate, 3 x 4TB Hitachi | FS/EXT3 diskpack
Peak performance >100MBps read/write (based on FS disk pack, no jumbo frames, no SSD)
DroboPorts: Plex, Transmission, OpenSSH, NFS, nano, screen, bash
Find all posts by this user
Quote this message in a reply
12-18-2012, 09:48 AM (This post was last modified: 12-18-2012 10:00 AM by Paul.)
Post: #4
RE: ground control to msata (idea)
edit (thanks for the link i'll read up more)

for me im thinking more in terms of a "ramdisk" concept,

eg, if youre tagging photosnaps as a mini project, chances are you'll be doing it as a once off (or at least not again anytime soon)
so by being able to pre-cache all the several hundred files (relatively low-size ones) into the ssd to be able to have the quickest access time when you sit down to do them, sounds useful to me.

(maybe in practice, and using modern computing, it "might" be quick enough to process them as is, but if you do lots of random file processing, and duplicate searching, or outdoor similarity searches, and thumbnail/contact sheets etc etc, it seems like it would make a difference in access/process times)
also if it were to be done in blocks, how does a drobo know what blocks all your data sets are part of in advance?
eg, if you have a wallpaper folder, full of images, and start accessing some files inside it, would it also recognise the other ones?
and if you plan on running a search/process on other related folders, would it know that you wanted to do something on them too? (some folders might reside in a parent folder) others might reside in a 2nd parent folder, etc)
- how would it know, what you "plan" on doing, unless you select those 2 parent folders and say "optimise these" Smile
my guess would be that the first time an msata is installed, is that it would run a basic 2-step logic.
- find all files smaller than xMB
- copy all such files onto msata, with priority on files accessed most recently first.
then it could further learn and adapt, maybe by using self measurement techniques, and if it finds certain files taking too long to deliver, it stores some of those too.

(at least that's how i would probably design it for an out the door 1st use, until i learn more about it) Smile

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
12-18-2012, 12:21 PM
Post: #5
RE: ground control to msata (idea)
In my mind it's kind of like SuperCache/prefetch - and at least in Windows world, an operating-as-intended prefetch makes a huge difference.

--Brandon | WHS2011+Drive Bender/2x Drobo v2/Drobo S G1/ Drobo S G2/Transporter
Drobo provides fault-tolerance, it's NOT a substitute for regular backups.
Drobo Best Practices - Official and Community-sourced.
Find all posts by this user
Quote this message in a reply
12-18-2012, 07:26 PM
Post: #6
RE: ground control to msata (idea)
The main optimization I'd like to see is for directory data to be cached. At least on network units, the biggest slowdown is in actually browsing the folder hierarchy. Of course, that gets complex as they'd have to integrate fairly heavily with netatalkd and samba to make it work.

Drobo 5N | 250GB Samsung 850 EVO mSATA | 2 x 4TB Seagate, 3 x 4TB Hitachi | FS/EXT3 diskpack
Peak performance >100MBps read/write (based on FS disk pack, no jumbo frames, no SSD)
DroboPorts: Plex, Transmission, OpenSSH, NFS, nano, screen, bash
Find all posts by this user
Quote this message in a reply
12-19-2012, 10:27 AM (This post was last modified: 12-19-2012 10:34 AM by Paul.)
Post: #7
RE: ground control to msata (idea)
diamond you might find this useful...
i managed to convert bhiga into using it lol Smile

its called cathy by a guy called r vasicek, it catalogues your volumes and then you use it like file manager and search etc, and when you find what you want you can open the file (or folder path) straight from there without waiting for folder trees.

http://www.drobospace.com/forums/showthr...http://www.drobospace.com/forums/showthread.php?tid=2204&highl
it might help you in the meantime, and its free.

(at one point the author was asking for postcards but i think he received so many its not a requirement anymore) Smile

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
12-19-2012, 10:36 AM
Post: #8
RE: ground control to msata (idea)
My brain thinks "tags" (an item can belong to multiple categories) by my fingers and technology likes linear trees...

Works great for stuff you infrequently use.

--Brandon | WHS2011+Drive Bender/2x Drobo v2/Drobo S G1/ Drobo S G2/Transporter
Drobo provides fault-tolerance, it's NOT a substitute for regular backups.
Drobo Best Practices - Official and Community-sourced.
Find all posts by this user
Quote this message in a reply
12-19-2012, 10:46 AM
Post: #9
RE: ground control to msata (idea)
and so do some database tables
"remember to break the crows feet" Smile

(btw i have XP home SP2, a Drobo v1 with 2x 1TB/2x 1.5TB WD greens, & a bkp Drobo v2 with the same + a DroboShare: unused)
& a DroboS v2 with 3xWD15EADS &2x1TB in DDR mode on win7, & a drobo5D (all usb)
  • btw i did a sustained (write) operation for about 6 hours, and got 13.2MB / sec ...objection? "sustained" :)
    (16.7MB/s on a v2 & 47-96MB/s drobo-s)
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: