RAID setup?

[JET]

Alright, after losing 2+TB of data due to a 6 week old Seagate drive puking, I am going to be doing backup of some sort. I tried to research RAID's a bit, but there are like 50 different kinds.

This is for my media server, the computer runs 24/7. I run a 1TB Samsung F3 drive for the boot and program drive. I will be running two 3TB WB Red's for the main storage drives, I will use the 2TB and 3TB seagates I have right now for backup drives. I am only back up to 2.5TB of data now, but adding on fast. I will probably be around 4TB at the end of the month. I will add more Red's as needed if I am happy with them.

I am fine with backing up once a week and I would prefer the seagates to only run when backups are being performed. My 6 month old 2TB is now starting to make a little noise when worked hard.

So, what do you guys think will be the best option for backing up for me?

[Super Bleeder!!]

Why dont you go with some solid state drives?

[Trogdor]

SSD isn't really unnecessary for mass storage. The primary benefit is access speed, on a media server, that's not all that important.

[Super Bleeder!!]

I'd say the primary benefit is zero moving parts!

[JET]

I don't have $4k sitting around to spend on SSD's when I can get 3TB Red's for $150 each.

[A//// Guy]

SSD can still fail as well, just get a nice 2TB NAS device which utilizes multiple hard drives and is stand alone.

Otherwise just get 2 or more drives and use windows built in Raid to manage the drives.

[19TsiAwd90]

ssd are nice, really fast.. but they are very exspensive.. i say pick up a couple 2tb hd's and run them in raid 10(i think thats where both drives are clones or eachother) that way if one fails the other one has the same stuff on it

[JET]

Yeah, I run a 128GB SSD in the HTPC upstairs and it took about 15 sec of boot time off. It boots in 21 seconds now. I have been checking out Peter's suggestion of using Windows to run a RAID 1, which just mirrors the drives. I didn't even know Win7 would do that. Thanks Peter!

[tehehodi]

[scheides]

First off it'll be best to get your terminology straight....there's no such thing as RAIDs...RAID is a set of methods for setting up redundant disk.

Here's what I have run for the last several years:

Hardware:
4U cheapo rackmount case with room for like 10x 3.5" disks and a variety of ATX-sized motherboards
450W power supply (replaced this several times in the last 8 years)
2x 1TB Seagate disks
1x old computer guts...currently running:
-Basic ABIT motherboard
-AMD semperon 2200+
-768MB RAM
-few extra fans for good measure

So I ran mandriva linux on it forever but I swapped that out for Centos6 a while back. Disks are configured in Raid1 (mirror) with linux software raid, supported out of the box. I have samba configured to share out a directory and just copy files to/from that directory as I choose.

I have a 2TB exernal seagate disk that I wrote a simple script to mount and do full backups to once per month.

Upstairs I have a Western Digital WDTV. $99 and it does more than my appleTV ever did right out of the box.

The whole setup is super easy to maintain and cheap as hell....if I ever need more space I'll just convert to a striped RAID5 setup or multiple RAID1 mirrors. 1TB has proven to be plenty for me, I don't need to keep old copies of 80% of the movies that land on it, I'll just re-download them if I really want them again.

You could skip the USB drive thing and just run backups to an internal disk but that adds a little risk I guess.

What services do you need for your setup? Does it have to run windows? The biggest thing is to just keep it dust free and not let the hardware get *too* old as that just is asking for problems with data corruption. And once you get a setup that works, don't mess with it too much. I touch mine about once every two years but I think I'm on like year 4 of this setup.

[JET]

I prefer it to run Win7 as right now I am using media player to send files to a PS3 and I also have a HTPC that sees the most playback. I am curious about the WDTV, if that can take the place of the PS3 then I am open to that.

HTPC runs Win7 AMD phenom 2 @ 3.8ghz with 4GB ram and a 128GB SSD.

The computer downstairs stores all the files, but it also used as a desktop. It is an i5 3570k, 16GB ram, 12 TB of disk space once my red's get here, etc.

All the movies are HD, with most files being 8-24GB per movie. Everything is hardwired on a gigabit network.

All of the file sharing works great, except many of the new movies have Cinavia protection and the PS3 has issues playing them. The HTPC has no issue with them.

[bramagedained]

http://en.wikipedia.org/wiki/Standard_RAID_levels

Running just the disks as standalones and doing regular backups might be your best bet.

It sounds like you have too much storage for RAID 1 to be reasonable because it's too inefficient.

RAID 5 is popular, but, there is some concern about recovering from a disk failures now that disks are so big. As your odds of bad sectors among big disks is reasonably high. Data density has increased a lot in the past few years, but, the number of bad sectors/GB(or sectors that will fail over the life of a drive) hasn't really gone down a whole lot.

The 'cool' setup is what a friend of mine did, and likely what I will copy when I get around to making an actual file server. He runs a dedicated RAID controller that takes up a PCI slot with (I think) 10 2TB disks connected running a RAID 6 setup. Its really fast and can recover from multiple drive failures. It's probably somewhat excessive for home use. We had a couple of gigabit LANs setup with ~30 users in the barracks that pulled from it for streaming movies and such.

I don't/haven't worked in a datacenter or anything just what I've delt with for some of the government systems I've supported in the past where people would(did) get really angry if the data was lost.

[19TsiAwd90]

Originally Posted by JET

Yeah, I run a 128GB SSD in the HTPC upstairs and it took about 15 sec of boot time off. It boots in 21 seconds now. I have been checking out Peter's suggestion of using Windows to run a RAID 1, which just mirrors the drives. I didn't even know Win7 would do that. Thanks Peter!

face palm raid1 is what i was refering to not raid 10..

[scheides]

What do you use the PS3 for versus HTPC? Like what are some of the end goals you have for the whole setup? Why are your files SO huge? You'd be surprised at the quality of a nicely ripped/encoded 1.4GB movie file and even most blue-rays are under 5GB uncompressed. 8+GB seems just redonk man. Like I said I have a total of 1TB of storage and I have all seasons of Top Gear and a bunch of other tv shows, at least a hundred movies, and off-site backups of mitsystyle.com, scheides.com, etc.

Check out the WDTV or better yet ask Spazzy about it since he's around you a bit more...it's so badass for the money! All I use it for is watching um, legal copies of dvds and blue-rays I already own, and youtube (now youtube 'leanback'). It has netflix support and hulu plus as well, all in full 1080p.

http://www.wdc.com/en/products/products.aspx?id=330

I actually have the 'WD TV Live Plus' but I purchased it almost 3 years ago...I think the most basic model now does all the same stuff but is physically slightly smaller.


Don't forget that with RAID you loose some percentage of total disk space, that's the trade-off for redundancy. RAID6 is a great option but you gotta really pony up to make the investment for a bunch of similar/identical disks and upgrading is gonna just as painful down the road.

[scheides]

Also, just because I am all about KISS, you can REALLY do an out-of-the-box home entertainment setup with a WDTV and one of these:

http://www.amazon.com/Personal-Cloud.../dp/B00AGI7I2O

They offer a 4, 6, and 8TB model...all do RAID1 (as far as I can tell). The 8TB model is only $625 which is actually a pretty screaming deal...the drives are user-serviceable too so if one of them takes a digger you can just swap it out yourself.

*edit* 8TB is in JBOD mode, the 8TB model would be 4TB in RAID1 (as far as I can tell).

Add in data your favorite torrent site or some ripped media from any laptop and you're golden.

[JET]

The PS3 is just used as a media player because I don't have a 2nd HTPC. My files are so huge because I only play HD movies at full frame rate. 18+GB is full bluray quality. This part I do know a lot about, check the frame rate on the supposed 1080p movies if they are under 5GB. I play them on a 50" and anything under 3GB looks like crap. You can also tell the difference between a 8GB file and a 16GB, just like an upconverted DVD vs a Bluray.

I'll have to explore the set top boxes some more, it looks like there are 17 models out there now.

It looks like RAID 1 would be the best option, but I think I might just get a JBOD box and just do a backup once a month. With RAID 1 I wouldn't be able to unplug the backed up drives, would I? I only want them running when I do a backup once a month. I don't trust the reliability of the seagate drives to stay on all the time.

[scheides]

Hah well I guess not being a 'videophile' is worth it for me, dealing with that much data just seems like a huge PITA. Then again that's what I do on a daily basis and anything less than a 200TB infrastructure-grade disk array is just janky basement shit to me (not worth spending much of my free time on).

In any raid set you need to leave them all running all the time or you will be wasting your time scrubbing and frankly risk the other disk failing. You will also loose the read performance bennifit which frankly you may need to rely upon if you insist at pumping that much data; a single disk may not be able to keep up. You will need a whole extra drive to back your main storage pool up.

Keep in mind with JBOD (no raid) you will have to rebuild the entire array if any of the disks fail. Just like cars, do it right, do it once, more movies and less 'wrenching' time.

[JET]

But if I run no RAID and use just JBOD, I would just have to restore the one drive that fails if I use the same size drives in both the PC and the JBOD, right? As I understand it is basically a bunch of external HD's in one enclosure.

I am beginning to think I should just use one of these on my eSATA port and pop my backup HD's on whenever I want to do a backup.

http://www.newegg.com/Product/Produc...82E16817153112

That is exactly why I started this thread, I was going around between the options and couldn't really figure out which one was the best.

[scheides]

A true JBOD will take 'a bunch of disks' and present logical volumes to the network, with not a ton of regard to what it puts where. That's why I mentioned the warning about huge and lengthy restore jobs.

Now that being said, with what you have described you have a win7 machine that you use for a desktop or whatever...throw disks in it and share them out individually. Add the latest and greatest 'big' single disk that meets your budget when you need more space. File server, done.

For backups, I use an almost identical enclosure to the one you linked. When you buy a big disk above, budget for a second one the same size or bigger and dump your files to it (and your other secondary disks that match your other drives) and store in a safe, dry place.

Write some bat scripts (or better yet bash after you install Cygwin) and make the process easy on yourself.

This is a pretty simple setup but still plenty of room for failure and still not sure about I/O performance. Let me know what you end up doing!

[JET]

Alright, that sounds like the best option then. The drives only run when I have them on the dock and I can leave the dock hooked up all the time. I could honestly probably just do a copy from the main drive to the backup and tell it not to copy the files that are already there. There will only be about 300 files per drive max.