Technical help (External hard drives)

James / February 25th, 2016

Technical issues are always common. They’re the biggest time waster. I spend most of my days on Google, searching for solutions for various problems I encounter. Usually this goes on quietly behind-the-scenes, but this time I’d like to open it up, at the expense of sounding desperate. Maybe someone in the Cinemassacre community has some helpful suggestions. And if not, don’t worry, I am still always researching. I doubt this relates to most people, unless you work in media. Also, I use Mac. It seems everybody else uses PC’s. Long ago, I was told to get a Mac, and I’ve been stuck to my ways ever since. I can try to adapt to new software, but when it’s a whole different operating system, no.

 

The topic I wanted to ask about is external hard drives. A big part of making videos is data management. How do you store your files? As the years go on, making videos, I accumulate more data. Currently, I am working on a giant 30TB Lacie drive. That is correct. I said 30 Terabytes! Currently, it’s about 25TB full, and I nervously anticipate the day when I need to find something larger.

 

In the old days, after a drive became full, I would just buy another, but here’s where it gets complicated. In my new projects, I am constantly referencing old projects. It saves me a lot of time to be able to go back and recycle an old image, a video clip, a sound effect, or anything, whenever needed. I need instant access to all my data at all times. The only exception is the AVGN Movie which is too huge, it takes up about 20TB on it’s own, which is why it’s on a separate drive. Before I’ve had a stack of hard drives sitting next to my computer, all daisy chained together, each with their own power supply, and it became a total mess.

 

My current drive is 30TB, though technically, it is an enclosure housing 5 separate hard drives that are formatted as “Raid 0” so they all operate together, and appear on the computer as one drive. They came pre-formated as Raid 5, meaning it doesn’t use the full capacity, and instead duplicates data for protection, showing up at 23TB capacity. The Mac Disk Utility wouldn’t let me reformat. I remember this confused me for a week, and drove me completely nuts. I eventually found out that Lacie uses it’s own Raid Manager software. Even then, it was very confusing to use. I’d rather never go through that again.

 

I’ve been fine for a while, but lately I’ve had a few scary power outages. Some were regular power outages, but one time something else strange was going on. When I tried to export a certain project in Adobe Premiere, believe it or not, it actually shut down the computer instantly. It would happen halfway through the same project, over and over again. I’ve never seen anything like that, in all my years of editing video. Everyone says Premiere is better than Final Cut, but I have yet to believe it. Been using it for a year now and it’s been a disaster (That’s a whole other story). Before anyone says to update my shit, I am using the latest version of Premiere, and it’s a new Mac Pro. Even my battery backup failed to prevent the computer (and drive) from shutting down instantly. And after you’ve worked for so long on a video, there’s nothing more aggravating.

 

Let’s say a power outage or whatever kills my hard drive. It hasn’t happened yet, but let’s say IF it does. My data would not be lost, because I backup constantly. I have many smaller hard drives, that I often copy to, and they are sent outside the home, stored elsewhere. But they are very disorganized. If my 30TB drive fails me, it would take months to copy it all again onto one single large drive, so that I could access it all conveniently and efficiently again.

 

I tried buying the same drive as before, just for another large backup, and I tried figuring out the Raid Manager for the last time, which seems to be outdated since the last time I bought one of those drives. I wasted an entire day on it. I couldn’t figure it out. I’m sick of hunting down drivers and firmware, and shit. I want a drive that you can plug in, and start up, without a problem.

 

To sum it up, the question is, any hard drive recommendations?
-Must work on Mac
-Must be easy to format
-Preferably Thunderbolt 2 AND USB 3.0 (So I have the options) OR just one or the other I guess.
-30TB or more! I know it’s crazy.

Links to products would be helpful. That’s all. Or anyone in the same situation to relate, would be nice to hear any thoughts.

 

Regardless, I’ve been working on new videos/projects which I hope you’ll get to see sooner or later. One project in particular, I’m very happy about. It’s definitely something different.

Comments

  • February 25, 2016 at 4:13 pm

    RAID 0 is a TERRIBLE IDEA. If even one of those hard-drives fail, all data on all of the drives becomes impossible to recover. It’s extremely risky, it increase the chance of losing all your data 5 times. You should get a copy of that data onto a different storage medium ASAP.

    I don’t have a recommendation for a specific hard drive, although using RAID 5 or something similar is recommended. With RAID 5, one hard drive can fail, but the combined RAID remains functional, and you can replace the broken disk to make it fully backed up again (although hard drives tend to break at similar times – so it might be time to replace all of them then).

    • February 25, 2016 at 4:15 pm

      Managed to skip the part where you said you had backups on smaller drives somehow, but the rest of the comment still stands. It’s well worth the cost of 1 disk of storage space wasted to prevent the entire raid from dying 🙂

      • February 25, 2016 at 4:27 pm

        Thanks, and yes it’s backed up. My big issue is capacity. I need 30TB or more in one volume, so I can fit all my stuff, without having to plug in several drives at the same time. When I did that, it was a mess.

      • Brandter

        February 26, 2016 at 3:52 am

        I would recomend a larger NAS with raid 5 (or 6 if you really want to feel secure.
        There are rack mounted NAS solutions that can support up to 240TB or even more if you just need the space. Like D-Link DSN-6120 (just an example). You will have to be connected to it on your network but that seems like a no brainer in this day n’ age.

      • Robert Kulesza

        February 26, 2016 at 10:53 pm

        If the point is lots of storage at low cost without regards to anything else, you could feasibly take all of the spare drives you have laying around and make your own big RAID drive.

      • March 3, 2016 at 12:32 pm

        I would build out a rack mount server with a SAS controller and get like x8 6tb HDD, bonded network (This is basically double the bandwidth of a single ethernet line, to computers on your wired network…they will also need LACP https://support.apple.com/kb/PH18560?locale=en_US ) connected to your physical desktop. You will need a gigabit managed-switch to do this which is not expensive.

        This will basically future proof your setup and give you a nice rack for use to expand, clean network (VLANs!) and you can access your stuff from anywhere, if you set it up.

      • March 4, 2016 at 8:21 am

        My best advice is: DITCH MAC! Its a piece of SHIT! Whoever told you to get a Mac knows no shit about computers. Windows has its flaws but still much better than Mac any day. Thats why its the most sold software ever. And as for the hard drive. Use a network hard drive. That works well for any OS and you can connect several of them to have linked all your stuff simultaneously.

      • Lukas Sprehn

        March 6, 2016 at 8:39 am

        I also say get rid of Mac an switch to PC. There is a reason it’s the best sold operating system, and that it is practically the only system used for gaming on computers. The fact is that Windows usually have more power for less money. The only good reason to get a Mac, as I see it, is the sleeker design in and outside, and the easy workflow. But beneath that facade is, well nothing, in comparison to a powerful Windows PC. Windows is stronger for video and image editing (yeah, they’re on top of the game in those areas again now), in some ways audio as well, CG modelling, animation and handling, it has more options, features, and ways to customize and/or change one’s system. You could also do that on a Mac, but it would be more difficult, since most of Mac’s system is fixed the way it is, unless you know what you are doing. Then there’s also the issue of Apple’s cables, like chargers. They are way too easy to wear out and break. I say get a PC and never look back, unless you are going to play around with audio and music.

      • Lukas Sprehn

        March 6, 2016 at 8:45 am

        Also, I’ve never heard of that particular thing happening with Premiere Pro. It is definitely supposed to be better than Final Cut these days, at least in terms of what it can do. Maybe some little snippet of video were corrupted in such a way that it made it do that…?

        Also, what kind of project is this different thing? What genre? Length? Type? A new series, a pilot of some sort, drama, supernatural and story-driven…?

      • Lukas Sprehn

        March 6, 2016 at 8:52 am

        Sorry for writing so many times, but like people are saying, what you are doing is dangerous to a degree, and you should consider that NAS thing, as well as cloud storage. Now, I still recommend switching to Windows, and if you decide to actually do it, here is a good setup I thought up for you: 16-36 GB RAM, some SSD (Solid State Drive) as the main drive, it makes everything much faster, also while booting and turning the PC off, then some extra internal HDDs at several TB, maybe an Intel i7 processor, an NVIDIA graphic card (GTX 960 or 970 would be a good choice for what you are trying to do, which would be editing videos.) Some good coolers, maybe plenty of them. This is basically the same setup I’ve got at the moment (I don’t have 36 GB of RAM though, only 16, which is still kind of high for normal consumers.)

      • Lanlost

        March 17, 2016 at 12:40 am

        Lukas Sprehn, respectfully, you don’t know what you’re talking about as far as OSes go. I’m a Linux (at work) and Windows (at home) user but also a systems programmer. When you talk about the Mac OS you seem to be talking about the user interface and not the kernel. These are two different things. I have problems with the Mac user interface, as well as the customer base which are largely suckered into thinking it’s better for art. You aluded to this. This is debatable, but in my opinion, largely true. The KERNEL (as in, the backend) of Mac OS is much, much, better than the backend of Windows, although not nearly as much as it was in the 2000s and before then. It unix-esque and thus makes much smarter decisions about managing hardware, memory, file systems, etc.

        I mean, to the general consumer, these things don’t really matter and as far as the debates people are normally having over Windows vs. Mac are based on the user interface/experience of using it side of things. In that case I’d take Windows any day. Hell, I’d take Windows over Linux any day in a lot of ways, and I know how to use Linux. Let’s not get this confused though and act like the kernel of them are equal. Linux > Mac > Windows on the back end. Windows > Mac > Linux on the front end.

        More than anything though I think it’s funny that you said:
        “I say get a PC and never look back, unless you are going to play around with audio and music.”

        Really? Do you know who you’re talking about here? This is what James DOES. Even if you didn’t know about him though he still specifically made that clear.

        Finally, there is no need for him to “get a PC” when he has an x86 based Mac. Just run Windows on it, or stay with MacOS since it does RAID perfectly fine. The OSes here should honestly not have any significant effect here as RAID support has been around for a long ass time and you risk turning this into a Windows vs. Mac debate which it shouldn’t be.

    • February 27, 2016 at 3:55 am

      But it’s important to get hard drives that aren’t from the same production batch. I had this once and they all crapped out 1 week after another (luckily with enough time in-between to restore the data, but annoying nevertheless). But yeah, a NAS with RAID-5 of a bunch of 30 TB hard drives might be the best solution. Just remember that an odd number of drives is best performance-wise and that you will loose the size of one full drive (this is were the “backup” is stored). When you get it with 1 Gbit/s or 10 Gbit/s network connection the connection speed is comparable to USB 3.0, but not as fast as Thunderbolt 2.

    • Persian_Assassin

      February 29, 2016 at 2:40 pm

      Did you even read the article? James said he backs up his data a lot. RAID 0 is not a bad idea at all, you’re completely neglecting the aspect of speed. It’s much more efficient to have a faster drive, considering data loss is not an issue here.

    • Don Skiver

      March 1, 2016 at 6:05 am

      You need a NAS. You’ve already got network infrastructure in place because of the website, so implementing a large rack mounted NAS that uses a large spanning raid 5 or 6 wouldn’t be too difficult (yeah, I know you’re not the most literate guy, but get Ryan to help you). You’re seriously on a ticking time bomb with all your stuff in raid 0. Whether you want to go enterprise level or not with SAS vs SATA or whatever…that’s a totally different topic. A quick search on newegg for 10-24 bay SATA and/or SAS network enclosures that uses USB 3.0 and uses RAID 5 and/or 6 shows up over 100 results. Most of this shit is for small businesses but you’re running a small business so I don’t see why you don’t go that route. Getting a consultant might not be a bad idea in this case because you’re really on borrowed time here. You WILL eventually have to rebuild that backup from the many many scattered backups out there, and that’s a huge pain in the ass.

      Only caveat is that this isn’t going to be cheap. If you want to do this the RIGHT WAY and not cut corners and shit, it’s not going to be cheap. Talking an easy grand or more.

  • February 25, 2016 at 4:29 pm

    I previously had multiple problems with LaCie external drives plugged on Macs, wo I wouldn’t recommend you to continue there, unless you feel really safe with those. I went to buy a new one from Seagate, 3TB, although the Best Buy guy told me “No one returned one… yet”, with a face that said “You’re gonna wreck it, and feel sorry for it.” But it’s been 5 years, and I never looked back: no crashing, no loss of data, easy to use for montage (I use an old version of Adobe Premiere that never failed me either). Other than that, I used a PassPort device for travels. It’s about 1TB, very light.

    I know it’s not the same size, but maybe there’s a new window here for you in those companies. Maybe we’re getting closer to that darn PetaByte (yeh, sure…). Unfortunately, the bigger I saw on Amazon or Best Buy was a drive the size of an amplifier and that juiced up to 20TB, on its own. So… either you get five of these bad boys (1,000$ to 1,500$!), or you continue with the chained LaCies.

    Hope to see your new projects soon!

  • Pentium100

    February 25, 2016 at 5:21 pm

    I do not know if my previous comment got lost, if this is double post please delete this comment.

    I would get/build myself a storage server with the required storage (100TB or whatever) using Linux or FreeBSD and zfs. No proprietary software needed and zfs also does checksums, so that the data is not silently corrupted.

    For such a big array, I would not use RAID0, but RAID50 or 60 (RAID0 array of RAID5 or RAID6 arrays), with 20 6TB drives, you can get 96TB (2 arrays of 10 drives each with RAIDZ2/RAID6) which would survive two to four drive failures.

    I myself have a smaller 24TB server (2 arrays x 6 drives x 3TB, RAIDZ2), it works really well.

  • blipsqueak

    February 25, 2016 at 5:30 pm

    I’d get a NAS (network attached storage) like a Netgear ReadyNAS or a Drobo. Get one that can take a crapload of drives. Get what you can afford and you can add more drives and expand the storage as your needs change.

  • February 25, 2016 at 5:39 pm

    I guess we can all agree a NAS is in order. I have good experiences working with Lenovo EMC and Ctera products. Don’t even think about getting a Terastion.

  • February 25, 2016 at 5:45 pm

    James – I do tons of video editing and store large amounts of raw video and progress files for myself and clients. I used to work the way you do with external drives but I found a solution that works much better for me.

    I do my local editing on a computer with a 3TB Western Digital Black internal drive. When the project is done, I store it on my Synology NAS. I can always bring that stuff back very quickly. With GB Ethernet, I can pull up even the largest project in less than 20 minutes and then edit it locally. (Editing over the network is possible, but it’s worth waiting for stuff to move local so everything is nice and fast.)

    I use the Synology DS line. I would recommend the DS1815+ and 8 6TB Western Digital Red drives in RAID5. That will net you 42TB of space. When you need more, you can add up to two DX513 expansion modules which hold 5 drives each. You can expand the original volume to use the new expansions so it appears as “one big drive” instead of three.

    • darmon Mackey

      February 26, 2016 at 8:46 am

      I strongly recommend the Synology NAS. I do 3d renders of high resolution laboratory microscope data and I used to use external hard drives like you. I set up a Synology NAS and it changed my entire workflow for the better. Synology is better than it’s competitors because the UI and software support is much better. You can keep adding drives to it and can even daisy chain multiple NAS’s together that will appear as 1 drive. If you want to see what it’s like I can setup a guest account to mine for you to poke around in.

  • Josh O

    February 25, 2016 at 5:49 pm

    I work for Apple and recommend contacting your local Apple Store and asking to get in touch with the business team. They could easily figure out a solution for your hard drive fiasco and get you running efficiently. You don’t have to buy anything either you can just ask for advice and have them make some reccomendations. I could also get them to reach out to you if you’d like.

  • February 25, 2016 at 6:03 pm

    James,

    My recommendation if you want a lot of storage that’s easy to work with is to consider these guys: http://www.drobo.com/

    They use their own version of RAID that is fully redundant, so if a disk ever dies, all you have to do is pop it out and pop a new one in. It can also grow with you dynamically, automatically, if you put a new drive into the enclosure. You can use any drives you want. The 5D, for example, has USB 3 and Thunderbolt 2 connections and supports up to 5 drives of any size.

    Also worth mentioning: get yourself a UPS (uninterruptable power supply – a backup battery) for your storage. That way, even if you lose power, you have a chance to safely power down your computer and your drives before the battery runs out.

  • February 25, 2016 at 6:36 pm

    I also recommend the Drobo 5D, which has Thunderbolt and HFS+ capability. Load it up with 8TB drives, plug, and play!

  • Trepidity

    February 25, 2016 at 6:41 pm

    Store unused or archived data on the cloud. Google Drive 30 to for 300 a month which since this is your business it can be written off to offset some of the cost. There are other services so you can shop around but you would essentially use the local hard drives as working drives. While 3600 a year is costly the data is safer spec across multiple servers. The peace of mind is worth a lot

  • laser

    February 25, 2016 at 7:27 pm

    I’ve never used them, but at a quick glace NAS looks like a good local storage solution. These units (up to 24TB each) from Western Digital look promising:

    http://www.wdc.com/en/products/nas/expert/

  • February 25, 2016 at 9:43 pm

    You are beyond “hard drive” land when you are coming up to the 20TB ‘ceiling’.

    I’d heavily recommend a FreeNAS, or similar solution if you need that great a capacity locally.

    Getting a case that can host 16 6TB drives could net you a very nicely performant 48TB (in RAID 10 for that sweet speed and resilience you’re missing on your primary storage).

    A good start/jump off would be:

    https://toys.lerdorf.com/archives/58-Building-a-NAS.html

  • Kryssen Robinson

    February 26, 2016 at 12:30 am

    Ah, one of the only times James Rolfe will ever read the comments.
    Oh and check out the priority setting on Premiere before you try to export, it helps if it’s set to the lowest priority.

  • ToadRoad

    February 26, 2016 at 1:14 am

    Not to hijack, but maybe it will be useful to the nerd as well.

    So if I have 8 externals ranging from 1 TB to 5TB each, there’s a way to simplify other than using the 7 Port USB Hub I have? I tried looking up NAS + RAID, but I’m confused what it actually does with your drives.

  • February 26, 2016 at 1:27 am

    What Mac OS are you using? Mavericks, Yosemite, El Capitan? Might be possible that you might need to update your OS or maybe the recent OS isn’t yet working properly with Raid Manager or Premiere. Also speaking of Premiere Pro, I’ve been using Premiere for 4 years on mac and just started last month on editing with PC. I’m assuming you’ve been having premiere glitches? Since you’re still using Final Cut I’m assuming those ProRes 422 files are a big reason why you need so much storage since those files are huge.

    • February 26, 2016 at 1:29 am

      This external HD may be useful for you? http://www.pcmag.com/article2/0,2817,2473012,00.asp It’s a G-Technology G-Speed Studio (24TB)

    • February 26, 2016 at 12:27 pm

      Hi, yes it’s the latest El Captian. For me, the two advantages in Premiere are 1) less rendering 2) less storage space, since it doesn’t need ProRes. But still, I have almost 30TB of stuff accumulated over the yeas, and I want to keep instant access to everything.

    • February 26, 2016 at 3:08 am

      think also of getting yourself an APC (backup battery pack so you can have time to save and close things if power goes out) – if you are having outage problems

      • February 26, 2016 at 12:25 pm

        Hi, yes I have a battery backup. I just got another one lately, after my old one failed.

  • Nighthawk

    February 26, 2016 at 6:42 am

    What you have is completely untenable, and the fact that you rely on it for earning income is dangerous. You’re literally one head crash away from disaster! This is exacerbated by the fact that you’re running RAID 0, so you’re six times more likely to encounter a failure!

    You should rebuild your NAS with seven 8TB drives, and run them as RAID 6. This will give you the ability to take two drive failures, as well as having 40 TB of space.

    You should then install whatever cloud backup software that you find appropriate, that lets you take versioned file snapshots to off-site storage. Ideally it will work without your intervention. CrashPlan or Dropbox would work here, with one of their business accounts.

    It’s good that you’ve identified this as a problem, but PLEASE make sure you address it sooner than later. You’re one failed drive away from killing the nerd!

    • February 26, 2016 at 12:31 pm

      Yes it’s a problem. that’s why I made the post. Not sure if you read correctly though. I backup the 30TB onto many smaller drives. If the 30TB (Raid 0) fails, I have my backups. But still, it would be a major inconvenience if it fails. I’ve never heard of NAS til now, but thanks to you and everyone for bringing it up, so I can research it. Would Dropbox or some kind of online storage back sense? 30TB is a lot! And my internet upload speed is not fast at all.

      • February 26, 2016 at 1:04 pm

        there is no way to upgrade your upload speed with your ISP ? – cause if it would be a decent possibility – just for the backups – not for actual work. (dropboxes, google, apple cloud whatever else…)

      • darmon Mackey

        February 26, 2016 at 2:26 pm

        If you use a Synology NAS you can schedule automatic backups to off-site storage solutions like Amazon cloud storage. You can set it to run only in the middle of the night or other low use times. The Synology UI is super intuitive as opposed to any kind of home brew solutions like free bad.

      • darmon Mackey

        February 26, 2016 at 2:29 pm

        *free bsd

      • March 3, 2016 at 1:14 pm

        http://365datacenters.com/ 365 Data Centers 3701 Market St, Philadelphia, PA 19104
        Phone:(215) 764-6479

  • February 26, 2016 at 8:44 am

    If you really want to preserve media, HDDs will always fail over time. Physical discs dvds bds etc. are better for long time preserving ’cause as long as you don’t scratch the discs and have hardware to read them the data on them will be ok.

    • February 26, 2016 at 12:32 pm

      Not sure. Imagine how many DVD’s it would take to save 30TB of data, and how long it would take to burn them all. and I wouldn’t be able to work off them. Those would be backup only.

      • ToadRoad

        February 26, 2016 at 10:29 pm

        As someone who used to backup A LOT on DVDs, it’s not worth the man hours when you’re dealing with data in that quantity. Of course I was using regular old 4.37 GB DVDs. Although even if you can burn triple layer blank Blu-Ray disks (100 GB eac), You’re still looking at 300 discs to cover 30 TB, and that would be the furthest thing from instant access.

  • maximuscesar

    February 26, 2016 at 9:43 am

    I’d recommend using Cloud Storage. It being a remote cloud server, or if your internet bandwidth isn’t good enough, a local cloud server. With a local cloud server you can put as much as internal drives as you need (they are way way way more reliable than external drives) and internal storage is even cheaper. If your MAC can have it network card changed for a 10G network card you will even have better transfer speeds than any kind of USB 3 or Thunderbolt. Oh, and you will have access to all the data at once without the need of plugging/unplugging stuff.

    • ToadRoad

      February 26, 2016 at 10:32 pm

      I don’t see bandwidth being an issue, as uploading is a fairly passive activity. It would take a while to get it all up initially, but a few GBs each day is manageable.

      Depends on his ISP and whether he has a business account, which I’d imagine he does. I’ve had ISPs with bandwidth caps of a paltry 250 GB a month.

  • February 26, 2016 at 9:49 am

    Hello James, I have never used a Mac but I moved to Linux almost 10 years ago, so not everyone of your followers is using Windows. The easier (less technical), but more expensive solution would be to get an 8 bay NAS with Thunderbolt 2 support and probably an extension enclosure to hook up even more drives when necessary. If you’re interested in a more technical solution, then you should look into ZFS (a filesystem with RAID capabilities and much more https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs/). From what I heard OpenZFS is available and also quite popular on Macs, but may be you don’t even need that on your Mac and set up a dedicated BSD or Linux server (FreeNAS or Open Media Vault) to handle all the drives and hardware and only connect to your Mac through Thunderbolt networking (I read that TB offers networking, but I have no experience with that) so that it just accesses it as a network share. A safer option than running RAID0 would be to configure the drives as RAID6/RAIDZ2 and ideally you want to have another device with the same amount of storage space available (following the 3 copies backup rule). You probably heard of the backblaze storage pod? Something like that, but not that big. 10 5-8TB drives and a storage controller hooked up to decent high performance computer (leaning towards server hardware) that can hold 32GB of RAM or more should give you a good compromise of storage space, performance and safety (2 drives hold the parity data, the remaining 8 are near optimal performance, 4 would still be okay in RAIDZ2 mode, beyond 8 come 12 or 16 drives, but that’s too much risk of drive failures). (All of that times two for the backup device.) If you have doubts to find someone reliable who can help you out with this, you probably better go for an off the shelf solution. A nice place for technical Q&A by the way is StackExchange, they even have a subsite for hardware recommendations now.

  • February 26, 2016 at 10:42 am

    As others have said, You probably want a NAS. There might be some companies in your area that will build and configure it for you if you don’t want to take the time. Also as everyone has said you need a different version of raid. I know a lot of people use western digital reds in their NAS but i think 4tb is the biggest they go. HGST seems to be what people are using for 8TB drives. Also I would recommend a cloud solution for your backups. Crashplan is pretty cheap for unlimited storage (i think its unlimited) and ive heard a lot of good things about it.

  • February 26, 2016 at 11:32 am

    Hey James, been a fan of yours for a long time.

    If you don’t have a reliable/tested offsite backup, one fire could wipe out all your local backups. I know you mentioned you copy data to other drives and send it offsite, which is awesome. But that could be a very time-consuming and tedious process to try to get everything back, and you still could end up losing data. I work for an offsite backup company and see this stuff happen to people/businesses every day.

    In addition to your HD purchasing decisions, I’d recommend making sure you have reliable offsite backups, and ideally set up a schedule to test restoring capabilities in the event something were to go wrong. You don’t wanna know how many times someone tells me “oh yeah, we have all that stuff backed up”, and the data is corrupted, incorrect, or otherwise inaccessible. Just something to keep in mind.

    Best of luck going forward & keep up the good work!

  • February 26, 2016 at 12:46 pm

    i’m glad you’re reaching out for help and have identified the problems with your setup. you really need a proper enterprise level solution, forget about consumer level hardware. imo, you should contact a real professional who is able to set this up for you and *train* you in administration of the setup and how improve your workflow (backups etc). my impression is you waste a ton of time not knowing how to go about these things tech-wise and create time wasting and horrible workarounds (horrible like raid0 simply because you needed the space).

    if you get advice here, you will end up with more workarounds because you are not personally an IT professional who can set this up in a robust and future proof way. it will add some significant cost, yes, but you will make this up in time gained and additional reliability. please, don’t be cheap on your IT. let someone do it who knows his stuff.

  • Pablo1989

    February 26, 2016 at 2:33 pm

    Yo!

    Im using WD Elements portable HD its fine for me
    but he uses USB2 and had 1TB, i am not sure did WD makes stuff like you write up here.

  • iSkelly87

    February 26, 2016 at 3:41 pm

    I have read some of the comments on here and a lot of good suggestions when it comes to data, but I feel that you are missing true redundancy. I have a few suggestions, but really hard to talk about something has serious as your livelihood on comments. With that amount of data I would think about getting into companies that specialize in data like Nimble Storage would be worth looking into and even talking to. We are currently using their servers at work with other cheaper alternatives as well. That may not be in the budget right now, so there are alternative ways to do it. The best NAS type drives you are wanting to use would be WD REDs or RED Pros and then look into a rack station server. Please let me know if you have any questions I am willing to help all I can.

    • iSkelly87

      February 26, 2016 at 3:43 pm

      *Synology Rack station

  • Brandon Halsne

    February 26, 2016 at 4:05 pm

    I would like to recommenced, if someone hasn’t, FreeNAS. This is a NAS solution that allows you to dynamically expand space, is somewhat affordable, and works with all modern file systems whether using NFS, SMB, iScsi, etc.

    http://www.freenas.org/about/features.html

    I use this for my personal file storage system, as well as at the Municipal ISP I am the admin for our backups. It is very robust and very agile.

  • February 26, 2016 at 5:37 pm

    James I do a ton of IT work. If you want to reach out to me I can give you some contact info and work with you on finding the right product for you.

  • stewes

    February 26, 2016 at 6:04 pm

    James,

    I work at a small local television news station where we deal with large volumes of data as well. I ran into a similar issue on Premiere Pro CC 2014 on Mac OSX 10.8 running on an iMac with an Nvidia GTX680mx video card with Cuda installed. A couple times when working on a project, scrubbing through the timeline (not even rendering out), the computer would suddenly shut down as if it there was a power outage. It powered down without going through the proper shutdown routine. When attempting to power back on it would not, until I unplugged and replugged the mains. It happened roughly around the same point in the timeline when playing back the same clip. It turned out that the hard drive was beginning to fail (it was internal), and it appeared that OSX shut down as a precautionary measure. Either that or it got confused and acted accordingly. I was told that it could possibly have been caused by the logic board (which controls power management), so as a workaround, I would unplug the mains after a proper shutdown, and then plug back in when I wanted to boot up again. This has worked out for months. Eventually I had to change the hard drive. The immediate shutdown hasn’t happened since. I also updated the Cuda driver, and it seemed more stable. I doubt that Premiere itself is the culprit, but I would lean more towards a hardware or video driver related problem.

    I know exactly how you feel about change. I’ve been using Premiere for 15 years now, on PC and Mac, and while I dabbled in the old but more familiar FCP7, the news FCPX hasn’t exactly worked out for me in terms of reliability and speed. To each their own, I suppose. Premiere does have it’s problems, but for me it works great. I guess in the end, it’s best to stick to what you feel most comfortable with.

    As for drives, I would go with everyone else on the idea of NAS storage. Presently, we use G-Technology Raid drives with Thunderbolt for external backups. In your case, I think it would be wiser to use your 30TB as a backup drive that will be filled up as you complete a project then stored on a shelf, and use a smaller main “work” drive for caches, renders, etc. You don’t want to burn out your “everything” drive by having it in constant use, but rather switch it on when you need to grab file footage, copy it to your work drive, and turn it off again. This is where the NAS would be useful, as you can have it on 24/7, but the drives themselves would only spin up when you need to access them, thus saving on wear and tear. Hope that helps!

  • February 26, 2016 at 7:15 pm

    I´ve recently had the same exact issues with Premiere as well, it never happened to me before and i´m using it for over 4 years now. What i found out was that the crashes happened because of too much heat generating in the CPU, and the fan on the cooler didn´t increase rotation after the temperatures went up, so the computer shut down as a safety measure when it got past the safe range. I´m on PC though, so i don´t know how things go on MACs, but i have an 8 core processor and those suckers heat up pretty quickly when utilised fully, especially with video rendering. Try to render that project again and open a benchmark program ,or an app that monitors temperatures in the computer, and see if this is causing the problem. My problem is that the fan absolutely refuses to go over half the possible speed, so i guess it´s time to replace that with some water cooling. Maybe your fan started to die on you as well, especially if it´s a stock cooler like mine. I hope this helps about those crashes.

    • ToadRoad

      February 26, 2016 at 10:34 pm

      Adobe Premiere, the reason I stopped video editing. I feel like everyone who uses it has a story like this.

      • Lukas Sprehn

        March 6, 2016 at 9:03 am

        Not me actually.

  • DiscountPizza

    February 26, 2016 at 9:44 pm

    You might want to look into cloud storage solutions. Amazon Glacier is apparently a low-cost archive system (infrequent access). Going by their website, your 30 terabytes would cost $21/month. (Transferring data out does cost money, but it’s still pretty cheap.)

    The problem is you’ll still want local backpack just in case.

    • DiscountPizza

      February 27, 2016 at 11:39 pm

      Whoops, my mistake. I was off by an order of magnitude. It would be $210/month.

      …Okay, building a solution might be the cheapest option.

      Going by PC Part Picker, the Seagate ST8000AS0002 8 terabyte drive is $0.027 per gigabyte.

  • February 27, 2016 at 1:27 pm

    why are you not using cloud storage?? use multiple companies. i am talking about for backup, not referencing. google m$oft and others offer this service.

  • February 27, 2016 at 11:59 pm

    Uses a software that emulates the Windows operating system like parelels

  • Icespider

    February 28, 2016 at 2:27 pm

    Cloud storage would be your best bet. if you have good internet (which I assume you do) then you can’t be happier.

    -guaranteed to never lose files( they back up a ton as well)
    -Power outages mean nothing to you except u have no power at the time
    -capasity is a joke as long as you pay for it

    just shop a bit and your set OR get another NAS bay and load it up with a bunch of WDREDS. also your power outages sound like a surplus of power being over your current set-ups threshold. cheers

  • dmaraio

    February 29, 2016 at 10:04 pm

    You might want to consider something like a storage cluster. You can get some servers and add hard drives to them and have a redundant storage cluster that you can expand as needed with more drives/servers. The CEPH project is an open source storage cluster that is very reliable. Unfortunately it doesn’t work like a normal hard drive. You would need to use client software to upload/download things from the cluster.

    However, this may be great for all your archive material. You can pull down things you need on your local scratch disk and then upload them to the storage cluster for safe keeping.

    You can even have the storage cluster hosted for you. I work for a company based in New Jersey that can provide managed storage clusters. Let me know if you are interested.

  • Stefan Grassberger

    March 1, 2016 at 7:44 am

    I would highly recommend getting a Synology NAS. They have their own RAID solution which is very flexible: If one drive dies, you can simply replace it with a new one without losing data. You can expand the storage, the interface is very simple, it shows up as one drive and integrates well with a Mac. You could even get two and mirror the data from one to the other stored off site. Check out their website, there are lots of options and they’re all explained very well. Good luck!

  • March 1, 2016 at 1:07 pm

    You gotta use the Commodore 64 with the Superman floppy disk, You pop that fucker in, Turn the son-of-a-bitch on, play some Superman and type the code: “LOAD “*”,8,1 and then Return. There you go. Welcome back.

  • March 1, 2016 at 1:36 pm

    As far as hard drives are concerned, I am not very knowledgeable. In my community of friends, the words, “Solid State” have been used frequently but I am unaware of the current sizes and how affordable they are, nor am I sure if that is even an option for you. As far as power outages are concerned, the kind where the power flickers throughout the building and/or house, battery backups might also be a reasonable addition. I doubt a generator is an option, so we’ll say this is, after all at least you would have some time to power down and save. As for the computer wanting to…we will call it, “Feint” it might be a power supply issue that has begun acting up do to all the aforementioned referencing you need to do, putting more stress on the computer. Add in the fact that you are using Premier, etc., you get the point. Hope it might help somehow if at all.

  • greedo

    March 1, 2016 at 1:57 pm

    James, I wish I could help in this regard, but sadly, I can’t. However, I hope you do have a system backup thing, like Time Machine, or a similar third-party solution, in addition to your hard drive.

  • hell giver

    March 1, 2016 at 3:18 pm

    James, get yourself a NAS. I suggest Synology or building your own. Drobo is also a good brand but maxes at 12 hard drives with no expansion options (Drobo b1200i). I would specifically recommend the Synology DS3615xs. It has 12 drive bays and can be fitted with 2 expansion modules with 12 more drives each. This gives a maximum RAID 6 size of 278 TB.

    I suggest using RAID 6 with off site backups. RAID 6 allows for two drives to fail with no data loss. These Synology units allow hot-swappable drives so you just replace the dead drive with no interruptions.

    For off site backups I would highly suggest Crashplan. You can run it on your NAS for just $60 a year and unlimited cloud storage. You can even then have your home PCs backup to the NAS and it backs up to the cloud. Crashplan does not officially support running it from NAS devices but if you Google “Synology Crashplan” there are tons of people who do it anyway.

    I know you have said your internet upload is not a lot (my initial Crahsplan upload took about 2 months) but it will be worth it in the end because you no longer have to deal with managing a ton of small external drives.

    Also, when purchasing the hard drives check out Backblaze’s hard drive failure rates. They publish the failure rates of all the drives they use and it can really help when deciding which hard drives to purchase.

  • warwagon

    March 1, 2016 at 8:25 pm

    As I was reading your story I got to the part were you said all 5 drives are in “Raid 0” and I gasped and uttered the word “OH GOD!”. The funny saying about Raid Zero is “They call it raid 0 because of 1 drive dies, all you’ll be left with is 0.

    As others have mentioned i would strongly suggested a NAS in a Raid configuration could handle 2 drives failing. This way with 2 drives failing at the same time your data would still be intact and the raid rebuilt when the new drives is inserted.

  • March 2, 2016 at 11:29 am

    I think i screwed up and linked the wrong product this was the product I was talking about. http://www.bhphotovideo.com/bnh/controller/home?O=&sku=1184475&gclid=Cj0KEQiAu9q2BRDq3MDbvOL1yaYBEiQAD6qoBuvLSodpc661TAYcyoCtnJTcrSCNofcRrjQGkNMopbAaAorO8P8HAQ&is=REG&ap=y&m=Y&Q=&A=details

    8 bay NAS with a good enough processor and ram to be able to edit directly from the NAS. Also I did say 6tb hard drives before but you can put in 8TB hard drives into it making it capable of holding 64tb of data in Raid 0, honestly I would use raid 5 then if a drive goes bad you can hot swap it with a new drive. Feel free to email me if you have questions or would like me to help you build a custom NAS

  • March 2, 2016 at 6:38 pm

    I deal with BigData a log at work so 30TB is not that big a deal when using applications that are use to dealing with data that large.

    I feel you are trying to solve the problem backwards. Your use case is kind of in-between personal and business, which makes the requirement of 30TB sound absurd to many. I think you should think about what you are willing to manage and be responsible for and what are you willing to take on yourself, if things were to go wrong.

    The options I see for you are

    – Cloud storage (Google and AWS both have cloud storage options but there things to consider)

    Pro
    – Accessible everywhere
    – Can build a system that can mount the file system to any machine, i.e. NFS…performance will not be greate
    – Potential cost savings
    – Built in redudancy and high availability

    Cost
    – Can also be expensive
    — cloud solutions allow you to leverage Amazon or Google services, but may cost more money. Building your own system could save you money…could…
    – Performance. Transfer speeds could be a pain in the ass.
    – Transfer rate will be slower than local.

    Craigslist for a rack server or managed
    You can look on Craigslist for an older Dell rack server¥, like an R700, with 16 hdd bays and look in to ways to leverage a combination of RAID and software, RAID0 or HDFS…

    Advice, leverage networking with whichever solution you pick, i.e LACP/bonding, not thunderbolt 2 or USB 3.0. This will potentially make it easier to access your data from anywhere and it will be fast.

    I keep this short and not too detailed as I can go on forever on this, but I would not look at a consumer solution to solve this problem.

  • March 2, 2016 at 9:49 pm

    Hi James. I would honestly at this point consider offsite, cloud based data storage. The perfect scenario would be to back up to a RAID 5 or 6, using just 5 TB of storage on 4 or 5 HDD’s in a RAID 0+1 or 1+0 config. That will give you the speed (striping RAID 0) coupled with the redundancy backing up (RAID 1-mirroring). From there, have the cloud back up pull data in an “incremental” backup mode from that 1+0 configuration. That way, your data is always safe in the cloud, and the latest data is always safe in the cloud, plus on the RAID 1+0 config.

  • March 3, 2016 at 4:09 am

    Store the data on 11 trillion cassette tapes.

  • March 3, 2016 at 1:07 pm

    Lease space in a Data Center. It is what the pro’s do

  • redbalmung

    March 3, 2016 at 8:05 pm

    Bruh, just git gud.

  • March 4, 2016 at 2:42 pm

    “Everybody else uses a PC’s” Wow, I can say that it’s the complete opposite here! Pc is gone, just macs. I still have a beige pc tower haha

    • Lukas Sprehn

      March 6, 2016 at 9:01 am

      Huh? Where do you live? The most sold system is still Windows to this day, and more and more people are in fact going from Mac and back to it these days…

  • timbo92203

    March 6, 2016 at 1:42 am

    I’m a designer and will have to deal with this issue eventually as well. Cloud storage is hundreds a month for the size of data you have, and it would take days to upload that much(possibly weeks?). Download times to retrieve the data you need will take quite awhile too. Aren’t there companies that can set up this kind of data management for you? With whatever system they recommend? I’m not a fan of these technical issues and would rather pay someone to deal with the hassle of setting up something like this and have the peace of mind that experts handled it. I am interested in what you figure out, please post.

  • March 7, 2016 at 11:31 am

    Get a PC, then we’ll talk

  • March 9, 2016 at 1:09 am

    As a former film student at around the same time you were I was also told to get a Mac and use FCP. That is how I learned and I’ve found no reason to switch. I switched to Premiere after many people claimed it was better and I’ve only ever had issues with it. In this thread there are many people telling you to get rid of your Mac and switch to PC but none of them explain how this would help you in any way, especially since none of your problems seem to be OS related.

  • Lorin Heath

    March 9, 2016 at 11:46 am

    MAC does what PC Dont!

  • March 10, 2016 at 5:01 am

    Im just going to echo what has been said in a few comments. A NAS (network attached storage) would be your best bet. Either raid 5, or something similar (I think Raid Z1 or Z2 is similar. Dont have much experience with those) would be perfect for what you are trying to do. There are a lot of ways to do this, but for now, I would say your cheapest option is a used unit on ebay. Now the drives will be used as well. However, if they were to fail while utilizing raid 5, you can shut the device down, and replace the hard drive. It will rebuild itself and your data is not lost.

    There are obviously way better ways to do this. However, with your primary concern being just a storage haven and not running a server or anything off of it, IMO this is your best bet without breaking the bank. With you wanting at least 30TB, I would look at an 8 bay NAS.

    Even better, here is a 12 bay NAS server (diskless so you need to supply your own hard drives, which will cost more money.) but runs FreeNAS, which is windows and OSX compatable. This also should run Plex, which allows you to stream to smart tvs, apple tvs, rokus, so on and so forth.

    http://www.ebay.com/itm/Intel-Atom-Mini-FREENAS-Storage-Server-12-BAY-8-HOT-SWAP-NAS-Server-/141907560454?hash=item210a598806:g:RNAAAOSwHnFVmqPN

    If not this NAS, any will do. Since it will be accessed VIA the network, USB or Thunderbolt wont matter.

  • StarReel

    March 15, 2016 at 1:32 am

    This is the hard drive you are looking for James. It’s the one i use. Very safe, VERY easy to use and you will NEVER loose any data, it’s also has USM that lets you have a backup within the backup, removable.
    http://www.span.com/product/Seagate-NAS-4-Bay-STCU200-empty~45966/spanstor=51060

    You REALLY should consider dumping MAC. Only clever marketing made it the “standard” for video editing in the early 2000s. In actuality it’s always been worse than Windows at, well everything.

  • March 18, 2016 at 6:14 pm

    The big question for help and advice for a solution is what is your budget? You can always buy >30TB of space, but then you need a way to back it up if you really do care about it. So you typically need more space as media doesn’t really compress/dedupe well.

    Then as other people have mentioned the RAID 0 is a bad idea. But then again if you buy a gaggle of 8TB disks you don’t want to RAID 5 them. You at least want to do RAID 6 or even better RAID 10. This is because if you lost a disk in RAID 5 with those densities you would take forever in rebuilding the data. RAID 6 and RAID 10 will guard better and give you quicker rebuild times as you replace disks.

    When you start getting big like that you can look at solutions from ZFS, storage spaces, or many different flavors of erasure coding and methods to give you redundancy, large amounts of space and quick rebuild times as disks die.

    Lots can be done, but it all takes budget and how much time you are willing to put in yourself to learn the technology. Paid drop in approaches can be worth it if you want to be fairly hands off. But they can cost you.

  • Gamer_2k4

    March 23, 2016 at 5:53 pm

    I have a file server with Windows Server 2008 on it and several RAID arrays holding all the things I want to store. While I’ve only ever accessed it from Windows computers, I have to believe the shared files would show up just the same on a Mac. It’s easy to maintain, and I can upgrade it whenever I need to (though the 20 or so TB of space should hold me for a while).

  • April 1, 2016 at 1:30 am

    sounds like it might be time to upgrade to a san as nas can only take you so far.

  • April 1, 2016 at 1:33 am

    without knowing your budget or ultimate size needs its hard to say but if you need really large storage this san would do you

    http://www.newegg.com/Product/Product.aspx?Item=9SIA6ZP3S84368&cm_re=san-_-22-107-272-_-Product

  • April 3, 2016 at 2:04 pm

    James I’m not trying to exagerate, but a raid-0 drive will kill AVGN. Get yourself a 7-bay nas from qnap or synology and an expansion bay. Pop in 10 4TB drives and you’ll have your 30+ TB in RAID-6. Don’t go below Raid-5. Even Raid-10 is so much safer than a striped volume. Check the raid calculator to compare Raid 5/6/10 and use your common sense.

  • christburner

    April 6, 2016 at 3:15 am

    I dont have a ton of experience with servers or raid arrays but just the other day i saw a video on NAS setups and it was pretty interesting. i see some other people have mentioned them too. that could be something to look into. sorry i couldnt be of more help, but that NAS rig is pretty neat so i figured id vote up on that suggestion.

  • April 7, 2016 at 7:02 pm

    It would be recommended to get an SSD if you haven’t already because it’s very scary to have a mechanical hard drive for a long time and suddenly, the dreaded clicking begins to start.

  • April 10, 2016 at 12:05 pm

    Hello James.

    I deal with large RAID environments.

    When it comes to RAID spans that are 10+ terabytes I recommend using RAID level 6. These are complicated reasons of how RAID 5 may not be enough if you need to rebuild. Its hard to give math lessons over a comment box however you should really be seeking expert advice IRL.

    SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. Which means that once every 100,000,000,000,000 bits, the disk will very politely tell you that, so sorry, but I really, truly can’t read that sector back to you. One hundred trillion bits is about 12 terabytes.

    This means when one of those drives fails in your 30TB raid 5 it will need to rebuild itself when you replace the faulty drive. It will read all 30 tb of data and there is a high probability that somewhere in all that data some bits will not be read properly and the rebuild will fail causing data loss.

    This is where RAID 6 comes into play. RAID 6 will give you fault tolerance for 2 hard drives. In the background the hard drives are constantly checking and calculating error parity and with 2 drive fault tolerance it becomes less likely. Basically its double checking itself to make sure that the data being read is accurate.

    You need some serious hardware there. I would not recommend a premade solution.
    If you cannot do this on your own,
    I would call around for a company to assemble for you a RAID 6 enclosure with USB3 + ESATA

    But right now with the way things are looking you are looking for trouble with just RAID 5

  • porizj

    April 20, 2016 at 1:04 pm

    James, if you’re still looking into this, consider CrashPlan as an offsite online backup service. They allow you to “seed” your backup; by paying a fee they’ll mail you a hard drive you can fill with files and send back to them so you don’t have to upload everything. Given how much storage you need you’ll likely need to pay for quite a few seedings, but after that you’re covered in case of emergency and can stop worrying about backing everything up to multiple other drives. As others mentioned, you should consider something like RAID 5 or 10; while you can recover your files in other ways, it’s a huge pain to rebuild a massive RAID 0 partition if one or two drives die.

  • April 25, 2016 at 3:54 am

    Duck Tales.
    Knock knock.
    Anybody “Home”?
    woop woop

  • April 25, 2016 at 3:54 am

    WHY Goemon goes to HELL? FUCK YOU

Leave a Reply

Around the Web