Tested’s Media Management Workflow!

In our latest behind the scenes video, Joey goes in-depth with his media management workflow for shooting and editing our Tested videos. Here’s how Joey handles the gigabytes of data from memory cards to DAS systems to long-term archiving on a Synology DiskStation server.

Comments (16)

16 thoughts on “Tested’s Media Management Workflow!

  1. So, you do a “push/pull” workflow, not editing on the NAS. At my current day job (working as a storage/Linux geek with CNN/Turner editors/producers), we use high performance storage to allow folks to edit on shared NAS. It lets us insure the snapshots and backups are done using the active projects and we’re able to do recoveries if/when something happens to the project by reverting to a previous version. It also makes handing projects off between different editors much easier (with ~20-30 editors, a few senior editors, graphics folks, etc. this is a big concern).

    Our footage ingest is done using workstations that have the storage mounted. It’s a manual process where the producer/editors drag and drop from the cards/direct attached disks into specific directories. They know the guidelines for the directories where they drop things and how they’re organized so nobody goes rogue because doing so makes everyone’s lives harder. It’s more labor intensive than the workflow you demonstrated, but it’s necessary due to the larger number of editors we have.

    We’ve gone from using old hard drives/tape to shoving that stuff up into S3. It does have per-month costs for maintaining storage (Glacier helps mitigate this a bit) and costs money to retrieve content back (Glacier exacerbates this a bit when you go over limits), but tends to be better than having hard drives/tapes in cabinets (which also means they’re vulnerable to physical damage/loss due to environmental factors).

    I’ll actually be moving out to the Bay Area in August for a new job. If you guys are up for media workflow discussions, I’d love to chat and find out some more of the history behind how you got where you are and see if I’ve got any good suggestions to help with expansion plans.

    And thanks for the look behind the curtain. I love these BTS segments!

  2. PERFECT time for me to see this feature. My world is more ProTools and Logic for audio productions but I am moving my way into animation projects. Thanks TESTED.

  3. My work flow for videos is still messy, but I’m getting better.

    Right now I just have a local copy on my secondary SSD, and then everything backed up on my OneDrive. I recently picked up a 1TB WD HDD to act as an additional local back up, but I haven’t gotten around to hooking it up yet. At the frequency I make videos that amount sound be plenty for quite a while. I think my next storage upgrade is going to be bigger SD cards. Right now I’m working with just a 16GB card in my video camera and either a 16 or 32GB in my DSLR.

    Giant Bomb’s new video gal Abby has some sort of video production/film degree, and she mentioned how it was drilled into her to give insanely detailed names for all of her video files. Up until now I’ve just kept the numbers my cameras give the files. But given how much dang B-roll I shoot, some of which looks exactly the same from the preview still, I think I should probably definitely start doing the same thing. (I don’t think I would do it retroactively however, as that would not only be a pain, but would also break my Premiere projects.)

    I also have a problem of not deleting files I don’t use. I mean, I’ll delete the really bad stuff. But, did I use a panning left shot? Well, I might as well keep the pan right one. What if I need/want them in the future?!

  4. Re the RAID setup, I would touch on a couple more things. When populating the enclosure, ensure the HDDs are not purchased from one source (get non-sequential serial numbers from separate sources, 50/50 is fine). Bad HDD batches do exist (manufacturing issues, shipping damage), leading to my second point. I would dedicate an entire drive as a hot spare, and also keep a dedicated spare drive on hand as a cold spare. Yes this means “losing” two drives worth of storage (1 to parity, 1 to hot spare), but this will safeguard against a cascade failure leading to complete data loss. Without a hot spare, the parity drive will allow recovery from a single drive failure, but what happens when a second drive fails before someone comes onsite to replace it, before parity is restored? *poof* All hard drives fail, the question is when and how close together. -Someone who runs various enterprise NAS, SAN and DAS systems.

  5. Joey,

    As someone who works with archived data, I enjoyed your talk on archiving video from Tested. Hey, we are all nerds here; I enjoyed any discussion on archives. Happy to see that you are doing this.

    As someone who works with archived data created by others, I hope that you consider those persons who will want to check out your work in 20-30 years – this is my life right now, check out data produced 20-30 years ago (and more). I suspect that the data producers had no ideas that I would want to check out their work.

    Will folks in the future have the ability to watch your videos given the technology 20-30 years from now? A climate scientist may want to check out your wonderful videos of Greenland. Does your archival system ensure that they can do this?

    Perhaps you might want to think of your work as a legacy describing the world now. No pressure 🙂

  6. Thanks Joey, always interesting to see how other people deal with their data bulk – I recognize a lot of similarities in the way I work with the bulk of media files. Did I get it right and do you actually delete all of the footage you didn’t use in a project (for clients)?

  7. Sorry to be the first one to say this, but this storage solution made me cringe. Hard drives should be kept online at all time, not stored in a container. Having them online will keep them active and S.M.A.R.T checks will keep track on them in case of failure.

    So add the disks in a large server and let freenas and ZFS do the rest. And all the video will be online at all time. And get 10GB network cards and a few SSD and you are good to go.

    http://www.45drives.com/

    Sorry Tested, but this is really amateur.

  8. I couldn’t agree more! I manage the special effects archives for Steve Johnson. We back up everything to a Synology RAID. We even went so far as to have two hot spares in the RAID as well as several cold spares. Very early on, we had three drives fail at once (same source, sequential serials). Months of work scanning and digitizing old VHS tapes would have been lost had we not had yet another backup. We narrowly dodged the bullet but learned a very valuable lesson.

    Synology devices are amazing and very affordable. They have active monitoring and alerting of the drive health (something we were not aware of initially but now use). I now have three of these, backing up to each other in different locations as well as slow backups to Amazon.

  9. I would be super interested in seeing the directory structure and file organization you guys use. It’s a constant struggle, and always has been, from my days at CafeFX to my humble little animation studio, data is always a challenge.

  10. Just wanted to thank you for a great piece. It had just the right mix of overview and detail. I’m the senior producer for the University of Windsor in Canada. Like everyone, I’m drowning in data. I’m going to use so much of this as we reorganize our media systems. Please do more.

    pete

  11. I’ve been wanting to get a Synology for a safer backup. All of my hard drives are aging, so it’s just a matter of time before they fail. I was thinking of setting it up for a 2 drive failure (just in case), or is that overkill?

  12. Hi Joey, just wanted to add that in the video, you said you turn the DS1817+ off to replace the drive…

    You don’t need to.

    The bays are hot swap. All Synology that have the drive release on the front are hot swap, its only the cheaper ones where you need unscrew the case that need to be powered down.

    Also not mentioned is the naming scheme, the 1817+ is a “Plus” so faster CPU and 3 year warranty, its the 2017 model and finally, the 18 means it can store 18 drives. You can buy a expansion boxes to increase storage, this model can add 2 x DX517 to give more drives.

    As others have said above, I use 2 disk fault tolerance, instead of one, its a small price to pay for extra security. I also use the SSD cache to speed up reading and writing to it.

    Finally, in the next few days DSM6.1.3 should come out that finally fixes all the Spotlight indexing issues.

  13. Joey, Any chance you can do a run through on your travel protocol for equipment and how you choose or source your equipment when you guys go out?

Leave a Reply

Your email address will not be published.

Elegoo Mars 3 4K Resin Printer Review!

Making

Elegoo Mars 3 4K Resin Printer Review!

This week in 3D printing, we review the Elegoo Mars 3, a $30…

Podcast - This Is Only a Test

Visions – This is Only a Test 618 – 9/24/21

Norm and Kishore check in to talk about the success of the Inspiration 4 mission, the just-announced Mario movie, and what surprised us most about the Star Wars Visions series. Plus, Microsoft's revamped hardware lineup, new iPhone reviews, and the future of wireless charging.

A Tour of Grant Imahara’s Shop

Show And Tell

A Tour of Grant Imahara’s Shop

Friend and colleague Fon Davis gives Adam Savage a tour of G…

Adam’s One Day Builds: Hasbro Mandalorian Helmet Repaint!

One Day Builds

Adam’s One Day Builds: Hasbro Mandalorian Helmet R…

Adam shows you how to take Hasbro's The Black Series Mandalo…

Adam’s Favorite Tools: Heat Gun and Iron!

Making

Adam’s Favorite Tools: Heat Gun and Iron!

Inspired by one of his favorite YouTube channels (Ted Woodfo…

Conserving the X-Wing at Smithsonian

Culture

Conserving the X-Wing at Smithsonian

How do you restore and conserve an artifact like the X-Wing,…

The Mystery of This Original Rocketeer Stunt Pack

Show And Tell

The Mystery of This Original Rocketeer Stunt Pack

Adam visits Prop Store to take a look at an original stunt p…

Portable Fog Machines for Propmaking and Photography!

Show And Tell

Portable Fog Machines for Propmaking and Photograp…

Want to add fog or fake smoke to your projects? We round up …

Podcast - This Is Only a Test

Apple’s Diminishing Returns – This is Only a Test …

Another duocast this week, as Jeremy joins Norm to recap the announcements and disappointments from Apple's California Streaming event, including the new technolgoy in the latest generation of iPhones and Apple Watch. Plus, more thoughts on The Matrix, the launch of the Inspiration 4 mission, and the first trailer for Marvel's Hawkeye sho…

Adam Savage Tours Ghostbusters: Afterlife’s Farmhouse Set!

Culture

Adam Savage Tours Ghostbusters: Afterlife’s Farmho…

While on location of the Ghostbusters: Afterlife set, Adam S…