Photo back up and compression

[Fighting_falcon_51]

Hey guys I have a quick question.

Right now I have 2x 1TB hard drives in a RAID 1 array to store my photo's and some other essential stuff. As it is getting full I plan to pop in 2x 2TB hard drives in a RAID 1 array to add more room.

One of my concerns is that if a catastrophe happens such as fire or theft all of my stuff would be gone.

I'm thinking about doing off site backups (2-3 different locations) but with 1TB drives at $60 a pop and 2TB drives at $99 a pop it's getting too costly for my budget. I can buy around 3x 500GB hard drives plus all the necessary hardware (PCI-E eSata card, eSata HDD docking station, shock proof cases ect;) for around $200. I do have a problem though. 500GB might not be enough space so I looked at compression programs like WinRAR and 7-Zip and their compression level did not fit the bill so I looked at imaging programs like Ghost and I'm still unsure whether 500GB is enough. So does anyone know of a utility that would let me heavily compress my photographs for back up? Also if someone has a better cost effective idea please let me know!

Thanks!

[Derf]

It is not possible, Jpeg is very highly compressed, that is why it is the most used file format for images... Your not going to compress it much at all with any app...sorry


I do not delete old images and I am probably at about the 1,000,000 mark or close to it. I have 2 1.2terabite drives full and have a 3rd drive offsite, one will be in Arizona next mont for long term storage. I have lost probably about 3k images due to a hard drive failure, I will not loose more images for that reason. Out of 1 million or so that is not too bad!

[gonzalu]

I will add a bit more to what Fred said... [KEEP A COPY OFF SITE!!!!] whatever you do, put off buying a lens, or a camera or whatever.. make sure you have ONE COPY OFF SITE... enough said. Ghost is an IMAGING technology, not compression. I also have completely abandoned Optical for long term storage as it has a poor long life history and it is untested at 100 years out. Magnetic MEdia is readily available, fast and cheaper every minute. Much easier too to carry a lot more in a much smaller space.

RAID 1 as a minimum with RAID 5 a good alternative... Stay away from SSDs for backups. 5400 RPM is better than 7200 or 10,000 RPM drives for long life and data safety. I also would suggest a lower aerial density for long life... meaning larger platters, more platters, less capacity. The larger the drive, the smaller the magnetic fields, the easier for corruption of data...

Now, compression. You can definitely compress JPGs and other compressed formats like TIFF and such. JPG is not more compressed than say a compressed TIFF. Smaller yes, but not because it has better compression. JPG is lossy compression, but that's a whole other topic

Suffice it to say, JPG is not as efficient as ZIP, RAR or 7z. While it does use Huffman coding after downsampling by discarding data, it can be further compressed. Right now the best compression for JPG is achieved by PAQ

http://en.wikipedia.org/wiki/PAQ

and it comes at a huge cost in SPEED and RAM. Not sure if you want to give up that much time but it will compress a lot. I say STAY CLEAR of compression. If there are any issues during the encoding or decoding, your files are hosed. There is a lot of error correction being done, but, be careful.

Other formats such as RAW files from Nikon and Canon, are highly compressible unless you have built in compression turned on such as Nikon RAW NEF Compressed, either loss-less or lossy compression)

BTW, 3TB Hitachi HD for $129 at Micro Center.. .I mean, AMAZING the low cost. I was laughing when you said you could not afford a 2TB drive for $99 (btw, now as low as $79) ... My first hard drive was a Micropolis 80MB (yes, MEGABYTE, not even 1 Gig) at a cost of $1,000 and it was a Full Height 5 1/4 inch behemoth and very slow for today's standards. How's that for affordable

So, please, do yourself a favor and don't skimp on backups...

Me? I have standardized on the wonderful Western Digital MyBook Edition II Quad for Mac... it has eSATA, FW800 and USB 2.0 interfaces, can be re-provisioned as RAID1 (default as RAID0 for speed, bad for backups) and has a really nice TOOL-FREE design for easy swapping of the drives in the event of a failure. Can be had for about $150 with two 1TB drives.

I have 6 of them for just pictures... I buy them in pairs.. .one goes to my Mom's one stays home. It is a real PITA to keep them in order if I edit anything on an archived photo but it is peace of mind.

All my main editing is done on-line on a RAID1 array made up of 2 x 2TB drives inside the PC. Lightroom is used to keep it all in check and all my moves are done from Lightroom... it then keeps track of which drive has what, regardless if it is sitting on-line or in the closet. You really do not want to do this by hand... The third drive is nearly full so today I bought the first of the new pair and will get the next in the pair for off-site once it is nearly full.

Kep in mind this can be done many many different ways and you should take opinion from anyone and decide which system works best for you. Definitely have on-site and off-site copies if you want to be safer.

[gonzalu]

On-Line (so called CLOUD) backup services are fine but can be costly if you get the amount you need for lots of photos. It is also really hard on your upload bandwidth and can take forever at typical 2GB upload speeds of most ISPs for home.

I suggest you look into a Pro account from SmugMug which for the yearly fee, it gives you UNLIMITED storage for your images. As long as the largest files are not over a certain size (this size is adjusted regularly for the latest in RAW file sizes) SmugMug will happily let you store it for not a penny more. If you want to store other stuff, you can pay extra for that and still may be cheaper than going it alone via DropBox or Carbonite.

SmugMug uses the wonderful Amazon S3 storage pool.

[lijk604]

Just one word about Carbonite. They only backs up what is on your hard drive. If you move files off your HDD, or switch your HDD then Carbonite drops those files. It's in the fine print, that they don't tell you about in their radio ads.

[Fighting_falcon_51]

I will add a bit more to what Fred said... [KEEP A COPY OFF SITE!!!!] whatever you do, put off buying a lens, or a camera or whatever.. make sure you have ONE COPY OFF SITE... enough said.

I would really love a nice flash right now but I was thinking the other day if I lost all my data I'm screwed. Before I started taking pictures all of my music, movies, tv shows are replaceable. Once my pictures are gone that's it. I'm going to prioritize securing my data before buy any other photo equipment.

5400 RPM is better than 7200 or 10,000 RPM drives for long life and data safety. I also would suggest a lower aerial density for long life... meaning larger platters, more platters, less capacity. The larger the drive, the smaller the magnetic fields, the easier for corruption of data...

For my storage purposes I use WD Green 5400 RPM drives they have served me well so far.

and it comes at a huge cost in SPEED and RAM. Not sure if you want to give up that much time but it will compress a lot. I say STAY CLEAR of compression. If there are any issues during the encoding or decoding, your files are hosed. There is a lot of error correction being done, but, be careful.

Other formats such as RAW files from Nikon and Canon, are highly compressible unless you have built in compression turned on such as Nikon RAW NEF Compressed, either loss-less or lossy compression)

I would let it sit and compress the files for weeks if I had to but I understand a lot can go wrong so I guess I'll stay away from compression

BTW, 3TB Hitachi HD for $129 at Micro Center.. .I mean, AMAZING the low cost. I was laughing when you said you could not afford a 2TB drive for $99 (btw, now as low as $79) ... My first hard drive was a Micropolis 80MB (yes, MEGABYTE, not even 1 Gig) at a cost of $1,000 and it was a Full Height 5 1/4 inch behemoth and very slow for today's standards. How's that for affordable

Well I meant if I was able to cram all of my photo's in 500GB and be set for awhile I would be happy because I can get 3x 500GB's plus all of the needed hardware for the price of the 2x 2TB's. Also my server can not handle 3TB's because the max the chipset will allow is 2TB.

So, please, do yourself a favor and don't skimp on backups...

Trust me I wont :p

lijk604
Just one word about Carbonite. They only backs up what is on your hard drive. If you move files off your HDD, or switch your HDD then Carbonite drops those files. It's in the fine print, that they don't tell you about in their radio ads.

Wow I did not know that! That is kind of sick that they do that.

After talking to people this is my current data management plan, if anyone sees any flaws please let me know. Also I appreciate everyone's help with this!

Plan:

Add 2 of these to my existing server case to have easy access to my drives.

Pick these up in pairs and RAID 1 them as more space is needed in the server.

Buy two 2 of these (just to store my photos and nothing else) and keep them off site and store them in this. (Back them up every 4-6 weeks)

Also get this for easy backup: http://www.newegg.com/Product/Produc...82E16817182197

[gonzalu]

How do you plan to INDEX and keep track of stuff? I sugggest Lightroom strongly!! One of the best features is the full size preview. Offf-line files are still fully visible, editable etc. Even if the drive or files are not on-line, LR tells you where they are and you can see them (the previews) Only when you need to output does LR need the original files.

All you have to do is label the drives uniquely so it makes it easy for you to identify them and pop them back in.

If you will be swapping them back and forth, you may want to look at a decent hot swap hardware solution... something with a DOCK and a tray for the drives or similar...

Don't forget about your interface either. It can make for painful transfers... eSATA, USB 3.0, USB 2.0, FW800/400 and of course, I can't wait for Thunderbolt!! ;-)

[Fighting_falcon_51]

How do you plan to INDEX and keep track of stuff? I sugggest Lightroom strongly!! One of the best features is the full size preview. Offf-line files are still fully visible, editable etc. Even if the drive or files are not on-line, LR tells you where they are and you can see them (the previews) Only when you need to output does LR need the original files.

All you have to do is label the drives uniquely so it makes it easy for you to identify them and pop them back in.

If you will be swapping them back and forth, you may want to look at a decent hot swap hardware solution... something with a DOCK and a tray for the drives or similar...

Don't forget about your interface either. It can make for painful transfers... eSATA, USB 3.0, USB 2.0, FW800/400 and of course, I can't wait for Thunderbolt!! ;-)

I am looking into Lightroom and it looks like a great utility. I would probably get a copy for my desktop and my Macbook.

As far as hot swap I am going to get one of those eSata docking stations and these hot swap bays. http://www.newegg.com/Product/Produc...-010-_-Product

As far as interfaces I am going to use eSata I can accomplish 6Gb/s and its compatible with all my machines (except my Macbook :( ).

Thanks for all your help!