The Syncing Nightmare of Too Many Computers, Backups and Cloud Drives!

The Problems:

  • I have three home computers, three work computers, four external hard drives, and six cloud drive accounts, with tens of thousands of original files that are multiplied into hundreds of thousands stored on backup and cloud drives.
  • I have personal files and work files but often I want access to both kinds no matter where I’m at.
  • If I delete a file from the computer I’m working on, it’s not deleted from all the backed up copies.
  • Every time I look at a different drive I have to constantly decide again if I want to keep or delete a file.
  • Because I have 4 PCs, 1 Mac and 1 Linux machine I really don’t have a primary My Documents folder.
  • I have copied files in so many locations that I’m not sure which is the primary backup anymore.
  • I had a 1.5 TB drive fail and lost 200+ documentaries I was saving.
  • I have too many files from using personal computers for over 30 years.

thematrix

The Goals:

  • I want two perfectly organized Master Filing Systems, one personal, one work.
  • I want the easiest system possible for maintaining order and security.
  • I want to get rid of the external hard drives.
  • I want the fewest copies that equals the maximum security.
  • I want each of my Master Filing Systems to be backed up.
  • I want the files to have an organization structure that makes it obvious where everything is and belongs.
  • I want this to be my last file reorganization that will last me the rest of my life.
  • I want to clean out all the clutter and ancient files I no longer need.

Questions to Consider:

  • Can I trust a cloud drive like Dropbox or SkyDrive to be my Master Filing System?   This certainly would make using six computers and my mobile devices the easiest to use.
  • Would it be practical to use a cloud drive as my Master Filing System, and then use software to mirror the  cloud to local computers as backups?
  • Which cloud drive service is worthy of being my Master File Location?
  • How do I handle deleted files so the deleted files are removed from all the backups, but yet stored somewhere for long term recovery?
  • Do I need to worry about music files now that I have Amazon Cloud Player, Google Music, Rdio, and Rhapsody?
  • How do I keep my photos organized in my Master File Location and in-sync with gallery sites like Picasa?
  • What’s the best place to store emails?
  • Should I have a Master Deleted File System?
  • Does any cloud drive service offer a journaling file system?
  • When I create a Master Filing System, what folder structure should I use?
  • Are some file types too large to save permanently?
  • Can Dropbox or SkyDrive work like a roaming profile/home drive on a Windows Server?

Some Answers to Help Decide:

  • Dropbox offers it’s Packrat feature of unlimited undeletes for $39/yr. 
  • Using Dropbox means spending $139 a year minimum – the price of an external drive, but external drives take power, eventually, die, fill full of clutter, and take work to move from computer to computer.
  • Dropbox and SkyDrive have virtual drives making them easier to use than Amazon Cloud Drive, and allowing software like Second Copy to access them.
  • Dropbox virtual drives are available for all my my computers and devices.
  • Second Copy would let me replicate files from cloud drives to my PCs, thus making them the backups and not the cloud drives.
  • I could buy Dropbox for my personal Master File System and use SkyDrive for my work Master File System.  (I have a 25gb SkyDrive account because of work).
  • I have a 50gb Amazon Cloud Drive account that I could use as a cloud backup.
  • If I use Dropbox as my Master Filing System I could go around to all my computers, backups and other cloud drives and re-file all the files I want into it.  That might be the easiest way to create a Master Filing System.
  • For $25 a year Amazon keeps up to 250,000 songs for me in their Cloud Player and a copy in the Cloud Drive.  They also give me 50 GB of cloud space for other files.  Is this secure enough for maintaining my music library?

Are Some Files Too Big To Store Permanently?

When I lost the 1.5 TB of documentaries from my HTPC I began to wonder if some files are too large to save permanently.  At Dropbox’s rates, I’d have to spend $1500 a year to have maintained my documentary collection online.  I’m not going to do that.  Nor do I want to run a home server with backups to support such a library.  Maintaining 140 GB of music files is annoying enough, with copies on my main computer, two other computers, two external drives and at Amazon and Google.  But keeping a perfect copy of my music library in sync is a nightmare.  Then I have a large library of audiobook files scattered across several computers to worry about.  Are they even worth the worry when I spend 99.9% of time listening to books from Audible.com?

The solution here is just to live with what Netflix, Audible and Rdio provides to me, and not try to own my own library of movies, music and audiobooks.  This would certainly simplify a good deal of file management.

Conclusions:

Writing all of this helped me to think things through.  I’ve decided to make Dropbox my Master Filing System for personal files.  Currently I have 13 GB of free space, but I might have to up it to 100 GB ($99/year).  I haven’t decided if I want to spring for the $39/year Packrat feature, but it’s tempting.  It will probably take me months of going through all my file locations and filing what I want to save into my new Master Filing System.  I certainly hope that Dropbox doesn’t go out of business.

I’ve been using Dropbox for a while now, but as a test, I’ll start using it as my primary My Documents folder for all my devices to see what happens.

For a backup to my Master Filing System, I’ll use Second Copy to replicate Dropbox to a folder on my local hard drive.  I haven’t decided if I’ll replicate to two different machines or not.

I might reduce my home computers from three to two and get rid of all the external hard drives.  Since I’d run Windows Media Center on both of them, I might mirror my recorded shows to both machines, but this means maintaining 2 TB drives on both machines, and I’m not sure I like that.  I’m awful tempted to give up trying to save recorded video or even collecting DVDs.

If I succeed with using Dropbox as my Master Filing System and I get a new computer, it will be very easy to set up and start working.  Just install Dropbox client and my software.  Then create a backup folder and start replicating Dropbox files to it as the new primary backup.

Settling on Dropbox means my home files will be available at work, but also on my iPad and iPod touch or even any computer I sit down to use as long as it’s on the internet.  Let’s hope this works out.

JWH – 8/3/12

Living in the Cloud: Google Music, Amazon Cloud and iTunes Match

I have over 18,000 songs ripped from CDs I’ve been buying since 1983 or 1984.  It was a big project to rip them, and I bought a couple external hard drives to back up my work.  One of those drives stopped working recently, and it’s a pain to keep master library and backups in sync, especially since I keep one drive off site.  In fact, I gave up on keeping my collection and backups in sync.  So when cloud music storage came out I thought wow, this is a great idea.

The first service I tried was Google Music since it was free.  It has an upload app that runs as a background service and I spent a couple weeks getting my collection online.  I mainly listen to music at my home office computer, my work office computer, and in the den with a HTPC hooked up to my stereo system.  I have an iPad and iPod touch, but I don’t like listening to music on those devices.   I just don’t like hearing music through earbuds.

I did test Google music out on my iPad and strangely enough the album listings look best on the iPad.  Google Music looks bad on large screen desktop monitors because I think the album thumbnail images are optimized for phones and tablets.  Their web player has basic controls for play/pause, next, previous, repeat and shuffle – nothing fancy but gets the job done.

Here is the album view.  You can blow up all images by clicking on them.

google-music

Here is what it looks like to play an album – the controls are along the bottom.

google-music-player

Then I uploaded my music to Amazon Cloud.  It also took a couple weeks, but it was a web app that kept crashing.  Also, Amazon’s upload app found all my audio books and uploaded them.  I really didn’t want it to do that, but it did.  I already had 20gb of storage at Amazon’s cloud because I had bought an album on promotion, and Amazon recently gave unlimited music upload space to anyone with 20gb of space or more.  When the renewal comes up, storing my music on Amazon Cloud will be $20 a year.  Here’s the album view for the Amazon Cloud player.

amazon.cloug

Notice the album covers are nicer looking.  Here’s what the album player looks like.  The same basic controls as Google Music.

amazon-cloud-player

iTunes Music Match works different.  It works through iTunes – which I hate.  I was hoping it would have a web client too, but it doesn’t.  So I can’t play music on my Linux machines.  Nor do I want to install iTunes on all my machines.  And for some strange reason iTunes in album view iTunes sorts by artist, so I couldn’t recreate the album images like I did with Google Music and Amazon Cloud.

itunes

Here’s the player view.

itunes-player

Because iTunes Music Match costs $25 a year, and it’s from Apple, which has a reputation for style and slickness, I thought I would like it best.  I didn’t.  I like it least because it’s tied to iTunes.  The music match feature worked beautifully, and within minutes 15,000 of my albums were online.  It took two days to upload 3,350 unmatched albums.  This is a more sophisticated way to get albums into the cloud, but playing them is limited to machines with iTunes.

Another strange thing about iTunes is it did the poorest job of finding album covers.  Apple is so visual that I found this disappointing.  I have spent a lot of extra time trying to find the covers and put them into iTunes so I can enjoy album flow viewing, but I gave up somewhere in the D’s.  Now there’s a company that fixes this problem with a program called Tune-Up.  However, it’s expensive.  $39.95 per year, or $49.95 for a lifetime license.  It annoys me so much not having the artwork that I am tempted to spend the money, but I’ve decided that I just don’t like iTunes Music Match if I have to use iTunes.

Finally, iTunes plays music differently.  Google and Amazon streams from the cloud.  No internet, no music.  iTunes lets you download songs from the cloud.  The others do too, but iTunes seems to emphasize download.  You can have up to 10 devices sharing your Music Match cloud library, but what appears to happen is the music gets downloaded to each new device.  You can tell your satellite devices to intentionally download the music so you can play it offline, and this will be a great feature for most people who use iPhones and iPads.  However, it will fill up your device with music.  I prefer streaming.

As far as I was concerned iTunes Music Match was $25 down the drain.  However, Mac users who own Mac, iPhone and iPad will always have iTunes and so Music Match will be worth it to them.  iTunes Music Match seems geared to iPhone, iPod touch and iPad users, to help on-the-go users get music down from the cloud.

Now I have to decide between Google and Amazon.  Because I’m a dedicated Chrome user I’m partial to Google.  Because I’m a dedicated Amazon customer, I’m equally partial to Amazon.  I’m leaning towards the Amazon cloud because the player looks better.  However, it will cost me $20 a year.  I’m going to maintain both for now, or until I see what Google charges.  I’m an Amazon Prime user, so I wish they’d made unlimited music storage free for Prime members.

My next project is to thin out my collection.  I’m not sure how well Google and Amazon update their clouds.  I want to make one perfect copy of my library in Windows Media Player and hopefully Amazon and Google will keep this master library in sync.  Another test will be to download my collection to my work machine to see how well these clouds can be used as backup to restore my collection.  But these are for future reports because it will take months to do all this.

JWH – 2/12/12

Living in the Cloud: A File Structure for Life

I’ve been messing with microcomputers since 1979, but it wasn’t until the mid-90s that I realized I was creating data I didn’t ever want to lose.  By the mid-00s with digital music, video and photography it’s obvious to everyone that we had invisible possessions we’d want to keep for life.  This presents a number of problems.  How long can I preserve photos like this of my great grandparents?

1920s - Dad's father on right - with parents and brothers - cropped

File Formats

My first efforts of writing fiction was on a Commodore 64 – and even if I had any of its floppies, I couldn’t read the discs, nor would I have a word processing program to read the files.  When I got a PC I bought Word Perfect, but that was many PCs ago, and I’ve since converted those files to Word.  If I live to be 100 (1951) will I still be able to read those files?  If by a miracle I do live a century I’m pretty sure I’ll be a sentimental old slob who cries over his ancient snapshots.  Will I be able to find the ones I want and still view them?  Is it .jpg forever?  I’ve been buying audiobooks from Audible.com since 2002, will I still have my audiobook library to play in 2051?  Or all the Kindle books I’m buying now?

Standard file formats are critical to long-term preservation of data.  How long will Amazon maintain the DRM copy protection on my Kindle books and Audible audiobooks?

Data Migration

I’ve lost count how many computers I’ve own after eleven.  Every time I get a new computer I need to move all my files over and that’s a pain.  I’m always making a new folder and throwing stuff in it, so the number of files I’m saving constantly grows, and every few years I try to clean things out and it’s a big job.  Usually when I get a new computer I just copy everything in My Documents to the new My Documents folder.  But what if I got a Mac?  Or what if in 2022 they come out with some far out new computer system?

File Organization

If you start with one folder, and organize your digital life into sub-folders, what is the best structure?  I sure wished that iTunes hadn’t put ripped audio books into My Music years ago, because that’s causing problems moving my music to the cloud.   Is there a way to plan for future snafus?

Is there an optimal structure that will stand the test of time.  By structure, I mean folder organization.

\Jim Harris

\Audio Books

\Data

\Ebooks

\Mind Maps

\Music

\Numbers

\PDF

\Photos

\Words

\Essays

\Fiction

\Videos

Let’s imagine a future where we have federally regulated data banks like we have money banks and we can trust them implicitly.  In this future, data bank replicates our data in layers of backups, that for anything short of Armageddon, will be completely secure.  Should we put all our data in one place?  In the above chart I could remove \Music because I have my music stored at Google, Amazon and Apple.  I could also remove \Photos because of Picasaweb.  I could also remove \Audio Books and \Ebooks because of Amazon.

Because we don’t have data banks and because my cloud storage is limited at Dropbox and SkyDrive, I will let those other companies maintain my media files.  But if we did have trustworthy data banks, I’d probably want all my content in one location, which means precise organization is important if I’m collecting files for life.

Data Inheritance

There is another thing to consider – what happens when we die?  When our parents die we inherit their papers, books, records, photos and so on.  Won’t we do the same things with digital files?  When I die I want my file structure copied over to my wife’s data bank, and if I wanted, I’d like to give copies to all interested relatives and friends too.  Having a well organize file structure would make it easier for people to go through my digital processions.

Cloud of the Future

Someday we will have data banks.  We might even have laws that require our data to be saved for historians.  Can you imagine scholars from 2782 AD trying to research our times?  I saw a wonderful show on Nova last night, “Mysteries of a Masterpiece” about art historians working to validate a work of art as Leonardo da Vinci.  The scientists had lots of physical artifacts to examine.  If our world goes digital, what will future researchers have to figure out how we lived?

Moving to the cloud is the first step towards this future where we have data banks and preserving digital data for all time.

JWH – 1/26/12

Living the in Cloud: Dropbox and Evernote

If you access the internet from only one device this article won’t mean much to you, so I won’t mind if you go read something more interesting.

However, if you own a computer and a tablet, or a computer, smartphone and tablet, then reading about Evernote and Dropbox might be worth a few minutes of your time.  If you’re like me and juggle a lot of devices then learning to squirrel your digital crap all over the cloud becomes more vital.  At home I have both a Windows and Linux desktop, at work I have Windows, Linux and Mac desktops, and between the two locations I have an iPod touch and iPad 2.

What a pain it is to think of something you want and realize you left it in your other computer.  Moving to the cloud is in its early stages, so 100% tried and true solutions are in the future.  As society evolves towards the day when internet access has five nines of uptime, 99.999%, then we can develop a new paradigm of trusting our files to the cloud, and that will be the difference between life before personal computers and life after them.

dropboxevernote

Although I’d like to be a cyborg and meld my brain with silicon I’m not quite there yet, but I do think of the Internet as my auxiliary brain and that presents some problems.  Before the Internet going to work meant leaving my main auxiliary brain at home – how inconvenient.   Sure, someone invented the laptop and it was a good idea at the time, but it was only a stopgap solution.  After we got smartphones and tablets it became pretty obvious trying to sync all our crap between every device we owned was a losing battle.  The solution was to put all digital kipple in one location and then let all the machines, big and small, fetch what we needed from that primary storage.

What this means is the cloud is our new auxiliary memory and the machine we use is less important.  The old fanboy battle between PC versus Mac becomes silly.  If I can read my docs, listen to my music and look at my photos from any device, does it matter how big or small it is, or who made it, or even who owns it?  Instant access is what counts.  Memories are best served fast.

When the cloud becomes our digital memory deciding how to organize memories becomes significant.  I’m playing with two tools, Dropbox and Evernote.  Both are free to use with an introductory amount of cloud space, but fill up your cloud attic, and you’ll have to pay for more space for your white elephants.  That’s cool, but I haven’t committed to either one yet because I’m still evaluating how they store my memories.  I’ll probably buy into both, but I haven’t decided.

Dropbox is like having a hard drive in the cloud.  You create folders and store whatever kind of files you want.  It’s very computer centric.   When you join you get 2gb of free space.  If you convince a friend to join they give you another 250mb of space.  If you get enough friends to join you can get up to 8gb of free space, but after that you rent larger blocks of space.  By the way, if you join from this link I’ll earn some extra space.

Evernote is different, it’s database centric.  Evernote is a free-form database where you leave notes, either ones you type, or ones you email via a smartphone, or clip from the web, or cut and paste from your own computer documents.  You can even embed PDF files.  If you spend $45 a year, upgrading to the Premium version, gets you more memory processing features and more storage space.

The neat thing about Evernote is being able to search your collection of notes.  Since I’m getting old and the access speed on my biological memory has become erratic, untrustworthy and slow, having cloud base memory with search is nifty indeed.  Because Evernote is a free-form database, throw your data in any old way, it doesn’t matter, and let search find it for you.  You can be as sloppy or neat as your personality.

Both programs install as programs on your computers, work from web apps, or install as apps on iOS and Android smartphones and tablets.

I can access Evernote and Dropbox from my PC, Mac, Linux, iPad, and iPod touch.  If I think of something I want to remember, or read something I want to remember, I can choose to remember it the old way, or I can memorize it in my auxiliary memory.

JWH – 1/24/12