Well, I'd like to state that we may actually start to see a third "twin": cloud computing.

To set the stage, let's start off with a very brief overview of the evolution of storage. First, we had direct-attached storage - big machines, with "big" dedicated storage attached. Often this was tape. Then we saw a movement to a shared, tape-based approach. That has spurred the evolution in tape capabilities and formats, notably from LTO1 to LTO2 to LTO3, then LTO4 and now with LTO5 in the near future. Each step in that evolution resulted in an approximate doubling of capacity in tape storage. But there are still the inherent limitations to only using tape: the serial-based nature of tapes, and the logistical delays when tapes are located off-site in a secure location.

Am I bummed out by that fact? Not really, and here's why.

First, when you look at the sources of data deduplication, backup datasets top the list for the amount of duplicate data because you're often backing up data that is between 95-99% the same as the last time you backed up the dataset. So if you have 30 days worth of backups, then even with a daily change rate of between 1-5%, you're going to get an awful lot of overlap.

If all you're looking to do is deduplication alone, there are a number of vendors out there that focus on just that piece of the puzzle. They'll happily ingest whatever data you throw at them. In most instances, these can be found in the form of a dedicated deduplication appliance. There might be some indigestion along the way, but generally they work OK, at least until they get to the 90-95% capacity mark, at which point many customers see a rapid downgrade in performance often as a result of the need for enough resources during the scheduled general processing. These processes include the need to calculate check-sums on the data to ensure that it's not been corrupted, or the need to move data between nodes for load-balancing (which therefore would improve performance).

What are the downsides to that approach?

The first is cost, especially compared to the prices of increasingly cheap commodity disk. When you actually look under the covers, there are only a handful of disk manufacturers worldwide. So you may be paying a premium for essentially the same physical drives and memory under the covers. And just to remind myself how cheap is cheap, I often pop over to places like eBay to get a real world gauge on prices. For general pricing for storage systems/software, you also can check out a very useful blog/resource here, courtesy of Storage Mojo.

But I want to take a quick look at two other underlying drivers that I believe will continue to underpin the growth in deduplication beyond a single budget cycle or fiscal year.

Backing up VM's requires deduplication

The first driver is of course, Virtualization. We have seen a huge amount of interest among our customers in virtualization (mostly VMware, but increasingly Hyper-V as well). The way I think of VM environments is sort of like the underlying premise of "The Matrix". Basically, we're digitizing what were previously standalone physical infrastructures. In addition, with VM environments, there is often a much more rigorous and enforceable need for standardization. For instance, if you're going to rapidly create 500 VMs, then having a set of five standard templates to choose from makes things a lot easier to get up and running, and much MUCH easier to manage!

So, the quick version is that Data Domain Boost is built upon Symantec's proprietary OST interface to move the hashing and comparison work largely to the Media Server. This brings them closer to the way CommVault does deduplication (along with the Media Server-based configuration of Symantec's PureDisk technology). I guess imitation is really the sincerest form of flattery after all!

But what I'm really wondering about is how EMC justifies asking for a huge hardware premium for their boxes if most (or all?) of the deduplication is actually being done outside of the Data Domain box? And why would customers be happy being locked into a proprietary interface that only works today if they use Symantec and Data Domain?

NetApp has been the most visible proponent and platform for deduplication of production data. And they're about to get some company: Compellent. CommVault was a Gigabyte sponsor at Compellent's recent C-Drive show, where we got some insight as to where they're headed during their CEO's keynote speech on Monday, May 3. There, Philip Soran called out how the world is beginning the transition beyond a service-based economy to one that's data-driven. For instance, if you look at Facebook, it only took four years to get to 400 million users. And of course deduplication, even at the production tier, starts to provide meaningful benefits, especially when your data is growing exponentially.

With all due respect to each of them, I think we all actually agree that better dedupe ratios are a good thing. Of course 20:1 is going to get 5% more savings compared to 10:1. But my main point with the dedupe ratios is that too many vendors start rattling off high dedupe ratios as "bait". It's a bunch of marketing hype, and yes, as a product marketer, I do see the irony.

You should try out different approaches and vendors, and see what best fits in your environment. The dedupe ratio is then going to be a by-product of your dataset change rates and retention policies. Focusing on dedupe ratios puts the cart before the horse.

So from my (evolving) perspective, focusing too much on dedupe ratios, rather than overall acquisition and operational costs, is an imperfect barometer of potential savings. But I still contend that once you've captured 90% to 95% of the potential savings from dedupe (10:1 and 20:1 dedupe ratios), there are likely to be other IT initiatives that then might be better candidates for your time and investment.

Below, I'm summarizing the essence of his arguments (as I interpret them, which you really should read for yourself) and my response and reaction:

1) Dedupe ratios do matter because that changes how much disk you buy.

I agree – better dedupe ratios translate into less outlays in terms of acquisition. BUT, there are a few points I was trying to highlight here to put the focus on dedupe ratios into better perspective (not eliminate, but put it in context):

• First, some vendors have in the past been touting dedupe ratios of 50:1, 100:1 or even 500:1. In my mind, that's more fluff than reality, so after some point it just starts to seem a bit absurd, especially when most vendors get about the same dedupe ratios (more on that in the next bullet).

• Second, as I noted, most dedupe vendors will get about the same reduction most of the time. There is the possibility of meaningful differences some of the time. So you should research which approach works best in your environment (for example source- versus target-based deduplication, or file-level versus block-level), and which vendor is best for that approach. But again I wouldn't get too wowed with impressive sounding dedupe ratios. You should really focus on picking the right vendor/approach first. Your actual dedupe ratios will be an outcome of your dataset change rates and retention timeframe.

• Third, incremental increases in savings are going to shrink. That's a fact of mathematics not deduplication: 5:1 delivers 80% savings, 10:1 delivers an extra 10%, 20:1 delivers an extra 5%. Trying to tweak your deduplication set-up to get an additional 5% savings may not be worth the time, especially if you're managing a smaller dataset. If you have PBs that could be huge, but then your selection criteria is likely to be a lot broader than using dedupe ratios when it comes to adopting deduplication and choosing the right vendor/approach.

• Using a 100TB example, the differences between 10:1 and 20:1 dedupe ratios is 5TBs, which is great since you're buying only half the storage (5TBs versus 10TBs). But at $2K/TB (CDW is great to do spot checks on the latest prices), that's $10K. It's not just about 5TBs versus 10TBs, it's about an additional outlay of $10K. Is that really the best thing to focus on to incrementally lower your costs? And again you're not comparing vendors but figuring out at what point your overworked IT team can go focus on other initiatives.

For those of your interested in learning more, you can access the report here.

The key message for me still is focused on simplicity. Storage Reporting Management (SRM), Archiving, and Deduplication have each been available in the market for almost a decade, more or less. So why haven't more organizations combined all three to maximize data reduction? Easy: complexity of implementation and management. These negated the benefits of less storage acquisition. You replaced hardware investment with headcount investment.

First, our own anecdotal experiences (and internal testing) indicate that when it comes time to see what's stored on disk, most vendors (ourselves included) end up with about the same capacity usage.

Your dedupe ratios are going to largely depend on your dataset characteristics: specifically the change-rate between cycles and the retention period. The higher the change rate (ex: 10% versus 2%), the lower the dedupe ratio, as the former rate implies that only 90% of the data was the same from the previous cycle, versus 98% in the latter case. Another factor that affects dedupe ratios is the retention period of your backup or archive jobs: the longer the retention period, the higher the dedupe ratio. That means that you're much more likely to find greater redundancy across 30 backup or archive cycles versus "only" 7 cycles. One easy way to think about this is to walk through a daily backup scenario, while keeping the change rate between daily backups to be 0%. In that scenario, retention of 7 daily backup jobs will result in a dedupe ratio of 7:1. Extending the retention period to 30 days increases the dedupe ratio to 30:1.

The graph below illustrates what your dedupe ratios would look like based on 5 different change rates (1%, 2%, 5%, 10%, and 20%) across a 90 day window, starting with your baseline of cycle 1. So you can see that if you have a fairly high change rate of 20%, that's going to plateau your dedupe ratios fairly quickly at about a 5:1 ratio. Archive data is an example of such data that changes a fair bit between cycles, for example if you're only archiving data every 30 days. However if you have a low change rate, that can continue to provide some fairly good ongoing results. In general, we see a range of change rates that are often related to the specific data types and workloads, rather than company specific. However, with deduplication we are seeing much longer retention periods for backups, usually increasing the retention period to about a month's worth of backups kept on disk for fast recovery.

I'll take a pass on that for this year at least. First of all, I think there are plenty of folks out there who'll cast their gaze into the crystal ball faster than you can say F-Y-I. So instead, I looked around and thought it would be more interesting to review the "deep thoughts" from some of the respected pundits in the industry. They are (in no particular order):

1. Steve Duplessie of Enterprise Strategy Group, who reviewed his thoughts on "Where Is the IT Spending Going to Happen in 2010?";

2. Curtis Preston, a frequent contributor to many industry articles and seen at many a storage event; and

3. Chuck Hollis from a little company outside of Beantown , gazing into his very own "Storage Crystal Ball".

Steve provided a nice overview of what he sees as good, bad and just plain interesting. Curtis gives us a nice moment to stop and reflect on the events and changes of 2009. Chuck gave us a somewhat extensive peek at where he sees the world going and EMC's place within that context.

Well, regardless of their offering, the answer so far has been the same. New EMC/Data Domain models don't change the fundamental differences between traditional deduplication versus CommVault's approach to deduplication.

All of the dedupe appliance vendors continue to update their models to respond to rapid data growth. That's pretty much expected for two reasons. First, a data growth rate of even 25% per year means that the total amount of data that needs to be backed up doubles in just about three years. Second when you factor in even tighter SLA agreements that demand more and more backup (versus retention) data be kept on disk, it's easy to see why every appliance vendor would need to scale to keep up with bigger workloads.

(For example: If you have a 10TB dataset and back that up 5 days every week, that's 50TBs which is stored on 5TB of usable capacity given a 10:1 dedupe ratio. Now if that production dataset doubles over time to 20TBs, and you decide to now keep 20 backups on disk instead of the original 5 backups, that's 400TBs of backup data. Deduplicated down 10:1, that would now require 40TBs. That 8X increase is driven by both the growth of the original dataset and the extension of your restore-from-disk window from 5 days to 20 days.)

Keep in mind I said "more than." Deduplication definitely has a role in the data reduction story. So once you actually know where your data is, how much of it is actually being accessed/modified, and have archived what's "stale" then it's time to consider how deduplication can lower the costs even further. Why dedupe now, versus at the beginning? With the stale data moved off to Tier2, you not only have the potential to double the capacity on your Tier1 storage (assuming 50% of data is found to be stale), but also reduce the amount of dedupe processing you need to do with the data that remains. And that's true no matter who you ultimately choose for deduplication.

So, what would happen with deduplication in the picture? For the sake of argument, let's use a conservative dedupe ratio of 5:1. Now keep in mind that deduplication ratios are much more a factor of data change rates and retention periods. So for ease-of-comparison, we're assuming that you get 5:1 no matter which dedupe vendor you choose to use.

Our baseline case started with backing up the same 10TBs of data 30 times = 300TBs of cumulative raw backup data. With a 5:1 ratio, that reduces down to 60TBs. Sounds great, and it is if you're only paying $2K/TB for the backup disk capacity. Given that number, the costs come down even further than "archive alone", down to $220K. That is $100K for the 10TBs on primary storage, and $120K for the 60TBs on Tier2 storage.

What does mean for you? Well we'll start with a greatly simplified example and walk through the various permutations.

First, let's establish the baseline for comparison (always need a baseline for comparison). Say you've got 10TBs of data and need full backups for 30 days to facilitate rapid recovery. In a normal scenario that would mean that you would have 10TBs on primary (Tier1 storage), and 300TBs on Tier2 (30 backups * 10TBs per backup). At a cost of $10K/usable TB for production, and $2K/usable TB for Tier2, you'd be spending about $700K for this kind of protection/retention. That's based on $100K for Tier1 (10TBs * $10K/TB), and $600K for Tier2 (300TB * $2K/TB).

In the meantime, it seemed to me that in pushing the next "big" thing, many folks out there are happy to pin their hopes and dreams solely on deduplication. But they would be doing themselves a disservice. For one thing, deduplication is a great approach, but it doesn't solve the underlying problems in processes, people or platforms on its own. If you don't fix the root causes, no amount of deduplication will prevent you from facing the same issues, just in one, two, or (if you're lucky) three years.

For one thing, in many cases it doesn't solve the issues on the front-end, on primary storage. Most folks are implementing deduplication for backup and archive, where they get the most amount of data reduction across multiple backup/archive cycles. However, if the breakdown in your backup process actually originates up-front with the size of your primary datasets and/or length of your backup windows, then deduplication may not be the total answer. So even though the backed up data is going to occupy less space, each uncompressed full backup job (generally required if you're using a device-based target dedupe approach) will still have to churn through larger and larger amounts of production data.

One episode in particular - "The Trouble with Tribbles" - still resonates with me, especially when I think about virtual environments. I'm sure I'm not the only one who's made the connection. That particular episode is centered on cute cuddly little creatures that start to multiply out of control. This is somewhat akin to virtual machines (VMs). No, VMs are not cute and cuddly, but they do seem to multiply like mad simply because it is so easy to create and deploy a new virtual machine compared to setting up a new physical server.

To paraphase the Wikipedia entry on the Star Trek episode: "The "trouble" with VMs is that they reproduce far too quickly and are capable of eating your spare storage capacity barren if their breeding is not controlled; in the words of Dr. McCoy, 'they are born pregnant' and threaten to consume all the available storage."

Deduplication is, in my mind, one of the key foundation technologies required to help get things (virtually) under control. This was evident during the week of VMworld 2009, recently held in San Francisco. We saw a tremendous amount of interest on the show floor, as the image at the top of this post shows, as customers (and prospects) kept us on our toes all day long at our booth. It was great talking to them about Simpana 8, and the support for virtualization and block-level deduplication that we introduced earlier this year.

Our approach to deduplication doesn't restrict the processing to the data source, or the storage target. It's a distributed approach that delivers more flexibility. So at the risk of perhaps re-hashing what you may already know, let's do a step-by-step walk through of what happens with CommVault deduplication and then we'll contrast that with EMC's multiple approaches.

Deduplication: Not Exactly a Big Stretch (if you're CommVault that is)

Regardless of whether you're deduplicating data or not, in a CommVault environment, the client (the server that hosts the application, database, file-system, etc.) picks the data that needs to be protected based on your backup policies, then optionally compresses the data, optionally encrypts the data and sends it on to the Media Agent (analogous to a Media Server in a Symantec NetBackup environment). The Media Agent then sends the data to the target storage system(s) and updates our index so that we know where the data processed as part of the backup job is held.

Now, when you throw deduplication into the picture, the client does one additional step: after picking/compressing the data, the client generates a hash signature that is sent along with the compressed/encrypted data to the Media Agent. The Media Agent then compares the data segment's hash signature to what we've seen before and either updates the index with a link to the existing data already on disk, or sends the unique data down to disk to be stored and updates our index and our dedupe database with the latest info. That's "it". The way we deduplicate the data is so deeply embedded into how we handle data in general that it wasn't much of stretch to then add deduplication into the data management mix.

Most of the commenters wanted to know how I could price generic storage so cheaply in the comparison, in which I used $1,500 per TB as a figure. Those are fair comments, so let's revisit this comparison using the prices below, which I pulled from the CDW website. Putting myself in your shoes, I wanted to stick to looking at "above board" prices versus getting into company/deal-specific pricing and all the discounting that could entail. So, the table below is just a selection of storage options available through CDW as of late...

To start, many traditional target device-based deduplication vendors often tout data reductions in the range of 10:1 to 20:1. So, let's be conservative and use 10:1 as a dedupe ratio to see what that really amounts to in dollars saved.

So, if you are backing up 100TBs of raw data, then with a 10:1 reduction, you're storing 10TBs on the target deduplication device. This sounds great because you're saving 90TBs of disk capacity. On the other hand, if you choose not to de-duplicate the data, then you'll end up with something like 50TB on regular commodity disk (on average, assuming you still compress the data).

That sounds like a lot of savings: 10TBs on the dedupe appliance versus 50TBs on commodity disk. But wait, decent commodity storage costs about $1,500 per TB (or even less), and based on our own research and experience, we estimate that the one of the most successful dedupe appliance vendors now charges a street price of between $7,000 and $9,500 per usable TB (averaging the two yields $8,250/usable TB). This is true even though the appliance uses essentially the same disk underneath, which is manufactured by just a handful of the remaining disk OEM manufacturers.

First, it became obvious that some folks are getting mixed up on some of the nuance. Stating that "tape is not going away" does not mean that tape is going to be used in the same way as it has been in the past. Disk costs are clearly headed in a direction where tape is rapidly becoming much less attractive for general backup and recovery. What I am saying is that there are cases (medical information for example) where the data needs to be kept for decades. For instance, OSHA regulations state the certain records need to be kept for 30 years, and a majority of hospitals surveyed in a 2008 study by the National Institutes of Health kept permanent records of their patients. So when you're looking at literally decades of data retention, tape will continue to be a viable option for those customers.

One thing I've noticed over the years is that a lot of time, what's old is often new again. - like the Pet Shop Boys, for instance. It seems that many developments in the industry happen in cycles. Take "virtual machines" as an example. While at IBM, I worked closely with the eServer Group at IBM on the software side of things. At the time, IBM was making a big push around Linux, and VMware was one of the big partners as we sought to evangelize that OS technology. At the same time I came by accident to the realization that IBM actually had a hand in virtual machines since 1972, thirty years earlier! So it made an impression on me that here we had two separate "waves" of adoption, decades apart, which in many respects addressed broadly similar goals but in different contexts.