An intro to data deduplication

George — Tue, 23 Mar 2010 08:45:19 +0000

Data deduplication is a data backup process that eliminates duplicated data. At first thought, one may think that the word deduplication means the negation of duplication! Although, this is what happens in data deduplication that connotation is not the right one! Actually, the word deduplication means the division of that which is one whole into two or more pieces. In fact, the data deduplication mechanism divides data into blocks or chunks of bits in order to eliminate the redundant pieces within data.

A typical example which many authors refer to when explaining data deduplication is the email attachment scenario. Although, as a concept this example will help the readers understand better data deduplication, I would like to point out that data deduplication mechanisms do not generally operate at file level. For the sake of completeness, the example goes that the same email attachment may be present in many email messages; however, only one copy of it is stored during the backup or archive process. A simple task that can be done with a less processing hungry mechanism than data deduplication.

Data deduplication looks within files to find unique blocks of data and then computes a hash algorithm such as MD5 or SHA-1 to generate a unique number for each block. Each unique number is then stored in an index. When files are updated, only the changed data is saved. That is, only the changed blocks or bytes are saved. In practice, data deduplication is combined with other mechanisms such as, delta differencing as to achieve better and safer use of the storage space.

Some benefits of data deduplication:

Data deduplication offers lower storage space requirements and hence, saves you money on hardware (disks & tape ) and/or remote storage location costs.
May provide better recovery times in Disaster Recovery situations if well planned.
Most importantly, data deduplication reduces the data that must be sent across the Internet for remote or online backups.

Go for online remote backups

George — Mon, 09 Nov 2009 14:11:06 +0000

With today’s inexpensive and large-sized disk drives, disk-to-disk backups have become the default choice for many SMBs! However, off-site backup procedures require the use of tapes and hence, the use of related peripherals such as tape drives. Traditional backup methodologies bring along the need for more storage space and a sharp increase in operational costs! Although, data archives and off-site backups have their advantages, and are a must when meeting compliance regulations they must adhere to more rigid safeguards such as, encryption mechanisms and safe storage. Adequate drills need to be put in place to test recovery procedures on regular basis. Traditional methods create data duplication. To eliminate duplication of data, various methods and applications were created which are known as data de-duplication. Data de-duplication improves data protection, increases the speed of service, and reduces costs.

Why online backups are the way forward – because they provide:

User friendly backup/restore applications
Native off-site and archival services
Reduce secondary storage requirements through data de-duplication concepts
Intelligent data transfer methods through hashing algorithms
Safe – same data protection means such as, encrypted storage and connection
Scalability for future growth
Cheaper TCO (total cost of ownership)
Introduce pay-per-use concepts
Data retrieval anywhere-anytime concepts

IT Info Magazine » de-duplication

An intro to data deduplication

Go for online remote backups