Imaging with dd

[Red Savina]

Suppose I want to create an image of hda3 that will be about 11GB and break it up to transfer to DVDs. Will the following work?

# mount -t ntfs /dev/hda3 /mnt/hda3
# dd if=/dev/zero of=/mnt/hda3/delete.me bs=8M; rm /mnt/hda3/delete.me
# umount /dev/hda3
# mount -t vfat /dev/hda7 /mnt/hda7
# dd if=/dev/hda3 conv=sync,noerror bs=64K | gzip -c | split -db 4096m /mnt/hda7/images/hda3.img.gz

to create the three files:

/mnt/hda7/images/hda3.img.gz00
/mnt/hda7/images/hda3.img.gz01
/mnt/hda7/images/hda3.img.gz02

Then to restore the image:
# cd /mnt/hda7/images
# cat hda3.img.gz00 hda3.img.gz01 hda3.img.gz02 | gunzip -c | dd of=/dev/hda3 conv=sync,noerror bs=64K
# cd /

When creating the file "delete.me" with a block size of 8M, will this zero-fill areas of contiguous unused space that are less than 8M in length? If not, would these typically be numerous or rare, or depend on the filesystem?

Is "-db 4096m" acceptable syntax for the split options? Can I combine size multipliers - for example: "4km" instead of "4096m"?

Is it acceptable (or wise) to use ddrescue instead of dd as in:

# ddrescue -b=64Ki -v /dev/hda3 | gzip -c | split -db 4096m /mnt/hda7/images/hda3.img.gz
# cat hda3.img.gz00 hda3.img.gz01 hda3.img.gz02 | gunzip -c | ddrescue -v -b=64Ki /dev/hda3

Are there other ddrescue options I should use in this case?

[Paul Komski]

The two principle parts of mounting are the device you are mounting and the mount point. The mount point must be a directory that already exists in the system. You can obviously call that folder hda3 - but it must already exist.

You may have a program called ddrescue but this may not be the same application as dd_rescue. I have only ever used dd_rescue for data recovery and so not attempted any compression. You have obviously sussed that the syntax is not the same as for dd. I see no reason the output cant be piped to gzip but it's not something I have done.

Best thing is to practice on a drive where you can do no harm and just make/restore some small images and files.

I'm not sure why you are writing zeros to a file and then deleting it.

[Red Savina]

I missed the underscore and got my information from ss64.com (http://ss64.com/bash/ddrescue.html). It seems that dd_rescue may be the Debian flavor of ddrescue.

I'm not sure why you are writing zeros to a file and then deleting it.To reduce the entropy of the partition (ref: Image Your Hard Drive using dd (http://www.linuxweblog.com/dd-image)).

Reading through the comments in the above reference it appears that depending on the system, any such methods involving pipes can (unpredictably) become problematic due the the pipe overflowing/loosing pressure because of processes running at different speeds. I don't know if there is any way to be alerted to a problem occuring in the pipe (that would certainly save a lot of headaches). It would seem that the only way to be assured that no data has been lost would be to do an independant CRC validation of the source and destination for both (image/restore) operations.

It would be straightforward to direct the output of cksum to two temporary files and make the comparison, but would there be a way to make the comparison without having to create the temporary files?

Obviously some practice and debugging on some easily restored media is necessary before there is ever a real need for a restoration.

I also found a reference to this The Secrets of DD and using it to backup your HDDS (http://www.rob-ward.co.uk/articles/The-_-Secrets-_-of-_-DD-_-and-_-using-_-it-_-to-_-backup-_-your-_-HDDS) buried in the above reference that seems worthwhile.

[Paul Komski]

I had missed the point of the delete.me file up to now - but I do now get the general idea.

When creating the file "delete.me" with a block size of 8M, will this zero-fill areas of contiguous unused space that are less than 8M in length? If not, would these typically be numerous or rare, or depend on the filesystem?

Without experimenting or finding a reference it is hard to say but I guess the file system must make a difference and espcially FAT ones with their file size limitations. File tips would obviously not be included regardless. Hower the blocks shouldn't need to be contiguous in order to create the file - its just that the resultant file would be a very fragmented one. The output is dependent only on how Linux regards the mounted partition and not of how dd operates.

If you knew the amount of "free space" before starting and then the size of the delete.me file you would get an idea of just how much of this free space had been occupied by the delete.me file. Under NTFS the reserved portion of the MFT would not be zeroed but all other available free space (contiguous or not) should be mappable within the MFT and any associated metadata files.

I wonder, in fact, just how much extra compression is achievable by this method and that would depend very much on prior use. File compression can be a double edged sword of course since it is possible for a compressed file to be larger than the uncompressed version! All I can say from average experience is that for this type of work gzip usually creates files about half the size of the existing data on the partition.

If you want to zero out the free space in another way then there are two possibilities that spring to mind. (a) first run a program such as heidi's eraser (http://eraser.heidi.ie/) or (b) first zero another partition of the same or greater size and format it then use Linux to copy the files from the dirty to this clean partition; it should then only contain the partition's metadata plus the copied files. Dirty file tips would still be there under (b) and only a program designed to clean them would clear those up.

[Sylvander]

1. I used ddrescue [NOT dd_rescue] to clone a 750GB [USB2 connected] external HDD [with faulty regions] to a 1GB new 1TB [USB2 connected] external HDD.
I was a total beginner in the use of ddrescue, but it all worked very well.
Delighted with the result. :)
See this record of the job:
Anyone know how to use ddrescue [NOT dd_rescue]? (http://www.murga-linux.com/puppy/viewtopic.php?t=71026)

2. Things that are good about ddrescue:
(a) It auto-makes a log file, so that if the job is interrupted at any point, you just restart the job-> and it automatically takes up where it left off.

(b) [B]IF....[and only if]...
[It detects that] There are faulty regions on the source drive...
It automatically does a whole lot of complicated stuff to deal with that situation.
e.g.
(c) When it discovers faulty regions...
It marks them...
And leaves them alone until later.
And it proceeds to copy all the good regions.
Then...
(d) When you run the [2nd] command to deal with the problem regions...
It deals 1st with the best of the problem regions [least problematic]...
And finally attempts to recover the worst of the problem regions.
You can specify in the command syntax, how many times it tries before giving up.

[Paul Komski]

It was too good to be true to have a thread about Linux without Puppy being introduced into the equation.

The rescue versions of dd are not all the same whether they be called ddrescue or dd_rescue. If you need to find out the specifics about any one of them installed in any distro then one should issue the man command followed by the application name: e.g.
man dd
man ddrescue
man dd_rescue

The basic concept behind all the rescue versions is that if they meet bad blocks they won't spend an inordinate amount of time trying to copy them but will relatively quickly default to replacing the area on the disk or in the file with zeros. The same is true of Windows forensic hex-editing software such as Nlite and WinHex; when bad sectors are met one is normally prompted to skip them or keep trying to read them.

The intense problem with all such software occurs when there are significant numbers of bad sectors on a drive because even with such software one may have to wait for days for even relatively small hard drives to be cloned. When the bad read blocks are small in number they work, as a rule, very well but if there are loads of bad sectors one may have to first try and map where these areas are and to then skip them altogether if possible.

Anyway this thread is predominately about imaging and not data recovery. The nice thing about Knoppix's dd_rescure as opposed to the Linux generic dd application is that one can follow the progress - even if this is stifingly slow at times. It is tempting to use it in place of dd just because of this but it is not a tried and tested method.

Personally I would stay with dd/gzip/split and experiment away until they are completely familiar. For myself I only regularly use dd nowadays for cloning partition and hard drive boot sectors but do all hard drive imaging from a boot floppy or CD running BiNG/IfDOSv1 or BiBM/IfDOSv2. One of the great advantages of these in the context of imaging to optical drives is they can directly create the clones to a bootable set of CDs/DVDs and thus need no other applications when it comes to recovery time. The CD images, as with other image files, can be verified into the bargain.