[Cialug] Efficiently removing the beginning of a file

[Nathan Stien]

On 5/21/07, Daniel A. Ramaley <daniel.ramaley at drake.edu> wrote:
> I guess i was just curious if someone knows an entirely better method of
> accomplishing the same thing, perhaps a method that does not involve dd
> since it does not seem to be the best tool for this particular job. Off
> the top of my head i couldn't think of anything. Sure, i could whip up
> a short Perl script that would do it, but it seems to me like the sort
> of problem that traditional command-line utilities ought to be able to
> handle.

I don't see how a Perl script (or anything else) would do this
significantly differently from dd.  At the end of the day, you still
need to copy most of the blocks in the file to another file, and this
will still cost you at best O(n) time.  And dd can be slightly faster
in practice because it's a tight little C program and doesn't require
loading a Perl interpreter.  But like I said, this should be an I/O
bound process, so the slowness of Perl shouldn't matter much as your
drive's throughput and seek time.

Of course, the exception is that if you have a clever underlying
filesystem that lets you do copy-on-write stuff you could do this in
nearly zero time.  The same underlying storage could be shared between
each file until a given block changes, and then only that block really
need be copied.  This is the mechanism used by versioning filesystems
like ext3cow. (http://www.ext3cow.com/).


-- 
Nathan P. Stien
Consulting Engineer / Software Developer
Embedded Systems Electronics and Software
http://linkedin.com/in/nathanstien
Mobile: 309.241.2581