[Cialug] Efficiently removing the beginning of a file

[Daniel A. Ramaley]

On Monday 21 May 2007 11:44, Nathan Stien wrote:
>On 5/21/07, Daniel A. Ramaley <daniel.ramaley at drake.edu> wrote:
>With a small block size it's nearly always schedulable and ends
>up making a lot of unnecessary loops.

Yes, reading a 70 MB file one byte at a time is certainly not efficient; 
dd spends all its time asking the kernel for the next byte (good thing 
software doesn't have personality or else the kernel would become quite 
impatient and frustrated). That's why i set ibs and obs separately so 
at least it wouldn't have to also write the new file one byte at a 
time.

I guess i was just curious if someone knows an entirely better method of 
accomplishing the same thing, perhaps a method that does not involve dd 
since it does not seem to be the best tool for this particular job. Off 
the top of my head i couldn't think of anything. Sure, i could whip up 
a short Perl script that would do it, but it seems to me like the sort 
of problem that traditional command-line utilities ought to be able to 
handle.

I suppose i could use split to break the file into chunks of 3635 bytes, 
remove the first one, and concatenate the remaining chunks back 
together. But on my 70 MB test file that would mean over 20 000 small 
files. And if i had only needed to remove, say, the first 2 bytes, we'd 
be talking about 35 *million* small files. Most file systems show 
somewhat disappointing performance if you try to create that many files 
in one directory.

------------------------------------------------------------------------
Dan Ramaley                            Dial Center 118, Drake University
Network Programmer/Analyst             2407 Carpenter Ave
+1 515 271-4540                        Des Moines IA 50311 USA