[Cialug] Is there a better way to sync two directories?

[Adam Shannon]

Josh, the strange thing is that rsync through ssh is working. It's
skipping files that already have been copied over, and isn't using
very much of processor or network.

On Mon, Jan 2, 2012 at 16:42, Josh More <jmore at starmind.org> wrote:
> Whoops.  Didn't read the whole thing.  Sorry about the rsync suggestion.
>
> I'd do what Barry suggested.  Specifically, I would launch a shell to
> each server and run:
>
> find . -type f -exec cksum '{}' \;
>
> on each box.  Then I'd compare the outputs and grep on ">" to identify
> all files that are different and pull only from the first list.  Then
> I'd pipe that through cut to get the file list:
>
> diff 1 2 | grep '>' | cut -d ' ' -f 4-999 > copy.list
>
> Then I'd feed that list into scp through a loop:
>
> for i in `cat copy.list`; do scp $i $user at host:/; done
>
>
> Note, this may break if you have files with spaces in the names.  Test
> stuff first.  I *think* that the problem would come in at the for loop
> stage, so you might want to run tr on copy.list before you parse it
> with cat.
>
> There is probably a more elegant solution where you could do it in
> fewer steps and it's more tolerant of odd file names.
>
> -Josh
>
>
>
>
>
>
> On Mon, Jan 2, 2012 at 4:14 PM, Adam Shannon <adam at ashannon.us> wrote:
>> I have a shell on both boxes, however my 1and1 setup won't allow me to
>> run executables. What I'm doing is sitting in my amazon box with a
>> manual rsync command to pull the 1and1 file list so it can pull down
>> the new images.
>>
>> Thanks Josh for the tip, by the way.
>>
>> On Mon, Jan 2, 2012 at 16:10, Barry Von Ahsen <barry at vonahsen.com> wrote:
>>> do you have shell on either?  could you upload a statically compiled rsync?
>>>
>>>
>>> if not, you could generate a file list on each machine, diff it, and I'm
>>> pretty sure scp/sftp can take a file of files to transfer
>>>
>>>
>>> there's also unison, I think that will work from your local machine to
>>> sync two remote machines
>>>
>>> -barry
>>>
>>>
>>>
>>> On 01/02/2012 02:20 PM, Adam Shannon wrote:
>>>> Hello all,
>>>>
>>>> I'm moving web hosts (from a restricted 1and1 box to an amazon
>>>> instance) and I was wondering if there was a better way to copy all of
>>>> the files over. The biggest part is my image hosting site, which I
>>>> would like to keep the amazon box as close as I can to my 1and1 box.
>>>> However, on my 1and1 box I don't have access to install programs (so
>>>> rsync is out). What I'm doing to start is using scp to transfer
>>>> everything over, however it looks like scp will copy files even if
>>>> they already exist. With 16,000+ images I'd rather not do that (if I
>>>> want to keep the two boxes as similar as possible during the move).
>>>>
>>>> My question is, is there a better way to ensure the two boxes are the
>>>> same even on this limited setup?
>>>>
>>>> Thanks,
>>>>
>>>
>>> _______________________________________________
>>> Cialug mailing list
>>> Cialug at cialug.org
>>> http://cialug.org/mailman/listinfo/cialug
>>
>>
>>
>> --
>> Adam Shannon
>> Web Developer
>> University of Northern Iowa
>> Sophomore -- Computer Science B.S. & Mathematics
>> http://ashannon.us
>> _______________________________________________
>> Cialug mailing list
>> Cialug at cialug.org
>> http://cialug.org/mailman/listinfo/cialug
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug



-- 
Adam Shannon
Web Developer
University of Northern Iowa
Sophomore -- Computer Science B.S. & Mathematics
http://ashannon.us