[Cialug] Is there a better way to sync two directories?

[Adam Shannon]

Josh, I should clarify that I'm ssh'ing _into_ my 1and1 box. So, it
looks like rsync is handling the diff'ing for me. The command is as
simple as this:

rsync -azvv --ignore-existing
u49157759 at decafproductions.com:~/IMGMI/SITE/* /mnt/imgmi.net/

On Mon, Jan 2, 2012 at 16:49, Adam Shannon <adam at ashannon.us> wrote:
> Josh, the strange thing is that rsync through ssh is working. It's
> skipping files that already have been copied over, and isn't using
> very much of processor or network.
>
> On Mon, Jan 2, 2012 at 16:42, Josh More <jmore at starmind.org> wrote:
>> Whoops.  Didn't read the whole thing.  Sorry about the rsync suggestion.
>>
>> I'd do what Barry suggested.  Specifically, I would launch a shell to
>> each server and run:
>>
>> find . -type f -exec cksum '{}' \;
>>
>> on each box.  Then I'd compare the outputs and grep on ">" to identify
>> all files that are different and pull only from the first list.  Then
>> I'd pipe that through cut to get the file list:
>>
>> diff 1 2 | grep '>' | cut -d ' ' -f 4-999 > copy.list
>>
>> Then I'd feed that list into scp through a loop:
>>
>> for i in `cat copy.list`; do scp $i $user at host:/; done
>>
>>
>> Note, this may break if you have files with spaces in the names.  Test
>> stuff first.  I *think* that the problem would come in at the for loop
>> stage, so you might want to run tr on copy.list before you parse it
>> with cat.
>>
>> There is probably a more elegant solution where you could do it in
>> fewer steps and it's more tolerant of odd file names.
>>
>> -Josh
>>
>>
>>
>>
>>
>>
>> On Mon, Jan 2, 2012 at 4:14 PM, Adam Shannon <adam at ashannon.us> wrote:
>>> I have a shell on both boxes, however my 1and1 setup won't allow me to
>>> run executables. What I'm doing is sitting in my amazon box with a
>>> manual rsync command to pull the 1and1 file list so it can pull down
>>> the new images.
>>>
>>> Thanks Josh for the tip, by the way.
>>>
>>> On Mon, Jan 2, 2012 at 16:10, Barry Von Ahsen <barry at vonahsen.com> wrote:
>>>> do you have shell on either?  could you upload a statically compiled rsync?
>>>>
>>>>
>>>> if not, you could generate a file list on each machine, diff it, and I'm
>>>> pretty sure scp/sftp can take a file of files to transfer
>>>>
>>>>
>>>> there's also unison, I think that will work from your local machine to
>>>> sync two remote machines
>>>>
>>>> -barry
>>>>
>>>>
>>>>
>>>> On 01/02/2012 02:20 PM, Adam Shannon wrote:
>>>>> Hello all,
>>>>>
>>>>> I'm moving web hosts (from a restricted 1and1 box to an amazon
>>>>> instance) and I was wondering if there was a better way to copy all of
>>>>> the files over. The biggest part is my image hosting site, which I
>>>>> would like to keep the amazon box as close as I can to my 1and1 box.
>>>>> However, on my 1and1 box I don't have access to install programs (so
>>>>> rsync is out). What I'm doing to start is using scp to transfer
>>>>> everything over, however it looks like scp will copy files even if
>>>>> they already exist. With 16,000+ images I'd rather not do that (if I
>>>>> want to keep the two boxes as similar as possible during the move).
>>>>>
>>>>> My question is, is there a better way to ensure the two boxes are the
>>>>> same even on this limited setup?
>>>>>
>>>>> Thanks,
>>>>>
>>>>
>>>> _______________________________________________
>>>> Cialug mailing list
>>>> Cialug at cialug.org
>>>> http://cialug.org/mailman/listinfo/cialug
>>>
>>>
>>>
>>> --
>>> Adam Shannon
>>> Web Developer
>>> University of Northern Iowa
>>> Sophomore -- Computer Science B.S. & Mathematics
>>> http://ashannon.us
>>> _______________________________________________
>>> Cialug mailing list
>>> Cialug at cialug.org
>>> http://cialug.org/mailman/listinfo/cialug
>> _______________________________________________
>> Cialug mailing list
>> Cialug at cialug.org
>> http://cialug.org/mailman/listinfo/cialug
>
>
>
> --
> Adam Shannon
> Web Developer
> University of Northern Iowa
> Sophomore -- Computer Science B.S. & Mathematics
> http://ashannon.us



-- 
Adam Shannon
Web Developer
University of Northern Iowa
Sophomore -- Computer Science B.S. & Mathematics
http://ashannon.us