how to keep 2 PCs partially in sync

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

how to keep 2 PCs partially in sync

Patrick Scribus
Hello,

two of my computers have a similar role as desktop. The installed
packages are nearly the same, the configuration is nearly the same and
the stored data in /home also. Especially the texts, the pictures and
the like require too much time and effort to keep in sync. At first I
wrote a little script that uses the power of rsync. This is much better
than no script at all. But I hope for a solution that automates this
like in those talks from 10-15 years ago when they suggest to use coda.
I would love to use coda but it seems like nobody is maintaining it
since quite some time. What happened in the meantime? What do you guys
use for similar tasks?

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

der.hans
Am 24. Mar, 2020 schwätzte Patrick Scribus so:

moin moin Patrick,

> Hello,
>
> two of my computers have a similar role as desktop. The installed
> packages are nearly the same, the configuration is nearly the same and
> the stored data in /home also. Especially the texts, the pictures and
> the like require too much time and effort to keep in sync. At first I
> wrote a little script that uses the power of rsync. This is much better
> than no script at all. But I hope for a solution that automates this
> like in those talks from 10-15 years ago when they suggest to use coda.
> I would love to use coda but it seems like nobody is maintaining it
> since quite some time. What happened in the meantime? What do you guys
> use for similar tasks?
I use unison rather than rsync when I don't have to worry about hard links.

For keeping /etc in sync, I have written small scripts that export an
etckeeper repo to the replica machine to apply changes.

dpkg --get-selections and dpkg --set-selections can be use to keep package
parity.

If you want something like coda I used to use MooseFS, which apparently is
now LizardFS in the repo.

Lots of people are using ceph.

ciao,

der.hans
--
#  https://www.LuftHans.com   https://www.PhxLinux.org
#  Linux Fest Northwest cancelled, working to get presentations online
#  https://www.lfnw.org/conferences/2020
#  "The purpose of IT is to seamlessly and transparently provide the other
#  9/10's of the iceberg for people who need to work with chunks
#  of floating ice." -- Strata Rose Chalup
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Brian Sammon-5
In reply to this post by Patrick Scribus
I've been using Unison (https://packages.debian.org/buster/unison-gtk -- there's also a commandline package) for quite a few years (a decade?) now.  It works well.

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Brad Rogers
In reply to this post by Patrick Scribus
On Tue, 24 Mar 2020 22:05:40 +0100
Patrick Scribus <[hidden email]> wrote:

Hello Patrick,

>But I hope for a solution that automates this

Would setting up a cron job to run your script do?

--
 Regards  _
         / )           "The blindingly obvious is
        / _)rad        never immediately apparent"
It's only the children of the f****** wealthy tend to be good looking
Ugly - The Stranglers

attachment0 (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Andrei POPESCU-2
In reply to this post by Patrick Scribus
On Ma, 24 mar 20, 22:05:40, Patrick Scribus wrote:

> Hello,
>
> two of my computers have a similar role as desktop. The installed
> packages are nearly the same, the configuration is nearly the same and
> the stored data in /home also. Especially the texts, the pictures and
> the like require too much time and effort to keep in sync. At first I
> wrote a little script that uses the power of rsync. This is much better
> than no script at all. But I hope for a solution that automates this
> like in those talks from 10-15 years ago when they suggest to use coda.
> I would love to use coda but it seems like nobody is maintaining it
> since quite some time. What happened in the meantime? What do you guys
> use for similar tasks?
 
Syncthing.

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

deloptes-2
In reply to this post by Patrick Scribus
Patrick Scribus wrote:

> What do you guys
> use for similar tasks?

I guess from share to a cloud.

I spent a lot of time in phone sync via bluetooth (calendar, contacts, todos
and notes).

The PC does not have any data on it - there is a share. From outside - VPN
to the share.

Keep your data at a central place. If you must edit documents from different
devices - perhaps a kind of cloud solution is preferable.


Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

David Christensen
In reply to this post by Patrick Scribus
On 2020-03-24 14:05, Patrick Scribus wrote:

> Hello,
>
> two of my computers have a similar role as desktop. The installed
> packages are nearly the same, the configuration is nearly the same and
> the stored data in /home also. Especially the texts, the pictures and
> the like require too much time and effort to keep in sync. At first I
> wrote a little script that uses the power of rsync. This is much better
> than no script at all. But I hope for a solution that automates this
> like in those talks from 10-15 years ago when they suggest to use coda.
> I would love to use coda but it seems like nobody is maintaining it
> since quite some time. What happened in the meantime? What do you guys
> use for similar tasks?

My mail is on one laptop (Thunderbird).


I use Firefox and Firefox Sync wherever available.


I have an SSH/CVS server (FreeBSD jail) for working files, software
development, system configuration files, and system administration notes.


I have a Samba server (FreeBSD jail) for bulk data -- downloads, music,
pictures, videos, etc..


David

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Mark Allums-4
In reply to this post by Patrick Scribus
On 3/24/20 4:05 PM, Patrick Scribus wrote:

> Hello,
>
> two of my computers have a similar role as desktop. The installed
> packages are nearly the same, the configuration is nearly the same and
> the stored data in /home also. Especially the texts, the pictures and
> the like require too much time and effort to keep in sync. At first I
> wrote a little script that uses the power of rsync. This is much better
> than no script at all. But I hope for a solution that automates this
> like in those talks from 10-15 years ago when they suggest to use coda.
> I would love to use coda but it seems like nobody is maintaining it
> since quite some time. What happened in the meantime? What do you guys
> use for similar tasks?
>

Syncthing.

https://syncthing.net/

Mark

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Peter Ehlert-2
In reply to this post by Andrei POPESCU-2

On 3/24/20 3:14 PM, Andrei POPESCU wrote:

> On Ma, 24 mar 20, 22:05:40, Patrick Scribus wrote:
>> Hello,
>>
>> two of my computers have a similar role as desktop. The installed
>> packages are nearly the same, the configuration is nearly the same and
>> the stored data in /home also. Especially the texts, the pictures and
>> the like require too much time and effort to keep in sync. At first I
>> wrote a little script that uses the power of rsync. This is much better
>> than no script at all. But I hope for a solution that automates this
>> like in those talks from 10-15 years ago when they suggest to use coda.
>> I would love to use coda but it seems like nobody is maintaining it
>> since quite some time. What happened in the meantime? What do you guys
>> use for similar tasks?
>  
> Syncthing.

+1 on syncthing

the downside is if you OOPS on one machine there is no retun.

I use syncthing together with LuckyBackup ... I only work with Local
Copies, and after vetting then back to the common synced folder.

walk to the other room, copy down the "vetted versions"
*my .thunderbird folder is one of many

>
> Kind regards,
> Andrei

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Charles Curley
In reply to this post by Mark Allums-4
On Tue, 24 Mar 2020 18:26:31 -0500
Mark Allums <[hidden email]> wrote:

> Syncthing.
>
> https://syncthing.net/

Concur on Syncthing. Since it is near-instantaneous, so are oopses. For
that I use rsnapshot.

Some other thoughts on backups.
http://charlescurley.com/blog/posts/2019/Nov/02/backups-on-linux/

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Anastasios Lisgaras
On 3/25/20 7:31 AM, Charles Curley wrote:
> On Tue, 24 Mar 2020 18:26:31 -0500
> Mark Allums <[hidden email]> wrote:
>
>> Syncthing.
>>
>> https://syncthing.net/

For the history:
Resilio (formerly BitTorrent Sync) was a very good *proprietary*
software tool.

However, after a while, it was developed by an awesome team the powerful
*open source* (Mozilla Public License 2.0) *Syncthing* (Pulse) and
really surpassed the proprietary software BitTorrent Sync - to the point
where the BitTorrent Sync started copying features from Syncthing!

So for the above, I'm voting +1 on Syncthing!
It must be an amazing tool!


On 3/25/20 1:40 AM, Peter Ehlert wrote:> +1 on syncthing
>
> the downside is if you OOPS on one machine there is no retun.
>
> I use syncthing together with LuckyBackup ... I only work with Local
> Copies, and after vetting then back to the common synced folder.
>
> walk to the other room, copy down the "vetted versions"
> *my .thunderbird folder is one of many


On 3/25/20 7:31 AM, Charles Curley wrote:
> Concur on Syncthing. Since it is near-instantaneous, so are oopses. For
> that I use rsnapshot.
>
> Some other thoughts on backups.
> http://charlescurley.com/blog/posts/2019/Nov/02/backups-on-linux/


Excuse me, but I don't understand exactly what you mean.
Could you explain a little more in detail?


Thank you,
Tasos
--
Kind regards,

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Andrei POPESCU-2
On Mi, 25 mar 20, 09:55:25, Anastasios Lisgaras wrote:

>
> On 3/25/20 7:31 AM, Charles Curley wrote:
> > Concur on Syncthing. Since it is near-instantaneous, so are oopses. For
> > that I use rsnapshot.
> >
> > Some other thoughts on backups.
> > http://charlescurley.com/blog/posts/2019/Nov/02/backups-on-linux/
>
> Excuse me, but I don't understand exactly what you mean.
> Could you explain a little more in detail?
Tools for keeping data in sync are often mistaken for a backup solution,
which they are not.

If a file is corrupted, deleted, etc. in one place that will be
propagated to all copies.

Depending on the features provided by the synchronisation tool they
could be *a part* of a backup solution.

See http://taobackup.com for what a complete backup solution should
provide.

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Joe Rowan
In reply to this post by deloptes-2
On Tue, 24 Mar 2020 23:50:00 +0100
deloptes <[hidden email]> wrote:

> Patrick Scribus wrote:
>
> > What do you guys
> > use for similar tasks?  
>
> I guess from share to a cloud.
>
> I spent a lot of time in phone sync via bluetooth (calendar,
> contacts, todos and notes).
>
> The PC does not have any data on it - there is a share. From outside
> - VPN to the share.
>
> Keep your data at a central place. If you must edit documents from
> different devices - perhaps a kind of cloud solution is preferable.
>
>

Until your phone line goes down for a week, as mine did a few years
ago. I had a mobile dongle for simple things, but it was unreliable and
slow, and completely unusable for any kind of backup of more than a MB
or so, or for actually working through.

Or you find yourself occasionally working (as I do now) in a 'managed'
office, which provides a (slow again) access to a 192.168.x.0/24 with
a single outside world connection shared with the rest of the building.
My client there does not push enough through the net to be worth paying
for his own separate connection. And no, I don't have these problems
often enough for it to be worth me paying for a fast mobile connection.

And I like having custody of my own data. And I use Unison for my own
sync work mostly, with FreeFileSync on the Windows partitions of my
mobiles.

--
Joe

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Dan Purgert
On Mar 25, 2020, Joe wrote:

> On Tue, 24 Mar 2020 23:50:00 +0100
> deloptes <[hidden email]> wrote:
>
> > Patrick Scribus wrote:
> >
> > > What do you guys
> > > use for similar tasks?  
> >
> > I guess from share to a cloud.
> >
> > I spent a lot of time in phone sync via bluetooth (calendar,
> > contacts, todos and notes).
> >
> > The PC does not have any data on it - there is a share. From outside
> > - VPN to the share.
> >
> > Keep your data at a central place. If you must edit documents from
> > different devices - perhaps a kind of cloud solution is preferable.
> >
> >
>
> Until your phone line goes down for a week, as mine did a few years
> ago. I had a mobile dongle for simple things, but it was unreliable and
> slow, and completely unusable for any kind of backup of more than a MB
> or so, or for actually working through.
I don't think he meant to imply using external-to-you "cloud" providers
(gdrive, dropbox), but rather creating his own personal "cloud".

Be it something pretty -- Nextcloud, for example -- or something
utilitarian (a central NFS or SSHFS server holding all the data).
>
> Or you find yourself occasionally working (as I do now) in a 'managed'
> office, which provides a (slow again) access to a 192.168.x.0/24 with
> a single outside world connection shared with the rest of the building.
> My client there does not push enough through the net to be worth paying
> for his own separate connection. And no, I don't have these problems
> often enough for it to be worth me paying for a fast mobile connection.

I've used Nextcloud in these situations, it's actually pretty good with
slower connections.  Since everything is a (machine-)local copy, in
addition to being stored centrally; something I work on "here(tm)" gets
updated "everywhere" shortly after I've saved the document.

I've only really ever run into problems with it when there was a
godawful slow connection with a machine that'd been offline for 2 weeks
while I was on vacation (and I forgot to spin it up at home before
heading out)

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

deloptes-2
Dan Purgert wrote:

> I don't think he meant to imply using external-to-you "cloud" providers
> (gdrive, dropbox), but rather creating his own personal "cloud".

exactly - thank you

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Charles Curley
In reply to this post by Andrei POPESCU-2
On Wed, 25 Mar 2020 11:07:36 +0200
Andrei POPESCU <[hidden email]> wrote:

> Tools for keeping data in sync are often mistaken for a backup
> solution, which they are not.
>
> If a file is corrupted, deleted, etc. in one place that will be
> propagated to all copies.
>
> Depending on the features provided by the synchronisation tool they
> could be *a part* of a backup solution.

That's pretty much it. I see backup solutions as a spectrum of tools,
with different tools suitable for different problems.

>
> See http://taobackup.com for what a complete backup solution should
> provide.

Excellent.

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

attachment0 (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Linux-Fan
In reply to this post by Patrick Scribus
Patrick Scribus writes:

> Hello,
>
> two of my computers have a similar role as desktop. The installed
> packages are nearly the same, the configuration is nearly the same and
> the stored data in /home also. Especially the texts, the pictures and
> the like require too much time and effort to keep in sync. At first I
> wrote a little script that uses the power of rsync. This is much better
> than no script at all. But I hope for a solution that automates this
> like in those talks from 10-15 years ago when they suggest to use coda.
> I would love to use coda but it seems like nobody is maintaining it
> since quite some time. What happened in the meantime? What do you guys
> use for similar tasks?
Hello,

there have been multiple answers already, so forgive me if my post does not
seem to add anything valuable. Still, it bugs me that there are many
different solutions proposed without their advantages and disadvantages
given?

I think that it is an important factor how the systems "online status" (in
sense of power and networking) is to be considered? Are both systems online
simultaneously? Are both systems online at the same time only for
synchonization?

I can think of different solutions depending on what is actually
wanted/neede:

 * Cluster File Systems.
   People have already mentioned ceph (which is more an object storage
   and thus slow on small files IIRC?). I can add OCFS (Oracle Cluster
   File System) to the list, although it is not so easy to set up.
   Cluster file systems make sense if both systems are online at the same
   time and should both access a common file system. Often, cluster
   file systems want a "third" machine for doing the actual storage work
   (e.g. an iSCISI target of OCFS). I have also tried out GFS2 in the past,
   but it is a PITA to set up!

 * Synchonization Tools.
   There are tools to invoke explicitly to call the synchronization.
   These make sense if both systems are online at the same time only
   for synchronization... if not, one will need to deal with "both changed"
   conflicts on a manual basis. I have no experience with syncthing
   (mentioned in the thread) -- syncthing might have a solution for this...

 * Network File Systems.
   If you have a constellation of: system1 and system2 where
   system2 online means system1 is online, too, then you might
   install a "file server" (NFS or similar) on system1 and share
   files through this mechanism. From all approaches proposed, I
   would recommend this as being the least complex in operation
   although this does not mean it is the least complex to setup.

 * "Cloud"-like file synchronization.
   These usually require a "third" server, too. And in my experience,
   whenever one is using "synchronized" files for non-trivial data
   processing (e.g. creating and reading a lot of files, storing a
   database, accessing the data with many processes...) most of these
   systems will fail one way or another (up to causing data loss). Yet,
   most people using such systems do not seem to have these issues :)
   These tools are useful in scenarios where there is no guarantee for
   any machine being online the same time as the other although this is
   achieved at the cost of running a "third" machine 24/7...

HTH and YMMV
Linux-Fan

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Anastasios Lisgaras
In reply to this post by Andrei POPESCU-2
On 3/25/20 11:07 AM, Andrei POPESCU wrote:

> If a file is corrupted, deleted, etc. in one place that will be
> propagated to all copies.
>
> Depending on the features provided by the synchronisation tool they
> could be *a part* of a backup solution.
>
> See http://taobackup.com for what a complete backup solution should
> provide.
>
> Kind regards,
> Andrei
>

Sorry, but again, I don't see the problem.
If I choose to delete a file from a machine - I will want the same for
the rest of the machines. What's the problem with that?


On 3/25/20 12:49 PM, Dan Purgert wrote:

> I don't think he meant to imply using external-to-you "cloud" providers
> (gdrive, dropbox), but rather creating his own personal "cloud".
>
> Be it something pretty -- Nextcloud, for example -- or something
> utilitarian (a central NFS or SSHFS server holding all the data).
>
> I've used Nextcloud in these situations, it's actually pretty good with
> slower connections.  Since everything is a (machine-)local copy, in
> addition to being stored centrally; something I work on "here(tm)" gets
> updated "everywhere" shortly after I've saved the document.
>
> I've only really ever run into problems with it when there was a
> godawful slow connection with a machine that'd been offline for 2 weeks
> while I was on vacation (and I forgot to spin it up at home before
> heading out)
>



I have also installed in the past the *Νextcloud* locally ( in a
Raspberry Pi ) and it really worked amazingly! In fact, it worked so
perfectly that, because of that I didn't deal with *Syncthing*.
In fact, I could not understand their differences.
Syncthing seems more restrictive than Nextcloud because it does not have
the "cloud" (WEB UI) function offered by Nextcloud.
However, I would really like to try Syncthing to see what it really
offers and to be able to compare it with Nextcloud.

What is your opinion?



On 3/25/20 7:44 PM, Linux-Fan wrote:

> Hello,
>
> there have been multiple answers already, so forgive me if my post does not
> seem to add anything valuable. Still, it bugs me that there are many
> different solutions proposed without their advantages and disadvantages
> given?
>
> I think that it is an important factor how the systems "online status" (in
> sense of power and networking) is to be considered? Are both systems online
> simultaneously? Are both systems online at the same time only for
> synchonization?
>
> I can think of different solutions depending on what is actually
> wanted/neede:
>
> * Cluster File Systems.
>    People have already mentioned ceph (which is more an object storage
>    and thus slow on small files IIRC?). I can add OCFS (Oracle Cluster
>    File System) to the list, although it is not so easy to set up.
>    Cluster file systems make sense if both systems are online at the same
>    time and should both access a common file system. Often, cluster
>    file systems want a "third" machine for doing the actual storage work
>    (e.g. an iSCISI target of OCFS). I have also tried out GFS2 in the past,
>    but it is a PITA to set up!
>
> * Synchonization Tools.
>    There are tools to invoke explicitly to call the synchronization.
>    These make sense if both systems are online at the same time only
>    for synchronization... if not, one will need to deal with "both changed"
>    conflicts on a manual basis. I have no experience with syncthing
>    (mentioned in the thread) -- syncthing might have a solution for this...
>
> * Network File Systems.
>    If you have a constellation of: system1 and system2 where
>    system2 online means system1 is online, too, then you might
>    install a "file server" (NFS or similar) on system1 and share
>    files through this mechanism. From all approaches proposed, I
>    would recommend this as being the least complex in operation
>    although this does not mean it is the least complex to setup.
>
> * "Cloud"-like file synchronization.
>    These usually require a "third" server, too. And in my experience,
>    whenever one is using "synchronized" files for non-trivial data
>    processing (e.g. creating and reading a lot of files, storing a
>    database, accessing the data with many processes...) most of these
>    systems will fail one way or another (up to causing data loss). Yet,
>    most people using such systems do not seem to have these issues :)
>    These tools are useful in scenarios where there is no guarantee for
>    any machine being online the same time as the other although this is
>    achieved at the cost of running a "third" machine 24/7...
>
> HTH and YMMV
> Linux-Fan


I really liked your *detailed* post! It really is a detailed and
comprehensive answer.

In my use case, I have both a safe and a secure remote server machine
(VM) and a raspberry Pi at my home to run continuously & incessantly 24/7.
I have three computers. One at work a laptop and a desktop computer.
What I want is for the three computers to have specific directories and
files shared/synchronized.
( Probably the whole /home directory of my user. )
I also want to be able to sometimes share/send my files - securely - to
third parties.
What do you think is the best approach for me?
Maybe Syncthing + Nextcloud ?


Thank you.
Tasos

Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

Curt
On 2020-04-04, Anastasios Lisgaras <[hidden email]> wrote:

> On 3/25/20 11:07 AM, Andrei POPESCU wrote:
>> If a file is corrupted, deleted, etc. in one place that will be
>> propagated to all copies.
>>
>> Depending on the features provided by the synchronisation tool they
>> could be *a part* of a backup solution.
>>
>> See http://taobackup.com for what a complete backup solution should
>> provide.
>>
>> Kind regards,
>> Andrei
>>
>
> Sorry, but again, I don't see the problem.
> If I choose to delete a file from a machine - I will want the same for
> the rest of the machines. What's the problem with that?
>

The problem arises when you "choose" to delete the wrong file by
inadvertence; if you're only thinking "syncing," your sunk, but if
you've backed up as well (and you ain't pell-mell), you're still looking
swell.


Reply | Threaded
Open this post in threaded view
|

Re: how to keep 2 PCs partially in sync

tomas@tuxteam.de
On Sat, Apr 04, 2020 at 01:12:33PM -0000, Curt wrote:

[...]

> The problem arises when you "choose" to delete the wrong file by
> inadvertence; if you're only thinking "syncing," your sunk, but if
> you've backed up as well (and you ain't pell-mell), you're still looking
> swell.

To add one data point for that: the most cases (by far) I've needed a
backup is when I have deleted stuff by mistake. Close second is some
buggy software having deleted or mangled files I've needed. Last [1]
come actual physical damage to storage media.

Over-eager synchronization of backup won't help in the two first cases.

Cheers
[1] In between would perhaps be malware mangling the data: my platform
   has never done that to me, but on other platforms it seems to
   happen regularly.

-- tomás

signature.asc (205 bytes) Download Attachment
12