View on GitHub

Mateix

File Synchronisation Tool :memo:

mateix

A simple file synchronisation tool.

Mateix means ‘same’ in the Catalan language, which is a language of Catalonia community of Spain.

Index

Problem Statement
Challenges
Final Solution
Usage
- Installation
- Commands
  - Mateix
  - MateixWatch
Development
Resources
License
Example
Original Problem

Problem Statement

We need to create a solution to synchronize two folders in two different devices, each folder have a file called data, which needs to be synchronized and it should be bi-directional. To finally decide upon our solution, we need to make sure the following characteristics are met.

Reliability: Solution should be reliable, and should take care of all the possible cases it can go wrong.
Automatic: Everything should happen in the background with minimum human intervene.
Cheap: It should do everything in minimum time while using very few resources, with least latency.

Find the original problem statement at the end of the file.

Challenges

Following are the challenges we need to tackle, mentioned with their possible solutions.

Challenge: How should we communicate between two computers?

When communicating over the internet, following possible scenarios:

No	IP-1	IP-2	Details
1	Static	Static	`Server to Server` Example: Backup servers connected to production servers.
2	Dynamic	Static	`Server to remote devices` Example: Dropbox, Google drive connected to remote devices.
3	Dynamic	Dynamic	`Two remote devices` Example: Two remote devices connected.

For our solution, we will only cover the 1st case. The same solution can be applied for other cases, which little changes. We have multiple methods to communicate over the internet, some are mentioned below.

No	Method	Secure	Reliable	Speed	Automatic
1	TCP (Transmission Control Protocol)	Medium	High	Fast	Yes
2	SSH (Secure Shell)	High	High	Medium	Not by default
3	UDP (User Datagram Protocol)	Less	Less	Fastest	Yes

Result

After comparing all the methods, I have decided to use TCP.

Challenge: When files should be synchronized?

Following are the options on when files should be synchronized:

Immediately inotify-tools
After a time gap crontab

Result

After comparing all the options, I have decided to synchronize immediately a change is detected. It will make it real time, will prevent merge conflict.

Note: if we update the device 1, updates will be sent to device 2, that will trigger the script, which will try to update the device 1, and this might go in a loop. We need to handle the butterfly effect.

Challenge: How should we measure the difference?

To measure the difference, we can use any of method mentioned below.

No	Method	Reliable	Speed
1	Time Modification (Metadata)	`Less Reliable` software can and does manipulate the modification time. Also, the user might change system time and confuse the sync program.	`Fast` O(1)
2	Checksum (Hash the file)	`More Reliable` It’s an (almost) certain way measure difference, hash collisions do happen, but It is rare.	`Slow` O(n)

Result

After comparing all the methods, I have decided to use Checksum.

Challenge: How should we tackle the differences?

Following are the 3 cases that we need to handle:

The file exists on device 1, not on device 2
The file exists on both devices and is identical
The file exists on both devices and is different

Following is the action table:

File 1	File 2	Action
Deleted	No Deleted	Delete
Deleted	Deleted	Nothing
No Change	No change	Nothing
Modification	No change	Use A
Modification	Modification	Merge

Note: vice-versa is also true in this action table.

Following is the action table based on the time:

Time x	Time x+1	Action
Does not exist	Exist	Created
Existed	Does not exist	Deleted
Exist	Modification	Modification

Result

For this solution, we are just tracking one file data, and we are doing synchronization immediately, so most of the cases won’t apply to us.

Challenge: How will you handle a merge conflict?

Following are the methods we have to prevent merge conflict:

No	Method	Details	Merge Quality	Automatic
1	Ask the user	Ask the user how to merge them or which one to pick.	Best	No
2	Lock other user files	Lock a file if it is owned by the other user.	No-Merge	Yes
3	overwrite with latest changes	We can overwrite the file with the latest changes.	Medium	Yes

Note: Resolving to merge conflict is technically impossible without a human to intervene

Result

Again, in our case, this won’t be a constant problem, we have just one file. So I have decided to use overwrite with latest changes method.

Final Solution

STEP 1: Required tools are installed.
        - inotify-tools

STEP 2: Binary and service files are downloaded, and service is enabled.
        - /usr/bin/mateix
        - /usr/bin/mateixWatch
        - /etc/systemd/system/mateix-watch.service
        - daemon-reload
        - enable mateix-watch.service

STEP 3: Create the dotfiles and config files.
        - /etc/.mateix/
        - /etc/.mateix/syncList
        - /etc/.mateix/log  

STEP 4: Creating a mateix watched folder.
        - Add the path to /etc/.mateix/syncList
        - Create PATH/.mateix/config.json

STRUCTURE OF `config.json`:

{
  targetIP: // Ip of other device
  targetDir: // Path of sync folder in other device
}


WORKING:
`inotifywait` is a tool part of `inotify-tools`, which is used
to catch changes in a folder or file.

`mateix-watch.service` is started when the system boot, which
activates the `mateixWatch` script.

`mateixWatch` to catch any changes using `inotifywait` in the files
and folders which are listed in `/etc/.mateix/syncList`.

Once changes are detected, `mateixWatch` send the $PATH to `mateix`
to do the synchronisation and log it to `/etc/.mateix/log`.

`mateix` is the binary tool which reads the `$PATH/.mateix/config.json`
file to fetch `targetIP`, then it will connect to the server using the
`targetIP`, and fetch the checksum for the `targetDir`, if the checksum
doesn't match, it will send the changes to the server, and the server will
update the local `data` file. Checksum also help in solving the butterfly
effect.

Usage

Using Mateix is very simple. All you have to do is install this CLI tool.

Right now, the install script is tested only on a Debian-Based Distribution, but it can be easily configured for the other distros aswell.

Installation

To install Mateix, open your terminal, and type the commands given below.

Download the Install script. $ wget https://raw.githubusercontent.com/ramantehlan/mateix/master/install
Make the script executables. $ chmod +x ./install
Execute the install script. $ ./install

Commands

Once Mateix is installed, now you can use it to sync folders. Following are the commands available to help you sync folders.

Mateix

Mateix is the main program which will get the job done, it will communicate with the other systems, and synchronize files.

Command (Prefix: `mateix`)	Working
init	To initialize a mateix watched folder.
update	To update a mateix watched folder. `--file` to provide the path which needs to be updated.
server	`--start` to start the server. `--stop` to stop the server. Server use port `1248`
uninstall	To uninstall the mateix from the system

Note: In any case, you must not rename your mateix watched folder. Since it’s location is added to /etc/.mateix/syncList, on rename, it will misbehave.

mateixWatch

mateixWatch is the script which catches the changes in files, and calls mateix command to take care of it. mateixWatch is automatically executed when the system starts with mateix-watch.service. It supports only the following commands.

Command (Prefix: `mateixWatch`)	Working
start	To start the watch program on the files listed in `/etc/.mateix/syncList`. It uses `inotifywait` to catch changes. Note: I do not suggest using this command to start the watch program. Instead, you should start the `mateix-watch.service` service. If you still wish to use it, make sure you are a root user.
stop	To stop the watch program, by killing all the `inotifywait` processes.

Sample

Let's say you wish to synchronize `data` file present in `SYSTEM-1` and
`SYSTEM-2`, at `192.168.43.99:/home/atom/mateixWatch` and
`192.168.43.189/home/mspp/mateixWatch`.

[SYSTEM 1]:
  IP: 192.168.43.99
  DIR: /home/atom/mateixWatch

[SYSTEM 2]:
  IP: 192.168.43.189
  DIR: /home/mspp/mateixWatch

Step 1: Create the desired folders in both the system, and add a `data` file
        in both of them.

Step 2: Install the mateix tool in both the system, with instructions given
        above.

Step 3: Go to folder, run `$ sudo mateix init` command, and follow the
        instructions.

Once all the above steps will be complete, you will see that all your changes
are reflected in both the files, in both directions.

So, changes made in `data` file of System 1, will be reflected in `data` file
of System 2, and vice-versa

Development

If you are interested in developing this project, feel free to read more about it below.

Pre-Requisites

If you are interested in the development, then here are some pre-requisites you need to have.

Familiarity of Go language.
Knowledge of Git
Unix/Linux Terminal

Setup

To set up the development environment in your system:

Install Go in your system.
Fork this repo, and clone it in your workplace.

File Structure

.
├── init.go
├── install
├── LICENSE
├── main.go
├── mateix
├── packages
│   ├── command
│   │   └── command.go
│   └── e
│       └── e.go
├── README.md
├── server.go
├── service
│   ├── mateixWatch
│   └── mateix-watch.service
├── uninstall.go
└── update.go

No	File Name	Purpose
1	`README.md`	Current file you are reading
2	`LICENSE`	GNU GPL V3.0 License
3	`install`	Install script
4	`mateix`	Complied binary to run the tool
5	`mateixWatch`	Script to catch changes in folders
6	`mateix-watch.service`	Service to call `mateixWatch` on boot
7	`main.go`	Main/initial file of the program
8	`init.go`	Initialize the mateixWatch in a repository
9	`update.go`	Sync a folder with targetIP
10	`uninstall.go`	Uninstall the mateix tool from system
11	`server.go`	Server to get changes from the client
12	`command.go`	Package to help in the execution of Unix commands
13	`e.go`	Package to handle errors

Resources

License

GNU GPL V3.0

Examples

Install mateix

Create a mateix watched file

System 1

System 2

Original Problem

Clone Wars 2.0
--------------

Prime Minister Lama Su,

I hope this letter finds you in the best of health.

The last batch of clones you built for us was faulty
and did not perform as expected (https://www.youtube.com/watch?v=b0DuUnhGBK4)

We unearthed some secrets about how the droid army was trained and hope that
you can use this information to make a better army this time around. With the
galaxy on the brink of another war, I cannot help but emphasize how much a
large discount will help the Republic in its efforts.

One of our allies came across these schematics in an abandoned base that shed some
light on the droid training exercises, master Yoda concluded that a pair of droids
undergo various kinds of battle simulations during which each droid records its
progress and learning in a force, currently unfamiliar to us, called "Data".
This force from both droids is then combined in a ritual called the
"Sync" resulting in both droids having an increased data force.

Please have a look at this schematic, your engineers may have better luck
decoding its mysteries.

            +----------------+                +----------------+
            |                |                |                |
            |   +--------+   |      Sync      |   +--------+   |
            |   |-|Data|-|   | +------------> |   |-|Data|-|   |
            |   +--------+   | <------------+ |   +--------+   |
            |                |                |                |
            |    Driod  A    |                |    Driod  B    |
            |                |                |                |
            +----------------+                +----------------+

May the force be with you.

- Sifo-Dyas


[....2 months later....]


Prime Minister Lama Su!,

I hope the army is coming along nicely. The force has given us more clarity in
the last few months. As it turns out, this "Data" that we were so worried about,
is just a method by which the droids store information about their experiences and
orders. Most importantly, the "Sync" ritual was just an exchange of files
from one droid to another in both directions. This is how their data force
increased after the ritual.

Master Windoo has been doing extensive research and has come up with a simplified
experiment to test if this training method can be implemented. He says that you
should start by figuring out how to synchronize data between a folder on one
device (say device A) and a folder on another device (say device B).
In addition to that, a change made to the data on one device should also be made
available to the other device as well. If we have a way to do this then we could
potentially improve the quality of the new clone army. I hope your engineers
are able to make sense of all of this information. Do write back to me if you
need more information.

Please share your method and implementation in great detail with us so
that it can be added to our records in the Jedi Temple. I wish you luck.

May the force be with you.

- Sifo-Dyas


                                 +---------------------+
                                 | What's going on here?|
                                 +------------------+--+
                                                    |
                                                    |
  _                                                 |
  \\                                                |
   \\_          _.-._                               |
    X:\        (_/ \_)     <------------------------+
    \::\       ( ==  )
     \::\       \== /
    /X:::\   .-./`-'\.--.
    \\/\::\ / /     (    l
     ~\ \::\ /      `.   L.
       \/:::|         `.'  `
       /:/\:|          (    `.
       \/`-'`.          >    )
              \       //  .-'
               |     /(  .'
               `-..-'_ \  \
               __||/_ \ `-'
              / _ \ #  |
             |  #  |#  |   B-SD3 Security Droid
          LS |  #  |#  |      - Front View -

(http://www.ascii-art.de/ascii/s/starwars.txt)