What is Reddit's opinion of git-annex?

From 3.5 billion Reddit comments

I don't understand why you were downvoted. There are 100 alternative solutions to Google Drive, self-hosted ones and cloud services.
I'm using git-annex syncing to two locations: S3 (encrypted) and a git repo that's on a server under my control. Works beautifully.

Edit: I just realized git-annex has a special remote for Google Drive, maybe you Drive users could check that out :-)

Ansible+git to back up the setup and configuration

git for code

git-annex for the odd binary file and random notes

I use Google Play Music and Netflix for music and movies respectively and therefore do not require a large-scale backup solution. I'm thinking of moving some lower-priority files into Amazon Glacier.

git-annex is great for syncing content between systems, including Android devices. I have set it up recently to sync photos and files between my desktop, phone and tablet.

It sounds like a simplified version of git-annex. git-annex manages files via symlinks to a special directory in .git where the actual data is stored based on a hash (and a few are available, SHA256E being the default. The E means that the file extension is also part of the hash).

It has a separate branch where it keeps track of what files exist and which client has them. It can automatically move/copy data around and tell you where it's supposed to be as well as check the files for corruption.

See http://git-annex.branchable.com/ for more info, especially the internals page.

one shot version:

on host1: find . -exec md5sum {} \; | sort > files_host1.txt

on host2: find . -exec md5sum {} \; | sort > files_host2.txt

copy both files_host1.txt and files_host2.txt on one of the hosts; then run this: diff files_host1.txt files_host2.txt

long term version: use git-annex, particularly the 'git annex whereis' command, and: http://git-annex.branchable.com/tips/finding_duplicate_files/

Welcome to the wonderful world of git-annex and how it works(video).

Check out git-annex.

> git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space.

The backing files themselves can be stored on the filesystem or on various cloud systems.

git-annex is what I use quite a lot. It appears to have some sharing features which I have just discovered: http://git-annex.branchable.com/assistant/share_with_a_friend_walkthrough/

I am also pretty sure that git annex will work over tor, but this comes down to the exact functionality that you want.

git + git-annex. Even if you don't need the meaty features of git-annex, having dropbox as a special remote is very convenient for browsing stuff from other devices, sharing with plebs (a.k.a. anyone not using source control, etc). You can even explore the other special remotes such as rsync and bup, or mr for keeping multiple repositories in sync when you're using one per class or per project.

You could take a look at git-annex assistant (available for Lin/Mac/Win).

It uses git with special handling for large files and the assistant is really easy to use.

I guess you need to put some work into it to preconfigure it nicely, but I think that it could do the job.

I don't quite know if the Retroshare "channels" would be something that could help you.

I do however know that git annex might be worth a look if Retroshare doesn't work out for you.
Here's a video about that

I understand, but you get the same thing with vcsh + mr eventually. No need to learn it all at once, but then you might also consider throwing git-annex into the mix for maximum awesomeness. Have fun! :)

Backups

I personally use a tool called rlcone (rsync but for the cloud). Since I'm using a VPS, I do a cronjob every hour(this is very overkill) to sync the server folder to my Google Drive (I have a Google One subscription).

My crontab entry 0 */1 * * * rclone sync --exclude "/plugins/CoreProtect/database.db" -v "/path/to/server/" "Google Drive:rclone/serverv4-paper" ^ Coreprotect database can be very big. So I exclude the database on my backups.

You might want to pause the Minecraft world saving while the backup is running. I wrote a custom plugin to simply stop server saving 1 minute before the hour hits and continue the saving 5 minutes after it.

Snapshots

If you really want a snapshot backup, I recommend you to use git-annex using full backup mode. You can simply use a private GitHub repository (I self host my own git repo using GitTea) while having a separate server for the actual backup file.

Map Plugins

I don't really like using dynmap because it hogs CPU & memory usage (personally). I use Overviewer and run a cronjob every night to update the map. After that, I use rclone to push the file to my webserver (separate from the Minecraft server) and serves it automatically (Caddy server).

There are two commonly used tools to use git for large binary files. git lfs and git annex. Both can be challenging to set up for the first time and require you to be able to install and configure some software.

Git annex in particular showcases a "use case" very similar to your question so maybe try this first. I don't think git lfs allows this kind of fine grained tracking.

Can you simply pair an Rsync pull with a push?

Also, I've never used it, but you might also want to check out BTSync.

Personally, I keep wondering if someone could combine Git with Rsync (or some other synchronization tool). Some of the slowness with Git push/pull is due to it not doing a synchronization, but rather a fetch and fast-forward merge. So, it might be faster to run a synchronization on the .git directory, then force a checkout of HEAD.

The advantage being you get Git's version control, etc., but the data transfer efficiency of some other tool.

I presume Git works the way it does because it was built for text files, so a fetch and merge is effectively an delta-only update. But when given other types of files, it falls back to transfering the entire file, since it can't “see inside” them, and it's never been optimized for large-file transfer.

Though, maybe there is some transport helper for Git that has been optimized for that? Oh, and there's also some Git add-on for dealing with binary files, the name of which escapes me at the moment.

(Pardon the rambling, my coffee seems to be kicking in.)

Edit: And then there's git-annex.

git-annex was designed to make managing large binaries in git more sane. You have to explicitly tell it when to up/download a file or directory, giving back some of the granularity of SVN but with the decentralized model of git around it.

http://git-annex.branchable.com

I really like the approach of git-annex. I use it very much for personal data and also a little bit for personal projects. The advantage is that not every clone needs to have the full data present. It is also possible to drop files forever if they are not needed anymore (from all repos or from a single one). This is also possible in plain git, but involves a rewrite of the history which can result in a mess.

I see some problems for larger and distributed projects:

• It seems that every clone is registered in a branch called “git-annex”. This would become a big list of all the cloned repos. Maybe one could forbid merging this branch in the project’s master repo.

• Locking is not possible, nevertheless. I never needed locking myself. Merging binary assets is often impossible and I understand that locking is an easy way to not introduce conflicts. (Especially in larger groups.)

• GitHub and other (private) hosting solutions probably do not allow usage of git-annex out of the box.

My conclusion: git-annex is awesome. Commands like “git-annex list” are really useful to me, but they do not make much sense for larger groups with many peoples (it is not necessary to know if another person has the file in his repo, generally). It could probably scale to a group of many persons, but my only experience is working with git-annex alone as a single person. It is also good to know what git-annex is not.

The git-annex assistant creates a synchronised folder on each of your OSX and Linux computers, Android devices, removable drives, NAS appliances, and cloud services. The contents of the folder are the same everywhere. It's very easy to use, and has all the power of git and git-annex.

http://git-annex.branchable.com/assistant/

git-annex springs to mind as an appropriate tool. You can easily encrypt your devices underneath the annex.

Not sure how easy it would be to integrate par2 or similar with the git-annex workflow, but in the worst case you can run par2 over an entire remote.

I am no git expert, but I don't think the problems are that fundamental. git add will work only after merge, and merge itself does not take long. Sync takes time, and git-annex syncs to another branch, which is never checked out. After sync, both master and sync branch can merge remote and local changes respectively.

These pages explain how it works. http://git-annex.branchable.com/sync/ http://git-annex.branchable.com/design/assistant/

You should give git-annex a go. I'm currently using it to keep the files on my laptop and desktop in sync with one another. Since it got a ton of funding on Kickstarter, the dev will also be creating a web-frontend and an Android app sometime in the next year, but if you use linux, the actual program is already available. Of course, if you're afraid of using the terminal, you'll have to wait.

Also, with git-annex you only use your own machines: none of your data gets onto Google or Amazon-owned servers.

> Combining those branches is a merge, even though no single file was modified in more than one branch.

I see now. Yes, that kind of merge would have to be handled by even a "simple" VCS.

> Perforce sucks, and yet it's still the best option around, by far, because it's the only option that permits for everyone to work within the same repo.

We are not so far apart then.

I do wish that large-files were handled better by default. From what I've read, the most time-consuming part is packing with deltas, and though the behavior is configurable, the default is quite slow.

But I wouldn't mind if git discouraged big files. Minimally, I wish that git would issue a warning when you try to add files larger than some configurable threshold.

If you really want big files in git, you might want to try git-annex. It's what I had in mind, maybe what you had in mind, and much more, including support for remote storage via scp, rsync, RESTful web-link, or Amazon S3.

>> "git-annex is not some flaky script that was quickly thrown together. I wrote it in Haskell because I wanted it to be solid and to compile down to a binary. And it has a fairly extensive test suite."