14 December 2009
Synchronize two folders on a Mac with Automator and Rsync
Power of Automator and Unix combined
I really miss some good Automator tutorials online. It's such a great and I guess underrated and underused tool. Here I'll show you how to combine it with rsync to easily synchronise two folders.
Make a backup before you try it out on your files!
I suggest you make some dummy folders and fill them with some random files to test it out first.
You can find the finished workflow (for Mountain Lion) from the tutoria on the right so you can use it right away and see how it's done for yourself.
And here it is for the patient ones, step by step...
You'll start by opening automator.app, you can find it in your Application folder it has a nice icon which reminds me of eve from Wall-E. Once you start it, it will ask you what kind of workflow you're going to make, we'll choose Workflow:
First action we're going to add is "Ask for Finder Items" (it's inside Files & Folders category, but best way to find actions is to use search), that action simply opens finder window where you can choose file(s) or folder(s).
And here we can write some custom text in "Prompt" field, you can specify which folder will it open as a starting point (Start at), and we'll specify that we want only folders and we do not want multiple selection option.
Next we'll add "Set Value of Variable" action which will get the folder that you chose and write it's name into a variable that we'll use later with Rsync.
Next, we'll repeat it, we need another pair of Ask for Finder Items and Set Value of Variable actions. First we add Ask for Finder Items:
The important thing here, is to ctrl + click (right click) on the Ask for Finder Items text and choose "Ignore Input" option, if you do not do this the whole thing will not work.
Then we'll add Set Value of Variable
And we'll click on Variable drop down and choose New variable and name it for example "target_folder".
Next step is optional, we'll add confirmation window (Ask for Confirmation action) that will ask us if we chose the right folders and print out the names of the folders we're going to sync:
Here you start writing something like "You'll be syncing from" than you drag the variable "source_folder" from variable list below, continue by writing " to " and than drag "target_folder" variable from the list, and finish your confirmation text, for example " is that right?".
Here you're also have to say to automator that you want to ignore input from previous action.
Next we'll add two Get Value of Variable actions that will pick up the values of source and target variables:
Again, ignore the input from the Ask for Confirmation action.
After that, add Run Shell Script action which will tell rsync what to do:
You have to change "Pass input" to "as arguments" and than write rsync -va --delete "$1/" "$2/" $1 represents first variable that you're sending in, and $2 second one (first one being source_folder and second one target_folder). You can play with rsync options here and maybe exclude some files from syncing. You can omit --delete option if you don't want rsync to delete files from the target folder that don't exist in the source folder.
Because of the "--delete" option rsync will delete any files it finds in the target folder that are not found in the source folder! Omit this option if you want just to add missing (and newer) files from the source folder.
And here is another optional action, we'll add New TextEdit Document at the end. That will open new TextEdit window with output from Rsync, so you can see what Rsync actually did.
And that's it! Now we're going to save it, the best way I think is to save the workflow into your Scripts folder (you go to your home folder and there you'll find Scripts folder). I named mine workflow "RsyncTutorial". Then if you have Script Menu turned on in menu bar (If you don't you can enable it by running AppleScript Utility, located at /Applications/AppleScript/, and checking the "Show Script Menu in menu bar" checkbox) you'll see something like this when you click on the Script Menu:
And if you click on it your workflow will start by asking you for folder to sync from.
Comments
What a nice Workflow. Same people with brains gotten figure this out, well done.
Marvelous !
I was searching such tutorial a long time ago, thanks so much :-)
This works great! I was wondering if there is a way to set it up so that once you choose to run the script from the menu bar, it does not ask you to choose folders at all.
Sure, if you want to have it with predefined folders you delete everything from automator workflow except "Run Shell Script" and "New TextEdit Document" (if you want to see the output). And in "Run Shell Script" enter something like this:
rsync -va --delete "/path/to/folder1/" "/path/to/folder2/"
Great tutorial, thanks!
I too have been looking for some time for an easy solution for this. So thank you! However, I keep generating a shell script error. I started with exactly what was cited, leaving the values "$1/" and $2/" then tried replacing the text within the quotes to "Source_Folder" and "Target_Folder" respectively. Still an error. What am i doing wrong? Thank you!
Hi Eric. I'm not sure I understand it right. You should leave "$1/" and "$2/" in the run shell script window. It basically should look just as it is on the screenshot above. Folder names (paths) are passed as arguments to shell script ($1 is first argument we pass to it and $2 is second one) so these names we use in automator itself are irrelevant to bash script.
Hvala puno puno puno i pozdrav iz Pule!
Really nice tutorial, thank you very much
Hi, like Eric I keep getting a shell script error. The folder copies, but the contents do not. Help needed!
Hi, Nivesh, could you send me your automator workflow? If you send me your e-mail over the contact form I have on http://www.bananica.com/Contact/ I'll reply with mine, so you can send me the files and I'll take a look.
Hi there,
your idea of using automator is great!
Do you know if is possible to do a bydirectional syncronization? meaning a mirror sync where the most recent file version (no metter where is stored if in target or destination) is copyed to the other folder.....
That's what I'm looking for....
thanks
Luca
Hi Luca. Yes it's possible. You can read more about it here
Really awesome. Thanks for posting. Just a quick question though. With this script, the original "Source" will not be altered so only the "Target" will be altered?
Yes, that's right. Source folder is left intact.
Also my thanks, this will help me a lot.
Actually, my second "ask for finder items" always get TWO items. the first one again, and the second one. Even if I drag your first three commands into a new window, this happens, whereas it does NOT happen in your script. some hidden details?
oops read read directions ... the bit about ignore input ...
last comment. It might improve the tutorial if you say what the "ignore input" does even graphically. E.g. breaking the link between the above panel and the panel one is editing. Thanks again.
Awesome tutorial. Love the layout for your blog :D was super easy to follow and make sense of automator!
Thank you Aaron! Glad you liked it, and the layout as well, some find it weird with right/left alignments :)
Very clear tutorial. Thanks.
Is it possible to use this or something similar to do a bidirectional sync between two Macs over an network?
Thanks! Now I don't have to buy special software to do my backups. :-)
Is there an easy way to schedule running of the workflow in OS X Lion?
All my files and folders from source_code got lost. I can't find them anywhere, they didn't copy to destination_folder and they were deleted. Is there any way to recover that? Please help, im desperate.
Thanks - just what i needed. copies my scanned photos to an external disk once run.
This may be a bit confusing though on step no 8. I think should be on step number 7. 'Again, ignore the input from the Ask for Confirmation action.'
You were right! I changed it. Thanks!
Great tutorial, thanks! I am looking for a way to do this over a network between my laptop and my MacPro. I saw your tut on bidirectional synchronization on one machine, but is this possible between two machines? I've got about 8GB of files that I need to keep synced.
Thank you!
Jason A.
it is possible but this workflow won't support it without some editing. Quite frankly I haven't figured it out yet but for that I just manually do it in terminal or iTerm with the I.P. addy of the 2 computers.
Hi Jason!
I am myself syncing some folders on my laptop and desktop macs. I'm using rsync over ssh to do it. I'm gonna try to write a tutorial how to do it. There might be some other ways to do it, but this one works for me.
This tutorial is great, I h ave been looking for something like this.
However I have a question. Once the folder sync is set up, will it automatically then sync everything between the folders in the future. Without having to manually run it every time?
Thanks!
Hi there, this is an amazing idea, and was actually the first time I have ever used Automator; so go you for educating me there.
I was wondering if there was a way of making this script run permanently, so it updates the target from the source automatically?
i.e. I drop a file into 'documents' on my system drive, and it automatically backs up on my backup drive.
Thank you so much for this tutorial.
I tried the modified version of your instructions found on practically efficient dot com.
I have not used the terminal nor the Automator before, so I apologize in advance for my total lack of knowledge.
When I type this command in Terminal:
rsync -aE --delete ~/Desktop/testerfolder/ "/Volumes/My Passport/"
I am getting the error:
rsync: opendir “/Volumes/My Passport/.Trashes” failed: Permission denied (13)
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-42/rsync/main.c(992) [sender=2.6.9]
The files actually did copy though.
When I type this command in Terminal:
rsync -va --delete "~/Desktop/testfolder/" "Volumes/My Passport"
I get these error messages:
building file list ... rsync: link_stat "/Users/Jan/~/Desktop/testfolder/." failed: No such file or directory (2)
done
rsync: push_dir#3 "/Users/Jan/Volumes" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at /SourceCache/rsync/rsync-42/rsync/main.c(580) [receiver=2.6.9]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-42/rsync/io.c(452) [sender=2.6.9]
And no files are copied.
When I wrote and saved this in a RunShellScript in Automator:
rsync -va --delete "~/Desktop/testfolder/" "Volumes/My Passport/"
it didn’t work. I tried with both -va and -aE and with and without quotes around the source folder.
I got this message:
The action “Run Shell Script” encountered an error. Check the action’s properties and try running the workflow again.
Thanks again, so much.
I hope it isn't a bother to help me sort this out.
Hi Jan!
The problem with this one:
rsync -aE --delete ~/Desktop/testerfolder/ "/Volumes/My Passport/"
Is that you're syncing to root of your attached disk, and there are always some hidden folders in roots of hard disks (one is .Trashes for deleted files) and these are usually locked. And rsync is trying to delete .Trashes because there's no such folder in your testerfolder/ folder.
So I would suggest that you make a folder on you My Passport disk and then run it, for example:
rsync -aE --delete ~/Desktop/testerfolder/ "/Volumes/My Passport/test_target/"
Here:
rsync -va --delete "~/Desktop/testfolder/" "Volumes/My Passport"
you're missing "/" before Volumes, when you put it like this without "/" you're telling it to search for a folder Volumes in the folder you're currently in, so if you're in your home folder ("/Users/Jan/") it will look for /Users/Jan/Volumes/My Passport/ and there's probably no such folder.
And there's another problem. "~" is a shortcut for home folder, so instead of writing /Users/Jan/ you put just ~, but, if you use that shortcut than you can't put the whole path under quotes.
So the simple solution would be to create some folder on your My Passport disk and than sync it to that. Like I wrote above:
rsync -aE --delete ~/Desktop/testerfolder/ "/Volumes/My Passport/test_target/"
Btw. you need quotes around My Passport path because you have space in the volume name (My Passport), you could do without quotes, but than you have to escape space character like this:
/Volumes/My\ Passport/test_target/
Does not work here (Mac OS 10.6.5, MacBook Pro): Shell script seems to run in an endless loop.
Running the corresponding rsync script from the shell works flawlessly though.
May I send you my automator workflow for examination?
Does this also subfolders? if i have a subfolder with files in the source_folder, does it sync that to the target_folder?
Great workflow. I have been using this for some time now and find it really useful. I would like to know if it is possible to add a if/then/else command to the apple script? I want to be prompted if i would like to run a dry run first. Something like:
set question to display dialog "Would you like to run a Dry Run?" buttons {"Yes", "No"} default button 1
set answer to button returned of question
if answer is equal to "Yes" then
rsync -vaun --exclude=".*" --exclude=".*/" "$1/" "$2/"
end if
if answer is equal to "No" then
rsync -vau --exclude=".*" --exclude=".*/" "$1/" "$2/"
end if
I know i could just create a separate workflow for dry run mode... but that would be too easy!
Thanks
Thanks for the clear tutorial! I've never used automator before and followed perfectly ( I think). Two questions, though:
I'm trying to sync about 1 tb of data from one external drive to another. When I do this, my Mac HD begins to fill up with data. Does all of the data temporarily visit my HD, or did I do something wrong?
Second question is, do you know of an action to display a progress bar to estimate the time to run this script?
Thank you so much for your help!
Thanks. Great info and tutorial.
If I change $1 and $2 to valid paths, can I add it to the Source folder as a Folder Action so it runs whenever a new file is added or existing file changed?
Do I save the workflow as an app or a script or what?
Thanks.
great tutorial!
thanks guy now i've a question
this task run once the syncing. is it possible to create a script that constantly monitors a folder and real-time replication to target folder?
thanks anywaY!
regards
Simon
This is really wonderful stuff, automator is underrated indeed. Almost love how well this one works – i am missing a progress/status while updating/rsyncing two folders though… any chance for that?
Combined with the bidirectional approach from the other post and something like 'lsyncd'… hm : )
What I needed seemed so simple -- a way to synchronize a folder on my office MAC with a folder on a portable device so I could share them with my home MAC. I did NOT want to have to re-copy the entire folder everyday (it is 4 gigs!) and I did not want to have to keep track of which files I modified everyday. Information on the internet pointed to 3rd party software - some free and others with a fee. REALLY? Those were my options? A potential for malware/spyware or monthly charges? After a few hours, I came upon this tutorial. I took the time to follow the instructions, which, at first, seemed very complicated and intimidating. But each step was explained. I had to take a leap of faith when it came to interpreting "Here you're also have to say to automator that you want to ignore input from previous action." I wasn't sure if I'd missed a step, but it turned out to be the same step as 4. It took about 20 minutes to build it in Automator - a program I've never used before. I ran it. IT WORKED! It deleted what I deleted. It created what I created. The folders where in sync. I recommend this procedure to anyone trying to synchronize two folders on a Macintosh. Thank you bananica!
I think it's important to note that this is a one way sync. At least that's how it works for me. Still handy though and with another workflow to sync in the other direction it's very useful.
Thnaks for this great tutorial
I do have 2 questions :
Q1
Is there a way to select a sub folder coming from an app (Open Package Contents) ; it would be usefull to backup a huge MMO folder under Wine for example).
I tried the shell method but i got an error : rsync -va --delete "/Applications/Games (video)/Uru Live.app/Contents/Resources/drive_c/Uru Live/" "/Volumes/d2 Quadra/MMO Backup/Uru Live/
Q2
I used to use arRsync, but since I upgrade to Lion it crash whenever i tried to backup those huge folder plenty of files.
Your Automator Script either.
It work well on some medium size folder sync.
Are ther some usage limit with Rsync ?
Thanks
Q1 : R1
Work well with the command line method ; You just need to be very, very careful when typing the path.
Especially behavior differ if you use a / at the end of the source and destination path ...
/ mean "the content"
Without / mean the folder and its content.
If you omit / for dest, it seems that Rsync try to create a subfolder and crash if it exist...
Sorry for posting in hurry
Q2 : I don't find any answer. It crash ALL the mac with LOTRO folder (20G0 / 10 000 files). All other MMO are Ok ....
Well, I really really wish I hadn't done this. I have no idea what happened but the script didn't work, and it ended up deleting everything.
I set it to copy out of my main work folder and the whole thing has disappeared, it's not in my trash either. Seeing as I was trying to make this command work in order to back up in in the first place I don't have any back ups except an old one from several weeks back. I've just lost a whole load of work and I'm slightly in shock right now.
Do you know how I might be able to reverse this command. Also I can't see any of the files in my recycle bin.
I'm actually quite upset. I followed all the steps perfectly and then downloaded your script to test.
To anyone else, please please back up before you try this.
Apologies for my slightly dramatic post yesterday. I have re-analysed the Automator workflow I made and can only assume that I got the two folders mixed up. As the rsync command effectively makes a mirror of the source folder in the destination folder, selecting an empty folder as the source will leave you with an empty destination folder, regardless of what was in it.
I've tried it again with some assitance from Branko and it's working great, so don't be put off using this script, just be careful because it's potentially quite dangerous if used incorrectly, as I found out the hard way :(
Great tutorial, thanks for sharing!
Ran into it – adopted logic and bingo: works great !
Thanks for sharing. This is a lifetime saver!
This is a great tutorial!
I have some questions:
1. Is it possible to set default value for "Set Value of Variable" action in the workflow without showing a dialog when running it as an application?
2. Is it possible to show the progress of rsync output while it is running without waiting until it finishes and output to the textedit?
3. When there are files with Chinese names involved in the sync, the output in textedit shows, for example, \#346\#262... for Chinese characters. Do you have any idea to fix it?
Thanks a lot.
Here i find a usefull comment about extended attributes from extra88:
Yes, the version of rsync that comes with all versions of OS X is rather outdated. However, if you're going to use that version, you really should add another switch, -E. This tells rsync to include "extended attributes," various file properties, not just resource forks, that can be found on OS X files that aren't generally found on other Unix-based operating systems.
Note single-dash parameters do not have to be written separately 'rsync -a -v -E' and 'rsync -avE' do the exact same thing.
The easiest way to get a newer version of rsync on your Mac is to install Carbon Copy Cloner. Buried deep within the application is /Applications/Carbon\ Copy\ Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync
Within the current version of Carbon Copy Cloner (3.5.1) is rsync version 3.0.6. With this (and I think all versions of rsync that don't come from Apple), the -X parameter is what tells it to include extended attributes, so '/Applications/Carbon\ Copy\ Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync -avX --delete /path/to/source/ /path/to/destination/'. You might also add the -A switch to preserve ACLs (non-Unix file permissions) but if you just making a folder to folder backup within an account on your computer, preserving ACLs probably isn't important.
Great tutorial! Thank you! It worked perfectly for me!
Amazing.
Is there a way for doing this but on two different computers?
If i edit my personal directory on my MacBook Pro, i want those changes to be there when i turn my iMac at home.
Is this possible?
Hi- This tutorial is amazing... except I used another trick and a symlink to put my Dropbox onto one of my external drives. Now, when I run this script, trying to sync a folder in that Dropbox with a folder on another external drive, it hangs up on the 'syncing' bit and nothing seems to happen... any thoughts??
help!
I made a test folder with only 3 items in it and followed your steps carefully including the --delete command. Then I ran the workflow but it started deleting everything from my external drive that contained the target folder instead of only the folder itself. I managed to get it back from my other drives but what happened here? Any ideas?
This is great. Having a progress bar would make it excellent ;)
This still does not work. Even if I copy/paste your shell script mine is in an endless loop :(
Do the quotes in the shell script really have to be typed with quotation marks like "$1/" or should it be $1 instead?
Hey there! So, this works great out of the box for what it does, but I'm curious (and have seen a couple of other commenters ask this as well) about setting this up to automatically sync?
I have a networked drive and I'd love to automatically sync to that drive ala Dropbox whenever I add/remove files from my source folder. So, if I add a new movie to my folder, the syncing process will automatically run and send my new file to the external drive. The same would be true for deleting files.
Any thoughts?
So this is working great, except for it's copying(creating?) a .ovf file for everything it syncs to the other folder... not completely sure what an .ovf file is or how to make it stop making them! any insight into this matter would be greatly appreciated :)
Is there a way to show progress?
Why did it enlarge both folders? The folder I wanted to mirror was 49GB and the other folder was 45 GB. Now both folders are 54 GB.
Great workflow!
I was wondering if there is a way to not copy the icon of the source folder and delete the icon of the target folder. Can anyone answer?
What you want to do is exclude "Icon" file which is the folder icon. You can do that by adding exclude option: " --exclude='Icon'"
@Brank0 | I've written this script rsync -vaE "$1/" "$2/" and adding the exclude option it still copies the icon. Instead, if i don't put the -E it deletes the icon of the target folder and create an empty file named "Icon".
Just Awesome! This helps me soooo much at work, thanks thanks thanks :)
I've been trying to get this to work, but it just doesn't seem to want to. When I do this in Terminal, it works perfectly...but I'm telling it exactly where to go to and from.
This automator workflow works up until the Shell Script...then it errors out after a long wait and says:
rsync: opendir "/dev/fd/3" failes: Bad file descriptor (9)
WTF? That's not even close to any of the directories I selected, and I have no idea what that means. haha.
The drive I'm copying from is on a server connected to my Mac via fibre. I'm trying to actually just create the folder structure from that place into my target place.
The code I'm using is:
rsync -rv --exclude="*.*" "$1/" "$2/"
I've tried with and without the /'s...no dice. Any ideas?
Thanks!
Nevermind!!! I'm a dummie. I never added the Get Variable bit... heh... d'oh.
All good now!
This workflow works very well but I'd like to extend it to recursively sync files from subdirectories of the initial source directory.
E.g. my source folder may look like this:
source
subfolder 1
filea
fileb
subfolder2
filea
filec
subfolder3
filea
fileb
filed
filee
and target folder may look like this:
target
filea
fileb
and after the workflow, target looks like this:
target
filea
fileb
filec
filed
filee
where filea, fileb and any other duplicates are the NEWEST version of that file. I think rsync automatically keeps the newest version but not positive.
Another way to think of what I'm looking for is to take the basic script and repeat it (loop?) for each subdirectory of the source folder.
Any suggestions?
Thanks so much for this tutorial! Made making a local back-up super quick and painless!
Thx for the tutorial.
I was using the shell script function to sync folders when plugging in a certain usb drive... I am struggling because every time a drive is connected now, the shell script function pops up and asks whether to sync or not. How can I remove the work flow, since I already deleted it in automator?
I am looking forward to hearing from you.
Thanks a lot, i love this workflow.
But i've a question, is it possible to save this txt document automatically with the name log"and a variable for Date and Time", because i want to combine it with an ical appointment. Then it's very disturbing, if the TextEdit window constantly pops up.
Excellent workflow, really helped a lot and very well explained. I made a slight change to mine by choosing New Text File at the end and setting a custom time and date name for the file. I wouldn't have been able to do this without having read your walk through and use of variables. Thanks.
Thank you very much. Looked a very long time for such an explanation (I am not a mac user)
Shell script's running for ever on Snow Leopard. Any ideas?
gooood stuff.!
I saved the script as app; works better for me.
thanks
I was searching for this workflow the whole day!!!
Very, VERY Awesome!!!
THX Dag
I tried to create the same workflow to sync a complete drive backup, but i cannot find the Set Variable Task in automator (El Capitan) however I do find Get variable Task. Is this not possible in El Capitan?
Perfect, well documented solution! Great website.. Thanks!
Whatever you do, DO NOT select "Allow Multiple Selection" in the first ‘Ask for Finder items’ action. The rsync action will treat the second item of the multiple selection as the sync destination, and delete its contents completely if '--delete' is set (with no Trash recovery). Ouch! Fortunately, I didn’t lose anything too important. However, I did manage to completely wipe my external drive (400GB of media), and end up copying my Photos library over my Projects folder. Messy.
I am trying to do almost exactly what you have done here. However I would like to copy all files in a folder as well as all files in every sub folder without the folders containing them. Is this possible?
Is it posible to synchronize automatically? I know there are some folder options when a new file comes to the folder, but I would like to synchronize when I modify a file or, alternatively, every few minutes.
What a crap!! It will be easer to use rsync on a script ... or like the other world with an GUI interface that works ... see FreeCommander
What's going on with the Mac world. Why do you accept all this crap? My gerneration was used that Mac was the best, long time ago.
I know this is 13 years late but.... What a marvellous tool, - thanks!!!!
However I seem to not get the text editor at the end to show anything - it just opens a blank text editor window, is there something I'm missing to get it to actually import the output from rsync??