Fabric

I did some work on this, available in my git repo. In it I added a directory called test_scp (at the project root), which has a test fabfile and a directory structure for testing.

There are a few problems with the naive method (blind wrapping of scp.py). Some of them come from semantic differences between fabric and the scp implementation. Some come from bugs/edge cases in the module itself. A couple of the issues can be worked around, but others require a more serious look, possibly modification of scp.py itself. This is not a problem, but it does introduce the annoyances of us maintaining our own fork. To that effect I have contacted the original author of the module to see if he is amenable patches from me (and fabric as a whole I guess :))

Anyway issues:

1: scp.py preserves modes, but has no mechanism for changing them while copying. This doesn’t bother me, but I presume that the functionality was put there for a reason, I see no good way of adding it without modifying scp.py.

2: copying globs to a remote host: directories need to be filtered before passing the file list to scp.put. this is done in my code.

3: copying of remote globs to localhost: if there is a directory in the remote glob it raises an error. The only way around this is modifying scp.py.

A meta problem: Do we want to patch paramiko? maintain a fabric.paramikoextentions module… ?? Also, this quesiton affects the port forwarding monkeypatch in a different ticket.

Anyway recursive get/put works great, so that is something!

I had a conversation with the original author of the scp.py module. He suggested it would be better to have fabric use sftp and implement the recursive put stuff itself, as sftp is a bit stabler/more reliable than scp. As such, i have don this in my above-mentioned branch. This works pretty well right now.

Here are a few thoughts regarding where the code is right now:

1. Considering the number of things I had to implement on top of sftp, (e.g. remote_exists, remote_isdir), I think a core/sftp.py might be useful. This could implement a class which provides all these file functionalities. It also could help consolidate contrib/files.py stuff into core, since no external dependencies are required (sftp is built into sshd).

2. In all cases this is ugly as sin and needs cleanup of course — I am not sure on preferred style etc, so guidance is requested.

3. This also implements a remote_glob functionality which touches on ticket #28

4. There are issues re mode management etc that should prolly be discussed in context of this code before it is integrated (and other related tickets too).

Ok, so I did a bunch of cleanup on my branch related to this. The commit message has most of the info. Quick overview tho: added a module: sftp.py. This is the underlying object for operations.get and operations.put. All the remote file ops discussed above are in that module in the FabSFTP class. The code is quite a bit less ugly, and there are some fuller test cases too.

Bumping this to 0.9.2 for now, has potential to be a good new backwards-compatible feature to add.

Just checked in some “get rid of random debugging print statements” fixes, and verified that pulls from multiple hosts do directories in a sane way — specifically creating a directory of $HOSTNAME and pulling to that location.