Fabric

Spinoff from #183, though that’s mostly not important.

Need to test real, honest to God SSH level stuff, i.e. run() and its constituent underpinnings, if we’re to have any chance of a serious test suite.

There are, as I see it, 3 ways to do this:

1. Create our own high-ish level mock of the Paramiko Channel object, since that’s what our code interacts with, so that we can say “input X results in output Y”.
   • Pros: it’s a public API that won’t change much
   • Cons: practically reimplementing the entire class structure of Paramiko, insofar as threading behavior and state-keeping goes. So far, this has not borne out anything fruitful (I tried.)
2. Mock/monkeypatch lower-level Paramiko objects, which right now looks like it'd be either Transport or Packetizer.
   • Pros: lowest possible level of monkeypatching means smallest potential amount of mock code, closest to “real” execution without actually requiring a real SSH server.
   • Cons: heavily tied to private or semiprivate APIs within Paramiko; possibly just as complicated to effect as the higher level mock, depending.
3. Use real SSH server.
   • Pros: best testing possible, as it’s the real deal. No mocking needed, either.
   • Cons: tons:
     • Slows down tests, possibly by a lot if testing system is not adjacent network-wise to target system or test commands involve disk I/O.
     • Requires knowledge of state on target system in tests: return value of commands, sudo passwords, etc.
     • State might also need to be reset each time, which is subject to error.
     • Requires access to target system (potential security risk).
     • If target system is intended to be local VM and not a single specific machine, then the barrier to running tests becomes much higher than I'd like it to be. (Even if target system is simply localhost, then the user has to have a running sshd, which is common but not ubiquitous.)
     • If target is a distinct system, then tests will fail if system goes down or is unavailable or user is not online, so no more hacking on the airplane or out in the sticks.
     • Target could be CI server’s localhost, which is relatively neat and all, but then we’re leaning heavily on CI for full testing and that brings us back to “contributors cannot easily run the test suite”.
     • Probably more I haven’t even thought of yet.
4. Use fake SSH server. I.e. use Paramiko’s server side functionality.
   • Pros: while we'd probably still need to actually run the server in another thread or process, it’s still effectively mocking instead of hitting up a real system’s sshd, as we'd be able to mock out what it does when it receives commands (I assume).
   • Cons: might not actually be possible to fake the commands, could be a decent amount of work depending on the API required, requires allocation of a network port (not a huge deal tho).

As indicated, I've taken a moderately sized stab at option 1 already and failed, and am currently investigating option 2. There’s lots of support in IRC for option 3, including offers for a target testbed, which is good, though note all the cons. I'd very much rather stick with an in-code solution if it’s not too painful. Option 4 just occurred to me now, though I recall coming up with it in the past at some point. May be worth looking at 4 before 3.