StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POExecute arbitrary python code remotely - can it be done?
text
Body
copied!<p>I'm working on a grid system which has a number of very powerful computers. These can be used to execute python functions very quickly. My users have a number of python functions which take a long time to calculate on workstations, ideally they would like to be able to call some functions on a remote powerful server, but have it appear to be running locally.</p> <p>Python has an old function called "apply" - it's mostly useless these days now that python supports the extended-call syntax (e.g. **arguments), however I need to implement something that works a bit like this:</p> <pre><code>rapply = Rapply( server_hostname ) # Set up a connection result = rapply( fn, args, kwargs ) # Remotely call the function assert result == fn( *args, **kwargs ) #Just as a test, verify that it has the expected value. </code></pre> <p>Rapply should be a class which can be used to remotely execute some arbitrary code (<code>fn</code> could be literally anything) on a remote server. It will send back the result which the <code>rapply</code> function will return. The "result" should have the same value as if I had called the function locally.</p> <p>Now let's suppose that <code>fn</code> is a user-provided function I need some way of sending it over the wire to the execution server. If I could guarantee that fn was always something simple it could could just be a string containing python source code... but what if it were not so simple? </p> <p>What if <code>fn</code> might have local dependencies: It could be a simple function which uses a class defined in a different module, is there a way of encapsulating <code>fn</code> and everything that <code>fn</code> requires which is not standard-library? An ideal solution would not require the users of this system to have much knowledge about python development. They simply want to write their function and call it.</p> <p>Just to clarify, I'm not interested in discussing what kind of network protocol might be used to implement the communication between the client & server. My problem is how to encapsulate a function and its dependencies as a single object which can be serialized and remotely executed. </p> <p>I'm also not interested in the security implications of running arbitrary code on remote servers - let's just say that this system is intended purely for research and it is within a heavily firewalled environment. </p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload