Remotes - the new, exciting alternative to Externals.

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Remotes - the new, exciting alternative to Externals.

Alex Tweedly

I've been playing with writing externals for Revolution lately. It's
been interesting. I'm very glad to know I can do it, and should I ever
need to, I will. But I can't say it's been a lot of fun. Writing C code
is frustrating and not very productive in the best of circumstances
(e.g. supported by a symbolic debugger, protected processes, etc.) In
the circumstance of running an external with Rev's IDE (where many
errors cause immediate termination of the IDE, with no diagnostic info
provided at all, far less any chance to debug), it is very unproductive.
So while it's good to know that externals are possible for me, I also
know that they are to be avoided if reasonable alternatives exist.

So I started looking for alternatives .....

Reasons to write externals in the first place.
1. to get access to hardware or other facilities not otherwise available
2. to get higher performance (e.g. matrix ops, image processing, etc.)
3. to take advantage of existing library packages
4. other significant but less common reasons

As described above, I don't like C (far less C++). The language I do
like (assuming Transcript isn't an option) is Python.

Python is, in many ways, similar to Transcript. They're both scripting
language, dynamic types (or typeless), associative arrays, etc. - that's
largely why I like them both.  But they're also quite different, and
complementary, in many ways. So, without falling into the trap of
becoming a comparison between the two languages, I believe that a number
of the reasons for writing an external would equally apply to using Python.

Of course, it would be difficult to write an actual external in Python.
But I realized that one of the strengths of both Python and Transcript
is that they have good, simple socket facilities. So I decided to try
"Remotes" - socket-based Python add-ons to Transcript.

I wrote a small (30 lines) Python server which accepts "commands" over a
TCP socket, and returns its result over the same TCP connection  (see
the end of the email for the source code of the Python program). The
"commands" are in fact snippets of Python code; so I can develop the
Transcript stack and just change the snippets of Python to be sent and
executed, without even restarting the server. So the whole process is as
dynamic as Rev usually is.

As an example, I used the Python Imaging Library to calculate the
Histogram of a JPEG, done as two steps (to avoid making it too easy). So
there are two buttons - one to open and specify a file, the other to
calculate th histogram for it.
The script of the first button is

> on mouseUp
>   local t
>   answer file empty with filter "JPEGs,*.jpg"
>   put it into theFile
>   put "import PIL.Image" & cr \
>       & "from PIL import *" & cr \
>       & "global im" & cr \
>       & "im = Image.open(" & quote & theFile & quote & ")" & cr  into
> theScript
>   put theScript & cr & "endofscript" & cr into t
>   reStart
>  
>   SendPacket t
> end mouseUp

While the second is

> on mouseUp
>   local t
>   reStart
>   put "hi = im.histogram()" & cr \
>       & "return_result_list(wfile, hi)" into theScript
>   put theScript & cr & "endofscript" & cr into t
>   SendPacket t
> end mouseUp

Notes.
1. The "restart" handler is used to reset any open sockets, then open
the socket to the Python server. It could be done more efficiently, but
this was simpler (reusing some socket code I had lying around).
2. sendPacket sends the data in a packet, waits for the reply and puts
the response into a field so I can check it is getting back the right
answers. It also gives me the time between sending the packet and
receiving the end of the result.

Results ?

First, and most importantly, it succeeded in the primary aim. I was able
to make use of a library that exists and provides a lot of functionality
that wouldn't otherwise have been available to me. And it allowed me to
do so without writing C :-)

Second, it was reasonably easy to debug; each of the client and server
could be run independently, with trace or debug output or using a
debugger. The ability to track what was going on by monitoring the
packets sent was very helpful.

Third, the efficiency was very good. The time for the whole operation
(two  x  socket open + send + read) was around 42 msecs for the sample
photo being used earlier (the one that took 1250 ms in Transcript). More
importantly, a large photo took 480 msec versus 500 msecs for the naive
C external.  Remember the imageData is NOT being passed over the
interface;the filename is passed instead, and the Python code opens and
reads the jpeg file. The jpeg file is around 1.5 Mb.
Note this photo is 3008 x 2008, so the imageData is 24 Mb. I tried
passing the imageData and that added 250 msecs to the overall time - so
even with that, it's still within 50% of the speed of a straightforward
C implementation.

Fourth, it is very simple. Started yesterday evening, had it working by
midnight, had it still working in about half the number of lines of code
by 2 am :-)   Changing to a more efficient method (i.e. opening a single
connection and sending multiple command / result pairs over it would) be
a bit more work, and being less simple might be less robust initially.

Fifth, it offers lots of options. Develop Python functions and include
them in a special-purpose server, to get more efficiency and easier
coding; leave the server sparse and have all the flexibility. Do both. ....

Sixth, it's one more tool. I don't expect to use it often - but I have a
couple of projects that I started in Rev and abandoned (i.e. abandoned
use of Rev, switched to Python) which I could now do in 80% Rev + 20%
Python with very little problem (both because of availability of a
library in Python versus what was available in Rev).

Seventh (just because Mark W. mentioned it in the Q&A at RevCon), this
uses a separate process, and so will allow Transcript stacks to take
advantage of multiple processor machines (or multi-core CPUs), before
Transcript is extended to provide threads.


I'd be happy to hear any comments on the approach, answer question, help
anyone else who wants to play with it, etc. Or email the sample server
and stack.

Finally - a short list of areas where I think Python offers a library
that would be a good reason to try this approach. This isn't a list of
why Python is "better" than Transcript.

1. Python Imaging Library
2. NumericArray (matrix math)
3. email library (i.e. rfc 822 headers, MIME encoding/decoding, etc.);
also POP, IMAP and SMTP.
4. pysqlite - database access via SQLite
5. OpenGL
6. PDF generation (via ReportLab's toolkit)


Python server code:

> # This little program illustrates how easy it is to write network
> # servers using the classes in the SocketServer module.
> import SocketServer
>    
> def return_result_list(where, pList):
>     formatted = [ "%d" % x for x in pList]
>     where.write(",".join(formatted) + "\n")
>    
> class TranscriptHandler(SocketServer.StreamRequestHandler):
>     def handle(self):
>         global wfile
>         # Read lines of text, limiting each to 512 bytes.
>         # This will prevent someone trying to crash the server machine
>         # by sending megabytes of data.  
>         lines = []
>         while True:
>             new = self.rfile.readline(512).rstrip()
>             if new == "endofscript":
>                 break
>             lines.append(new)
>         text = "\n".join(lines)
>         wfile = self.wfile     # for use in user scripts, to
> wfile.write() results.
>         exec(text)
>         self.wfile.write("endofresult\n")
>
> if __name__=='__main__':
>     # Create an instance of our server class
>     server=SocketServer.TCPServer( ('', 7779), TranscriptHandler)
>     # Enter an infinite loop, waiting for requests and then servicing
> them.  
>     server.serve_forever()



--
Alex Tweedly       http://www.tweedly.net



--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.11/44 - Release Date: 08/07/2005

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

Jon-3
Well done, Alex!

:)

Jon


Alex Tweedly wrote:

>
> I've been playing with writing externals for Revolution lately. It's
> been interesting. I'm very glad to know I can do it, and should I ever
> need to, I will. But I can't say it's been a lot of fun. Writing C
> code is frustrating and not very productive in the best of
> circumstances (e.g. supported by a symbolic debugger, protected
> processes, etc.) In the circumstance of running an external with Rev's
> IDE (where many errors cause immediate termination of the IDE, with no
> diagnostic info provided at all, far less any chance to debug), it is
> very unproductive. So while it's good to know that externals are
> possible for me, I also know that they are to be avoided if reasonable
> alternatives exist.
>
> So I started looking for alternatives .....
>
> Reasons to write externals in the first place.
> 1. to get access to hardware or other facilities not otherwise available
> 2. to get higher performance (e.g. matrix ops, image processing, etc.)
> 3. to take advantage of existing library packages
> 4. other significant but less common reasons
>
> As described above, I don't like C (far less C++). The language I do
> like (assuming Transcript isn't an option) is Python.
>
> Python is, in many ways, similar to Transcript. They're both scripting
> language, dynamic types (or typeless), associative arrays, etc. -
> that's largely why I like them both.  But they're also quite
> different, and complementary, in many ways. So, without falling into
> the trap of becoming a comparison between the two languages, I believe
> that a number of the reasons for writing an external would equally
> apply to using Python.
>
> Of course, it would be difficult to write an actual external in
> Python. But I realized that one of the strengths of both Python and
> Transcript is that they have good, simple socket facilities. So I
> decided to try "Remotes" - socket-based Python add-ons to Transcript.
>
> I wrote a small (30 lines) Python server which accepts "commands" over
> a TCP socket, and returns its result over the same TCP connection  
> (see the end of the email for the source code of the Python program).
> The "commands" are in fact snippets of Python code; so I can develop
> the Transcript stack and just change the snippets of Python to be sent
> and executed, without even restarting the server. So the whole process
> is as dynamic as Rev usually is.
>
> <snip>

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

Glen Bojsza
This is what I think would make a great Rev Scripting Conference series.



On 7/9/05, Jon <[hidden email]> wrote:

> Well done, Alex!
>
> :)
>
> Jon
>
>
> Alex Tweedly wrote:
>
> >
> > I've been playing with writing externals for Revolution lately. It's
> > been interesting. I'm very glad to know I can do it, and should I ever
> > need to, I will. But I can't say it's been a lot of fun. Writing C
> > code is frustrating and not very productive in the best of
> > circumstances (e.g. supported by a symbolic debugger, protected
> > processes, etc.) In the circumstance of running an external with Rev's
> > IDE (where many errors cause immediate termination of the IDE, with no
> > diagnostic info provided at all, far less any chance to debug), it is
> > very unproductive. So while it's good to know that externals are
> > possible for me, I also know that they are to be avoided if reasonable
> > alternatives exist.
> >
> > So I started looking for alternatives .....
> >
> > Reasons to write externals in the first place.
> > 1. to get access to hardware or other facilities not otherwise available
> > 2. to get higher performance (e.g. matrix ops, image processing, etc.)
> > 3. to take advantage of existing library packages
> > 4. other significant but less common reasons
> >
> > As described above, I don't like C (far less C++). The language I do
> > like (assuming Transcript isn't an option) is Python.
> >
> > Python is, in many ways, similar to Transcript. They're both scripting
> > language, dynamic types (or typeless), associative arrays, etc. -
> > that's largely why I like them both.  But they're also quite
> > different, and complementary, in many ways. So, without falling into
> > the trap of becoming a comparison between the two languages, I believe
> > that a number of the reasons for writing an external would equally
> > apply to using Python.
> >
> > Of course, it would be difficult to write an actual external in
> > Python. But I realized that one of the strengths of both Python and
> > Transcript is that they have good, simple socket facilities. So I
> > decided to try "Remotes" - socket-based Python add-ons to Transcript.
> >
> > I wrote a small (30 lines) Python server which accepts "commands" over
> > a TCP socket, and returns its result over the same TCP connection
> > (see the end of the email for the source code of the Python program).
> > The "commands" are in fact snippets of Python code; so I can develop
> > the Transcript stack and just change the snippets of Python to be sent
> > and executed, without even restarting the server. So the whole process
> > is as dynamic as Rev usually is.
> >
> > <snip>
>
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

Alex Tweedly
Glen Bojsza wrote:

>This is what I think would make a great Rev Scripting Conference series.
>
>  
>
I can't see it fitting in the Scripting Conference series. AFAIK, that's
supposed to cover "introductory" material (though I've learnt a lot by
reviewing and/or attending the conferences, so introductory doesn't mean
basic).

And I just can't see "remotes" as an introductory topic. Also, I can't
see it as being interesting to a very large number of people (maybe to a
fair number of the tinkerers who are active on this list - but I think
that's a biased sample). The conference schedule already runs into
November - and I can think of quite a few more topics that ought to be
added before RR should allow a minority topic like this into the conf
schedule.

But I can see it might be a good topic to prepare more material about,
and then schedule some time to have a pre-arranged "chat" session about
it. So if Bjornke doesn't mind us hi-jacking a time slot on his chat
server, we could just pick an evening (or other time) and all get on the
chat session to talk about it.


--
Alex Tweedly       http://www.tweedly.net



--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.11/44 - Release Date: 08/07/2005

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

Björnke von Gierke
On Jul 10 2005, at 00:51, Alex Tweedly wrote:

> But I can see it might be a good topic to prepare more material about,
> and then schedule some time to have a pre-arranged "chat" session
> about it. So if Bjornke doesn't mind us hi-jacking a time slot on his
> chat server, we could just pick an evening (or other time) and all get
> on the chat session to talk about it.

come as you are :)
the chat is free for anyone, but if you want to make something
"official" just name a date and time to me, and i'll announce it...
I'd rather not have more then one event per evening, and i think next
friday is (hopefully) the announcement for the winners of the advanced
contest, and the sunday in a week is taken too (more on that on
wednesday).

greetings
Bjoernke

--

http://contest.wecode.org
Now running: the first ChatRev coding contest!
sponsors:
Altuit
Andre Garzia
Fourth World
Karl Becker
Runtime Revolution
TidBITS in cooperation with eHUG

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

mwieder
In reply to this post by Jon-3
Jon-

Saturday, July 9, 2005, 2:52:35 PM, you wrote:

J> Well done, Alex!

Indeed! <applause>

--
-Mark Wieder
 [hidden email]

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
--
 Mark Wieder
 ahsoftware@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

Charles Hartman
In reply to this post by Alex Tweedly
I for one would _love_ to be able to write externals for Rev in  
Python (especially Python 2.4), a language I know reasonably well and  
like very, very much.

Charles Hartman

On Jul 9, 2005, at 6:51 PM, Alex Tweedly wrote:

> Glen Bojsza wrote:
>
>
>> This is what I think would make a great Rev Scripting Conference  
>> series.
>>
>>
>>
> I can't see it fitting in the Scripting Conference series. AFAIK,  
> that's supposed to cover "introductory" material (though I've  
> learnt a lot by reviewing and/or attending the conferences, so  
> introductory doesn't mean basic).
>
> And I just can't see "remotes" as an introductory topic. Also, I  
> can't see it as being interesting to a very large number of people  
> (maybe to a fair number of the tinkerers who are active on this  
> list - but I think that's a biased sample). The conference schedule  
> already runs into November - and I can think of quite a few more  
> topics that ought to be added before RR should allow a minority  
> topic like this into the conf schedule.
>
> But I can see it might be a good topic to prepare more material  
> about, and then schedule some time to have a pre-arranged "chat"  
> session about it. So if Bjornke doesn't mind us hi-jacking a time  
> slot on his chat server, we could just pick an evening (or other  
> time) and all get on the chat session to talk about it.
>
>
> --
> Alex Tweedly       http://www.tweedly.net
>
>
>
> --
> No virus found in this outgoing message.
> Checked by AVG Anti-Virus.
> Version: 7.0.323 / Virus Database: 267.8.11/44 - Release Date:  
> 08/07/2005
>
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>

Charles Hartman
Professor of English, Poet in Residence
Connecticut College
[hidden email]
*the Scandroid* is at cherry.conncoll.edu/cohar/Programs.htm


_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Remotes - the new, exciting alternative to Externals.

J. Landman Gay
In reply to this post by Alex Tweedly
On 7/9/05 5:51 PM, Alex Tweedly wrote:

> Glen Bojsza wrote:
>
>> This is what I think would make a great Rev Scripting Conference series.
>>
>>  
>>
> I can't see it fitting in the Scripting Conference series. AFAIK, that's
> supposed to cover "introductory" material

Right, we're doing a beginner series right now. It is sort of a trial
run. If interest stays high, I'd like to do an intermediate series, and
maybe an advanced after that. We're talking a couple of years into the
future though, and that's too far away to really plan.

But I do jot down any suggestions I see on the list here. No promises,
but I'm keeping track. Meanwhile, the ChatRev idea sounds good if you
want to cover this sooner than 2007.

Still working on TimeWarpGadget...

--
Jacqueline Landman Gay         |     [hidden email]
HyperActive Software           |     http://www.hyperactivesw.com
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution