Hubub: an alternative strategy for a "standard port registry"

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Hubub: an alternative strategy for a "standard port registry"

david bovill
A few years back I worked on a project for embedded devices, in which we
looked at QNX and similar systems. These systems used a simple dispatching
mechanism between processes to achieve the equivalent of multithreading.
This architecture suggests to me that we could do something similar with
LiveCode and build upon the socket based interprocess communication that
certain apps are using (PowerDebug and tRev and others), but standardising
the communication so that we avoid socket collisions.

Therefore I think rather than adopting a simple convention, we should
consider implementing a simple convention in code, wrapping the message
dispatching, and socket-service discovery in simplest possible abstraction
layer we can devise.

In this scenario there would be one Livecode app, that acts as a
communication hub (lets call this app "hubub". Specific applications like
Powerdebug would register their ports with it, and we would have some
standard light weight wrappers around standardised dispatching methods.

I think this "service discovery" app could be very minimal but also offer a
couple of extras that justify it's more general existence on peoples
desktops as part of a LiveCode distribution. The two services I am thinking
of are:

   1. Background shell execution
   2. Plugin stacks for background execution

I want to create / use this app in order to allow background indexing of
large LiveCode projects using GIT, but I've had the need and the list shows
others also have had similar needs for background processing any number of
times. I think it should be possible to do this in a standardised way so
that all developers can use these facilities simply and elegantly.

I think this would be beast achieved by having the app listen one a single
standardised port to listen on for datagram connections? I'm aware that many
frameworks have been developed in other languages for this type of purpose,
and that they can grow large and unwieldy, that http may be a better way to
go, but I think starting with datagrams is the simplest and fastest?

*Background shell execution*
There would be a standard folder in which to place shell commands and
externals. This would make it easier to install these in a number of cases
where dependencies are not an issue, as the "background dispatching" app,
would be able to shell out to them directly in a known directory.

Shell commands like GIT, or image magick are large and complex beasts with
many uses. Developers could share access to these tools and therefore more
general purpose libraries could evolve around these code bases.

*Background Stack Execution*
It would be possible for any developer to create a stack, register it with
the "background hub" app - probably saving it in a plugin folder for the
hubub app. You could even do this dynamically - create a stack, save it and
have hubub execute it in the background.

What do people think - useful? Is there a better way to do this?

On 20 January 2011 00:37, Mark Wieder <[hidden email]> wrote:

> Richard-
>
> Wednesday, January 19, 2011, 8:28:03 AM, you wrote:
>
> > On 1/16/11 9:45 AM, Mark Wieder wrote:
> >> hi-
> >>
> >> David Bovill's post on the use-list today made me realize that we
> >> should maybe have a registry for ip ports to avoid conflicts. I'm not
> >> sure what a good way to do this might be, no doubt a web page
> >> somewhere. I'm using port 44553 for PowerDebug communication. It's
> >> the same port used by RealBasic for debugging, so I figured there
> >> wouldn't be a conflict since nobody would be debugging both of those
> >> environments at the same time. Has anybody else allocated ip ports for
> >> special purposes?
>
> > Not yet, but I can envision doing so later this year.
>
> > Good move to set up a list here for that.
>
> I forgot that I'm also using 44554 for the backchannel. So that's two
> for me:
>
> 44553 PowerDebug
> 44554 PowerDebug
>


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Hubub: an alternative strategy for a "standard port registry"

Robert Brenstein-2
On 20.01.11 at 11:19 +0000 David Bovill apparently wrote:

>
>Therefore I think rather than adopting a simple convention, we should
>consider implementing a simple convention in code, wrapping the message
>dispatching, and socket-service discovery in simplest possible abstraction
>layer we can devise.
>
>In this scenario there would be one Livecode app, that acts as a
>communication hub (lets call this app "hubub". Specific applications like
>Powerdebug would register their ports with it, and we would have some
>standard light weight wrappers around standardised dispatching methods.
>

I think you are on to sth... Sounds like a better solution than
central repository of fixed port assignments.

Robert
Reply | Threaded
Open this post in threaded view
|

Re: Hubub: an alternative strategy for a "standard port registry"

Bert Shuler
Sounds like http://en.wikipedia.org/wiki/Port_mapper

On Jan 20, 2011, at 6:27 AM, Robert Brenstein wrote:

> On 20.01.11 at 11:19 +0000 David Bovill apparently wrote:
> >
> >Therefore I think rather than adopting a simple convention, we should
> >consider implementing a simple convention in code, wrapping the message
> >dispatching, and socket-service discovery in simplest possible abstraction
> >layer we can devise.
> >
> >In this scenario there would be one Livecode app, that acts as a
> >communication hub (lets call this app "hubub". Specific applications like
> >Powerdebug would register their ports with it, and we would have some
> >standard light weight wrappers around standardised dispatching methods.
> >
>
> I think you are on to sth... Sounds like a better solution than
> central repository of fixed port assignments.
>
> Robert
>



[Non-text portions of this message have been removed]



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/revInterop/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/revInterop/join
    (Yahoo! ID required)

<*> To change settings via email:
    [hidden email]
    [hidden email]

<*> To unsubscribe from this group, send an email to:
    [hidden email]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Reply | Threaded
Open this post in threaded view
|

Re: Hubub: an alternative strategy for a "standard port registry"

Richard Gaskin
In reply to this post by david bovill


--- In [hidden email], David Bovill <david@...> wrote:
...
> Therefore I think rather than adopting a simple convention, we should
> consider implementing a simple convention in code, wrapping the message
> dispatching, and socket-service discovery in simplest possible abstraction
> layer we can devise.

I think this is a very powerful compliment to a reference list of known ports.  Together, the two provide options for a wider range of solutions than either could support by itself.

The IANA maintains a list of assigned port numbers:
http://www.iana.org/assignments/port-numbers

This provides helpful one-stop-shopping for ports known to be in use in existing tools.

A new framework for handling ports would be very powerful and useful, but would require changes to existing tools.

I think it's very worth pursuing, and to the degree that it's easy for developers to use I would encourage its adoption going forward.

But I can also see a good many cases where devs making simpler tools could benefit from the ease of just checking a reference list and avoiding known ports, as the IANA has provided the larger dev community for years.

Having both would address both sets of needs, allowing devs to use whichever is most appropriate for the task at hand.



SOMEWHAT-RELATED BACKGROUND:  
The sad tale of stdLib, why no one uses it, and why frameworks are only somewhat better
--------------------------------------------------------------------------
This may or may not be all that relevant, but touches on issues that seem related so here goes:

The original mandate for stdLib was to provide a common place to put common handlers, things that most people use in most projects that you can just call as needed, and ignore anything that's not of interest for the project at hand.  When Andre proposed the idea at the first Monterrey con back in '03, it seemed a good idea worth fleshing out.

While it's certainly desirable to have handlers as discrete things that can be used individually or not as one sees fit, in practice this ran into a stumbling block that has prevented stdLib from growing:

A single handler can be useful, but sets of related handlers that work together based on known structures can be orders of magnitude more useful.  Unfortunately, that means that the developer using such handlers must fist read far more about the system as a whole in order to use it.  Rather than just look for a handler with a name that seems to provide a solution, the developer must read at least some introductory material to understand how the handlers relate and what sorts of data they expect to be working with.

Preferences handling is a good example.  The version of stdLib posted here doesn't (currently) include prefs handling, because as simple as it seems up front it turns out that truly convenient prefs handling will make use of common app info (specifically, the vendor and the product name) in order to create the nested folders for the prefs file (e.g., ~/Library/Preferences/FourthWorld/devo.prefs).

In our own frameworks, Ken and I use a central global to store such common info, along with other stuff so we can have things like a set of handlers for doing prefs with ultra-simple syntax:

setPref  pLabel, pValue
getPref(pLabel)

We can use those anywhere in our code, and we don't need to also pass the sub-path info with them, which allows us to make the code we write using those getPref and setPref handlers portable among any apps we make.

This requires, however, that we first provide the necessary info to the system and that this be in accessible through a globally-available function, since relying on "this stack" to obtain the info from RIP props won't work in a setup that uses any number of components, some of which may have their own RIP props.

So we've accepted the modest hit of requiring an initialization call to set up a gRIPAppinfo global array with basic info such things need, including the short name of the stack file which is the mainstack of the app in gRIPAppInfo["stack"], allowing the system to know which set of RIP props are needed for the app instead of those for subcomponents.  Once initialized, other components (like the About box, updater, etc.) can use that info just as easily as the prefs subsystem, so that such code doesn't need to know anything about the specific app they happen to be used in at the moment and can therefore be used across all of our apps.

But in doing so we crossed a line:

What was once a simple library had become a sort of framework.


With a library you just need to know the one-liner you're calling, but a framework requires that you study at least some introductory material to know how the parts fit together.

Frameworks can provide orders of magnitude more benefit, but at an order of magnitude more cost to the learning curve.  The ROI is still there, but it's no longer the dirt-simple no-brainer that a library can provide.

Consider Trevors' excellent GLX framework:  it's an amazing powerhouse of utility, yet relatively few devs have been able to wrap their head around it well enough to adopt it for daily work.   Even Rev's libraries, used in the most popular IDE and forced into all standalones built with it, are often insufficiently understood to avoid issues at runtime.  

Moverover, the more comprehensive nature of frameworks made them well suited only for new work, inhibiting their usefulness in legacy code, while 10 years after Rev's premier much of the stuff we all work on is updates to existing apps.  For all the benefits a framework can provide, the cost of their learning curve explodes with considerations of retrofitting, in many cases prohibitively so ("if it works don't fix it").  

That Apple only updated the Finder (arguably the most-used app anywhere in the Mac ecosystem) to the Cocoa framework in Snow Leopard, some 10 years after Cocoa premiered, and that Final Cut Pro still hasn't completed that transition, speaks volumes about the difficulty of retrofitting even when a new framework provides otherwise-unquestionable benefit.


Since to the best of our knowledge no one was using stdLib anyway, for the moment Ken and I have created a separate project to experiment with our more framework-like ideas, which we indend to share once they become usable.

Meanwhile, in spite of its simplicity (or perhaps because it's too simple), stdLib lies fallow, unused and unexpanded for most of the years it's been here.

I get three lessons from the stdLib experiment that may be useful as we consider other libraries for more specialized purposes:

1. Discrete handlers are much simpler to adopt, and therefore may ideally be a design goal where practical.

2. Discrete handlers are limited in scope, limiting their usefulness in spite of the ease of adoption.

3. Frameworks require a learning curve that inhibits adoption in spite of the breadth of their utility.


As we embark on the journey of making David's excellent suggestion into a living work, let's try to keep simplicity of adoption in mind, and remember also that such a library has a dual role that may compromise that simplicity if we're not careful:  it should be useful both in a standalone (where one dev controls the architecture) as well as in an IDE (where any number of devs may have written tools that are actively in use).  Ken and I have already run into this with prefs in the framework, and flexible solutions may require tradeoffs with the learning curve which risks lowering adoption.

This isn't a trivial task, but I think it's doable and well worth pursuing.

"Simplicty, simplicity!"
- H.D. Thoreau



Reply | Threaded
Open this post in threaded view
|

Re: Hubub: an alternative strategy for a "standard port registry"

david bovill
Comments below:

On 20 January 2011 14:38, richard g <[hidden email]> wrote:

>
> SOMEWHAT-RELATED BACKGROUND:
> The sad tale of stdLib, why no one uses it, and why frameworks are only
> somewhat better
> --------------------------------------------------------------------------
> This may or may not be all that relevant, but touches on issues that seem
> related so here goes:
>

I agree in general with your analysis, like you I see the problem in terms
of behavioural economics. In order for a project like stdLib to grow it must
be give back more than it cost to learn. The issue with LiveCode and the
current community is that it is so easy to create your own stuff that saving
people time by providing a library is extremely difficult.

My approach to a solution here is simply that the more traditional release
code early solution will not work, and a few integrated components need to
be in place before a general purpose set of libraries is able to take off.

Specifically:

   1. Finding a handler must be much faster than writing one. This is easier
   said than done. It needs a database with keywords, and some quality control.
   2. Documentation - not really text, but more like unit tests where you
   can see the handlers in action buy running test data against them
   3. Versioned backend that can support multiple hacking on the code base.
   If I can't hack it I ain't going to use it.

1. Discrete handlers are much simpler to adopt, and therefore may ideally be
> a design goal where practical.
>
> 2. Discrete handlers are limited in scope, limiting their usefulness in
> spite of the ease of adoption.
>
> 3. Frameworks require a learning curve that inhibits adoption in spite of
> the breadth of their utility.
>

I agree with this catch 22 - but do see a solution that flattens the
learning curve while providing the end utility of frameworks. It relies on
combining the instant hit of finding the handler you need, together with
dependency code which allows frameworks of interrelated handlers to be
linked together.

In terms of implementation, I'm working on using sqlite for the database of
metadata and keywords, with the handlers themselves exported from libraries
as text files, and versioned in GIT. In addition there is tree walking code
to pull out dependency graphs, and visualise these graphs. This allows you
to search for a handler while inside a script editor, test it out in a
debugger, and if it suits your needs pull it into the script along with any
dependencies. I think with these tools in place a distributed archive of
handlers is possible in the LiveCode community.

For this to truly work I think it also needs to be linked into the current
LiveCode documentation, and for that it will really help if RunRev release
this under an appropriate licence.

The technical bits of the puzzle I am missing or weak on are:

   1. A schema for the sqlite handler db. I believe it is important that
   this implements folksonomy or user tags
   2. Revision control workflow appropriate to LiveCode work - some testers
   to figure out how to simplify this for ease of adoption.


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Hubub: an alternative strategy for a "standard port registry"

Richard Gaskin
On 1/20/11 7:35 AM, David Bovill wrote:
>
> I agree in general with your analysis, like you I see the problem in terms
> of behavioural economics.

Excellent phrase, "behavioral economics".  I'm going to use that. :)

> In order for a project like stdLib to grow it must
> be give back more than it cost to learn. The issue with LiveCode and the
> current community is that it is so easy to create your own stuff that saving
> people time by providing a library is extremely difficult.
>
> My approach to a solution here is simply that the more traditional release
> code early solution will not work, and a few integrated components need to
> be in place before a general purpose set of libraries is able to take off.

Agreed; well and succinctly described.


> Specifically:
>
>     1. Finding a handler must be much faster than writing one. This is easier
>     said than done. It needs a database with keywords, and some quality control.
>     2. Documentation - not really text, but more like unit tests where you
>     can see the handlers in action buy running test data against them
>     3. Versioned backend that can support multiple hacking on the code base.
>     If I can't hack it I ain't going to use it.
...

> In terms of implementation, I'm working on using sqlite for the database of
> metadata and keywords, with the handlers themselves exported from libraries
> as text files, and versioned in GIT. In addition there is tree walking code
> to pull out dependency graphs, and visualise these graphs. This allows you
> to search for a handler while inside a script editor, test it out in a
> debugger, and if it suits your needs pull it into the script along with any
> dependencies. I think with these tools in place a distributed archive of
> handlers is possible in the LiveCode community.
>
> For this to truly work I think it also needs to be linked into the current
> LiveCode documentation, and for that it will really help if RunRev release
> this under an appropriate licence.
>
> The technical bits of the puzzle I am missing or weak on are:
>
>     1. A schema for the sqlite handler db. I believe it is important that
>     this implements folksonomy or user tags
>     2. Revision control workflow appropriate to LiveCode work - some testers
>     to figure out how to simplify this for ease of adoption.


This sounds ambitious, more than I could contribute to but look forward
to seeing it in place.

On that second item I believe Mark Wieder may have something in the
works which can help.

--
  Richard Gaskin
  Fourth World
  LiveCode training and consulting: http://www.fourthworld.com
  Webzine for LiveCode developers: http://www.LiveCodeJournal.com
  LiveCode Journal blog: http://LiveCodejournal.com/blog.irv