Making Revolution faster using dimensioned arrays

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Making Revolution faster using dimensioned arrays

see3d
To all the speed freaks,

I know that I have pushed for faster array processing and have even  
proposed an "Array sub-processor" as a possible solution.  However,  
after giving this much thought, I don't believe that a separate array  
sub-processor is needed to solve the problem of speed.  Transcript  
could provide the speed needed for processing arrays in a simpler way.

If Transcript had an array declaration command that allowed the user  
to fix the dimensions and data size/type for an array and limited the  
"keys" to integer indexes, then high speed array processing could be  
built-in.

Normally it would be a pain to have to dim all arrays.  Arrays in  
Transcript are sparsely populated and very general now --and could be  
further extended to make them even better for many general purpose  
solutions.  However, there is a class of problems that need speed and  
generally lend themselves to fixing the type and size.  These  
problems are quite common, but rarely seen implemented in Transcript  
because of the slow array processing speed.

The key is that once an array is fixed in size and type, the runtime  
does not need to do extensive checking of the data to determine the  
operation code, and the keys do not have to be looked up in a hash  
table to find the location of the element --just use the indexes  
directly.  The compiler can then generate the code for a much simpler  
runtime operator.  These declared arrays would run 10 to 100 times  
faster than the type-less ones for the kind of operations that are  
required by image processing, AI, statistics, and many other  
applications.  This is the main reason array processing runs so much  
faster in other languages.

Having fast array structures then opens up the possibility for a  
larger set of array operators or even an efficient way to pass the  
array pointers directly to an external.  The links between the rest  
of the data structures in Transcript and these dimensioned arrays  
become seamless.

Conceptually, the implementation of a dimensioned array would be as  
if it were a single string variable.  In other words, just a  
contiguous chunk of memory that you can index into based on the  
supplied dimensions.  We already have the concept of a pointer in  
Transcript.  When we say "get char (expression) of variable",  
"(expression)"  is a pointer.  That expression can be the index into  
a conceptual n dimensional character array.  However, at present we  
are limited to directly accessing only a single byte.  We could say  
"get char (expression) to (expression+4) of variable", to get 32 bits  
of information, but even if we know that it is supposed to represent  
a 32 bit floating point number, we have no way of telling Transcript  
that, so it will just hand us back 4 characters that could at best be  
translated into a 4 digit (including decimal point) number like  
3.14.  However, if we told the compiler that we want our variable to  
have 1000 rows by 1000 columns of  4 byte words, then the "get  
variable[25,64]" could get a 4 byte string of characters in lightning  
speed.  We could say that If we initialized our array with strings,  
we would get strings back.  If we initialized it with numbers, we  
would get numbers back --which is very Transcript like, or we could  
just specify the type.  The point is that a single dimensioned array,  
like a single variable, can hold only one data type at a time.

The advantage of such an arrangement is not only speed, but also  
compactness.  If we are interested in speed it is likely that we have  
a large quantity of data.  Arrays in Transcript today have a large  
overhead due to the key structure.  Dimensioned arrays could easily  
take half the space or even less.  The syntax could be very simple like:

global myArray[1000,1000]=0 --numeric array
local myArray[1000,1000,5]=false --Boolean array
local myArray[1000]="" --empty character array
local myArray[100,300]="abcd" --4 character word array

Dennis
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Ken Ray
On 7/4/05 9:33 PM, "Dennis Brown" <[hidden email]> wrote:

> The advantage of such an arrangement is not only speed, but also
> compactness.  If we are interested in speed it is likely that we have
> a large quantity of data.  Arrays in Transcript today have a large
> overhead due to the key structure.  Dimensioned arrays could easily
> take half the space or even less.  The syntax could be very simple like:
>
> global myArray[1000,1000]=0 --numeric array
> local myArray[1000,1000,5]=false --Boolean array
> local myArray[1000]="" --empty character array
> local myArray[100,300]="abcd" --4 character word array

I really like this idea, Dennis, but I'm not sure I like the approach to
dimensioning, only because of Transcript's history of being able to
interpret things as strings, even if not quoted. So is:

  local myArray[100,300]="false"

a boolean array, or a 5-character word array? Perhaps it's acceptable to be
more picky about this (i.e. the above would be a 5 character word array),
since we don't currently have a way of assigning a type, but I think there's
a better way. For example, using your approach, if you wanted a 25 character
word array, you'd have to provide a 25-character-long string.

It seems to me that the main thing is not the *type* of data, but for the
interpreter to know that there is a fixed size to array elements. So perhaps
something like:

  local myArray[100,300] length 5

or something like that...

Ken Ray
Sons of Thunder Software
Web site: http://www.sonsothunder.com/
Email: [hidden email]

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Jon-3
In reply to this post by see3d
"These declared arrays would run 10 to 100 times  faster than the
type-less ones for the kind of operations that are  required by image
processing, AI, statistics, and many other  applications."

Given the detailed work that Alex (?) did a week or two ago with my
algorithms, I am not at all sure that your optimism is warranted.  That
said, I love the idea <grin>

Jon


Dennis Brown wrote:

> To all the speed freaks,
>
> I know that I have pushed for faster array processing and have even  
> proposed an "Array sub-processor" as a possible solution.  However,  
> after giving this much thought, I don't believe that a separate array  
> sub-processor is needed to solve the problem of speed.  Transcript  
> could provide the speed needed for processing arrays in a simpler way.
>
> If Transcript had an array declaration command that allowed the user  
> to fix the dimensions and data size/type for an array and limited the  
> "keys" to integer indexes, then high speed array processing could be  
> built-in.
>
> Normally it would be a pain to have to dim all arrays.  Arrays in  
> Transcript are sparsely populated and very general now --and could be  
> further extended to make them even better for many general purpose  
> solutions.  However, there is a class of problems that need speed and  
> generally lend themselves to fixing the type and size.  These  
> problems are quite common, but rarely seen implemented in Transcript  
> because of the slow array processing speed.
>
> The key is that once an array is fixed in size and type, the runtime  
> does not need to do extensive checking of the data to determine the  
> operation code, and the keys do not have to be looked up in a hash  
> table to find the location of the element --just use the indexes  
> directly.  The compiler can then generate the code for a much simpler  
> runtime operator.  These declared arrays would run 10 to 100 times  
> faster than the type-less ones for the kind of operations that are  
> required by image processing, AI, statistics, and many other  
> applications.  This is the main reason array processing runs so much  
> faster in other languages.
>
> Having fast array structures then opens up the possibility for a  
> larger set of array operators or even an efficient way to pass the  
> array pointers directly to an external.  The links between the rest  
> of the data structures in Transcript and these dimensioned arrays  
> become seamless.
>
> Conceptually, the implementation of a dimensioned array would be as  
> if it were a single string variable.  In other words, just a  
> contiguous chunk of memory that you can index into based on the  
> supplied dimensions.  We already have the concept of a pointer in  
> Transcript.  When we say "get char (expression) of variable",  
> "(expression)"  is a pointer.  That expression can be the index into  
> a conceptual n dimensional character array.  However, at present we  
> are limited to directly accessing only a single byte.  We could say  
> "get char (expression) to (expression+4) of variable", to get 32 bits  
> of information, but even if we know that it is supposed to represent  
> a 32 bit floating point number, we have no way of telling Transcript  
> that, so it will just hand us back 4 characters that could at best be  
> translated into a 4 digit (including decimal point) number like  
> 3.14.  However, if we told the compiler that we want our variable to  
> have 1000 rows by 1000 columns of  4 byte words, then the "get  
> variable[25,64]" could get a 4 byte string of characters in lightning  
> speed.  We could say that If we initialized our array with strings,  
> we would get strings back.  If we initialized it with numbers, we  
> would get numbers back --which is very Transcript like, or we could  
> just specify the type.  The point is that a single dimensioned array,  
> like a single variable, can hold only one data type at a time.
>
> The advantage of such an arrangement is not only speed, but also  
> compactness.  If we are interested in speed it is likely that we have  
> a large quantity of data.  Arrays in Transcript today have a large  
> overhead due to the key structure.  Dimensioned arrays could easily  
> take half the space or even less.  The syntax could be very simple like:
>
> global myArray[1000,1000]=0 --numeric array
> local myArray[1000,1000,5]=false --Boolean array
> local myArray[1000]="" --empty character array
> local myArray[100,300]="abcd" --4 character word array
>
> Dennis
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
>
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Alex Tweedly
Dennis wrote:

> "These declared arrays would run 10 to 100 times  faster than the
> type-less ones for the kind of operations that are  required by image
> processing, AI, statistics, and many other  applications."
>
Jon wrote:

> Given the detailed work that Alex (?) did a week or two ago with my
> algorithms, I am not at all sure that your optimism is warranted.  
> That said, I love the idea <grin>


I too love the idea. I think it would extend the scope of pure-Transcipt
to cover a wide range of interesting problems. When it coalesces into a
solid Bugzilla enhancement request, it will get quite a few of my votes.

But this on its own will not make pure-Transcript feasible for image
processing tasks.

Imagine that the engine could be incredibly fast at accessing
these"dimensioned arrays". We can easily benchmark to see how long it
would take for various assumptions about how fast it *could* possibly be.

For a 6MP loop:
doing nothing take 430 ms
assigning to a scalar variable takes 1500 ms
accessing via an index (char i of var) takes 2500 ms
incrementing via an index (char i of var) takes 5200 ms
idealized version of histogram takes 10400 ms

So even if the optimization of "dimensioned arrays" achieves everything
it can hope for, and can access those arrays through a single index
look-up - it will still take 10 seconds to do a histogram for an image
of the size produced by last year's consumer cameras.

It is still, IMO, a worthwhile addition to the language, and would allow
us to do a lot of things that are beyond pure-Transcript today but are
less demanding than image processing. In some future combination with a
"compileIt" optimization (and perhaps some optimizing of "repeat with"
loops) it might stretch to cover image processing.

But honestly, image processing is a specialized area, and I think it
might well be better served by using a specialized (already written)
package, such as ImageMagick, or PIL, or .... either through an
external, a command-line or process call, or via RPC-like socket
communication.

In fact, an RPC-like mechanism would also help with the ability to take
full advantage of multi-core or multi-CPU systems. Since the near-term
(5-10 year) growth in PC performance is likely to come from multiple
processors rather than faster linear CPU cycle times, anyone doing image
processing needs to be looking at using multiple threads or multiple
processes to gain those benefits - and so an RPC-like mechanism has a
lot going for it ...

--
Alex Tweedly       http://www.tweedly.net



--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.8/37 - Release Date: 01/07/2005

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

see3d
In reply to this post by Ken Ray
On Jul 5, 2005, at 12:04 AM, Ken Ray wrote:

> I really like this idea, Dennis, but I'm not sure I like the  
> approach to
> dimensioning, only because of Transcript's history of being able to
> interpret things as strings, even if not quoted. So is:
>
>   local myArray[100,300]="false"
>
> a boolean array, or a 5-character word array? Perhaps it's  
> acceptable to be
> more picky about this (i.e. the above would be a 5 character word  
> array),
> since we don't currently have a way of assigning a type, but I  
> think there's
> a better way. For example, using your approach, if you wanted a 25  
> character
> word array, you'd have to provide a 25-character-long string.
>
> It seems to me that the main thing is not the *type* of data, but  
> for the
> interpreter to know that there is a fixed size to array elements.  
> So perhaps
> something like:
>
>   local myArray[100,300] length 5
>
> or something like that...

Ken,

Yes, that is why I wanted to start a discussion about his before  
putting in a BZ request.  There is more than one approach to the  
syntax even though the basic idea is the same.

It might be better to separate the declaration of the variable from  
the dimension statement.  Since Transcript does not care what type or  
length a variable is when it is declared, no change would actually be  
needed to declare its scope.  That way it is easier to include the  
dimension command later with non-constant arguments --which would be  
better for resizable graphics.  There is no reason that a variable  
could not be re-dimensioned more than once --however the contents  
would get reset.

I thought that it would be just as easy to specify the type implicit  
in the data value that was used to initialize the string --fixed  
dimension arrays have every element filled except in the case of a  
string array that includes a char count which could be zero.  I don't  
think the user should have to know the exact internal format of the  
data type chosen --unless he is writing an external.  If there are  
two sizes of integer formats, then perhaps they are specified a short  
or long instead of 2 byte or 4 byte.

Having a variable length string type is another fine point.  Does it  
make sense to have a variable length string type at all, since  
Transcript is already very robust in the string area.  Perhaps it is  
sufficient to have only a fixed number of chars that are padded out  
with a "blank" char.  I could go either way, but having a char count  
for a string type would be more general and it would increase the  
number of uses.  Perhaps the length is a max of 127 or 255 characters  
--only short strings would benefit from a fixed length array.

I still have a lot of questions about how different people would use  
this type of array.  Right now, I could use over half of the types  
below in just one application that does not even use any images:

local myArray

dim myArray[ imageHeight, imageWidth ] RGBpixel=0,0,0 --how many  
image pixel formats do we need?
dim myArray[ rows, columns, pages ] realNumber=0 --do we need 32 and  
64 bit numbers?
dim myArray[ rows, columns ]  short integer=0 --do we need 8,16,32  
bit numbers?  What does Rev internally support now?
dim myArray[ rows, columns ] string=20 --length initialized to 0,  
good for short variable length strings
dim myArray[ rows, columns ] chars="abcd" --always 4 bytes long, good  
for fixed format words
dim myArray[ rows, columns, map ] boolean=0 --compact one bit logic  
arrays, bit masks


Dennis
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

see3d
In reply to this post by Alex Tweedly

On Jul 5, 2005, at 9:26 AM, Alex Tweedly wrote:

> Dennis wrote:
>
>
>> "These declared arrays would run 10 to 100 times  faster than the  
>> type-less ones for the kind of operations that are  required by  
>> image processing, AI, statistics, and many other  applications."
>>
>>
> Jon wrote:
>
>
>> Given the detailed work that Alex (?) did a week or two ago with  
>> my algorithms, I am not at all sure that your optimism is  
>> warranted.  That said, I love the idea <grin>
>>
>
>
> I too love the idea. I think it would extend the scope of pure-
> Transcipt to cover a wide range of interesting problems. When it  
> coalesces into a solid Bugzilla enhancement request, it will get  
> quite a few of my votes.
>
> But this on its own will not make pure-Transcript feasible for  
> image processing tasks.
>
> Imagine that the engine could be incredibly fast at accessing  
> these"dimensioned arrays". We can easily benchmark to see how long  
> it would take for various assumptions about how fast it *could*  
> possibly be.
>
> For a 6MP loop:
> doing nothing take 430 ms
> assigning to a scalar variable takes 1500 ms
> accessing via an index (char i of var) takes 2500 ms
> incrementing via an index (char i of var) takes 5200 ms
> idealized version of histogram takes 10400 ms
>
> So even if the optimization of "dimensioned arrays" achieves  
> everything it can hope for, and can access those arrays through a  
> single index look-up - it will still take 10 seconds to do a  
> histogram for an image of the size produced by last year's consumer  
> cameras.
>
> It is still, IMO, a worthwhile addition to the language, and would  
> allow us to do a lot of things that are beyond pure-Transcript  
> today but are less demanding than image processing. In some future  
> combination with a "compileIt" optimization (and perhaps some  
> optimizing of "repeat with" loops) it might stretch to cover image  
> processing.
>
> But honestly, image processing is a specialized area, and I think  
> it might well be better served by using a specialized (already  
> written) package, such as ImageMagick, or PIL, or .... either  
> through an external, a command-line or process call, or via RPC-
> like socket communication.
>
> In fact, an RPC-like mechanism would also help with the ability to  
> take full advantage of multi-core or multi-CPU systems. Since the  
> near-term (5-10 year) growth in PC performance is likely to come  
> from multiple processors rather than faster linear CPU cycle times,  
> anyone doing image processing needs to be looking at using multiple  
> threads or multiple processes to gain those benefits - and so an  
> RPC-like mechanism has a lot going for it ...

Alex, Jon,

I completely agree that this would not make Transcript into a  
PhotoShop application.  The image processing I was referring to was  
more along the lines of image manipulation that could be produced 5  
years ago by a "C" program.  In other words, Transcript is lagging  
good compiled languages by two orders of magnitude in array  
processing (not including dedicated vector processors).  That just  
means that we have to look backwards in time to what could be done  
when CPUs ran at 20 MHz.  I would like to see us get up to the 200  
MHz era for general stuff.  Specialized native commands like a  
package of array operators that approximate the capabilities of APL  
would of course improve the performance significantly, but for now I  
would just like to get over the hump of having the base functionality  
that could support better things in the future.

Having this fixed format might also make it possible to have an  
efficient way to call an external with a pointer to the array to add  
a lot of missing functionality --but this is outside my experience.

Dennis

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Ken Ray
In reply to this post by see3d

> Having a variable length string type is another fine point.  Does it
> make sense to have a variable length string type at all, since
> Transcript is already very robust in the string area.  Perhaps it is
> sufficient to have only a fixed number of chars that are padded out
> with a "blank" char.  I could go either way, but having a char count
> for a string type would be more general and it would increase the
> number of uses.  Perhaps the length is a max of 127 or 255 characters
> --only short strings would benefit from a fixed length array.

Good questions - I think Mark Waddingham's going to have to weigh in here to
let us know exactly how Rev stores/manages arrays for us to go too much
farther other than to declare dimensioning syntax.

Ken Ray
Sons of Thunder Software
Web site: http://www.sonsothunder.com/
Email: [hidden email]

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Marielle Lange
In reply to this post by see3d
Hi Dennis,

Sorry, I am not in favour of this if this means that I would loose the
possibility to index arrays with text or to have to specify the array size in
advance.

Personally, I need to index with text parts that I don't know in advance what
the size of the array will be. With this type of constraint, the solution you
suggest in fact considerably slows down the programs I write. My experience
with a program doing exactly the same thing: Visual Basic 5 hours (which does
*exactly* what you request, Awk 10 min which does *exactly* what you complain
about.

Gawk does it very fast without the need for any of these two options.
Look at this link, for evidence:  http://www.cs.wustl.edu/~loui/sigplan
"Two pearls in GAWK:  its regular expressions and its associative arrays."

By chance, could SQLlite help to speed up your array processing? (after all, a
database with two columns of data is just another representation for an array
and you can apply a formula to transform the values of one column)

Marielle


To all the speed freaks,

I know that I have pushed for faster array processing and have even
proposed an "Array sub-processor" as a possible solution.  However,
after giving this much thought, I don't believe that a separate array
sub-processor is needed to solve the problem of speed.  Transcript
could provide the speed needed for processing arrays in a simpler way.

If Transcript had an array declaration command that allowed the user
to fix the dimensions and data size/type for an array and limited the
"keys" to integer indexes, then high speed array processing could be
built-in.
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Jon-3
Marielle:

I did not interpret Dennis' suggestion as meaning that ALL arrays would
henceforth ONLY have integer indices; rather, I think he was suggesting
that a NEW type of array be created which had this attribute.  All of
the old code would still work.

:)

Jon


Marielle Lange wrote:

>Hi Dennis,
>
>Sorry, I am not in favour of this if this means that I would loose the
>possibility to index arrays with text or to have to specify the array size in
>advance.
>
>Personally, I need to index with text parts that I don't know in advance what
>the size of the array will be. With this type of constraint, the solution you
>suggest in fact considerably slows down the programs I write. My experience
>with a program doing exactly the same thing: Visual Basic 5 hours (which does
>*exactly* what you request, Awk 10 min which does *exactly* what you complain
>about.
>
>Gawk does it very fast without the need for any of these two options.
>Look at this link, for evidence:  http://www.cs.wustl.edu/~loui/sigplan
>"Two pearls in GAWK:  its regular expressions and its associative arrays."
>
>By chance, could SQLlite help to speed up your array processing? (after all, a
>database with two columns of data is just another representation for an array
>and you can apply a formula to transform the values of one column)
>
>Marielle
>
>
>To all the speed freaks,
>
>I know that I have pushed for faster array processing and have even
>proposed an "Array sub-processor" as a possible solution.  However,
>after giving this much thought, I don't believe that a separate array
>sub-processor is needed to solve the problem of speed.  Transcript
>could provide the speed needed for processing arrays in a simpler way.
>
>If Transcript had an array declaration command that allowed the user
>to fix the dimensions and data size/type for an array and limited the
>"keys" to integer indexes, then high speed array processing could be
>built-in.
>_______________________________________________
>use-revolution mailing list
>[hidden email]
>Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
>http://lists.runrev.com/mailman/listinfo/use-revolution
>
>
>  
>
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Dan Shafer
And if we had an array OBJECT, we could subclass it to get what we  
really want.

Ahem.

Dan
On Jul 7, 2005, at 4:26 PM, Jon wrote:

> Marielle:
>
> I did not interpret Dennis' suggestion as meaning that ALL arrays  
> would henceforth ONLY have integer indices; rather, I think he was  
> suggesting that a NEW type of array be created which had this  
> attribute.  All of the old code would still work.
>
> :)
>
> Jon
>
>
> Marielle Lange wrote:
>
>
>> Hi Dennis,
>>
>> Sorry, I am not in favour of this if this means that I would loose  
>> the
>> possibility to index arrays with text or to have to specify the  
>> array size in
>> advance.
>>
>> Personally, I need to index with text parts that I don't know in  
>> advance what
>> the size of the array will be. With this type of constraint, the  
>> solution you
>> suggest in fact considerably slows down the programs I write. My  
>> experience
>> with a program doing exactly the same thing: Visual Basic 5 hours  
>> (which does
>> *exactly* what you request, Awk 10 min which does *exactly* what  
>> you complain
>> about.
>>
>> Gawk does it very fast without the need for any of these two options.
>> Look at this link, for evidence:  http://www.cs.wustl.edu/~loui/ 
>> sigplan
>> "Two pearls in GAWK:  its regular expressions and its associative  
>> arrays."
>>
>> By chance, could SQLlite help to speed up your array processing?  
>> (after all, a
>> database with two columns of data is just another representation  
>> for an array
>> and you can apply a formula to transform the values of one column)
>>
>> Marielle
>>
>>
>> To all the speed freaks,
>>
>> I know that I have pushed for faster array processing and have even
>> proposed an "Array sub-processor" as a possible solution.  However,
>> after giving this much thought, I don't believe that a separate array
>> sub-processor is needed to solve the problem of speed.  Transcript
>> could provide the speed needed for processing arrays in a simpler  
>> way.
>>
>> If Transcript had an array declaration command that allowed the user
>> to fix the dimensions and data size/type for an array and limited the
>> "keys" to integer indexes, then high speed array processing could be
>> built-in.
>> _______________________________________________
>> use-revolution mailing list
>> [hidden email]
>> Please visit this url to subscribe, unsubscribe and manage your  
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-revolution
>>
>>
>>
>>
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
>

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Richard Gaskin
Dan Shafer wrote:
> And if we had an array OBJECT, we could subclass it to get what we  
> really want.

What would such code look like?

--
  Richard Gaskin
  Fourth World Media Corporation
  ___________________________________________________________
  [hidden email]       http://www.FourthWorld.com
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

see3d
In reply to this post by Marielle Lange

On Jul 7, 2005, at 7:19 PM, Marielle Lange wrote:

> Hi Dennis,
>
> Sorry, I am not in favour of this if this means that I would loose the
> possibility to index arrays with text or to have to specify the  
> array size in
> advance.

Marielle,

I also use the current array capabilities and do not want to lose  
anything we have.  In fact I am in favor of expanding them further.  
What I propose is a different array organization that is optimized  
for execution speed and space efficiency at the cost of a more  
restricted organizational flexibility.  The restrictions would only  
occur if you wanted to make this tradeoff for a particular array.  
There are many problems that would benefit from this, but not all.  
Mathematical problems would benefit from the packed arrays I propose,  
while many indexing and database like problems benefit from the  
sparse arrays we now have.

Dennis
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

see3d
In reply to this post by Dan Shafer
On Jul 7, 2005, at 7:49 PM, Dan Shafer wrote:

> And if we had an array OBJECT, we could subclass it to get what we  
> really want.

Dan,

This is an interesting concept.  Would an array object have a  
spreadsheet as its visual representation?

Dennis
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

M Young
In reply to this post by see3d
Dennis,

I am not Dan, but I would like to answer because I would love to have
array objects also.

I would say sort of yes to spreadsheet representation although I am not
exactly sure how the physical representation would look beyond two
dimensions.

I would suggest the array as a "container" of more fundamental
elements: field, button, etc.

I have seen another product use the concept of array for an array all
of one element type and cluster for an array of multiple element types.

Array object visual representations (might be simpler with attachments
sent to list):

One dimensional array:
Index in a box. Index can be hidden.
Element is in another box structure that can be resized as needed.
[ index ]| element 1     |
               | element 2     |
               | element 3     |
               | element etc. |
or

[ index ] | element 1     | element 2     | element etc. |

Two dimensional array:
[ index 1 ] | element 1,1 | element 2,1 | element etc. |
[ index 2 ] | element 2,1 | element 2,2 | element etc. |
                   | element 3,1 | element 3,2 | element etc. |

I hopes this post makes some sense. :-)

Michael

On Jul 7, 2005, at 8:38 PM, [hidden email]
wrote:

>
> On Jul 7, 2005, at 7:49 PM, Dan Shafer wrote:
>
>> And if we had an array OBJECT, we could subclass it to get what we
>> really want.
>
> Dan,
>
> This is an interesting concept.  Would an array object have a
> spreadsheet as its visual representation?
>
> Dennis
>
>

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Dan Shafer
In reply to this post by see3d
An array object could have any number of methods to draw itself and  
any number of methods to calculate itself. I imagine a spreadsheet  
would be one (perhaps the favored) visual representation. But some  
arrays don't need a visual presence.

SO we might have an array object that does pretty much what the  
current one does and a fastArray or an integerIndexedArray object  
that sub-classes array and defines properties for indexing that  
override the default and provides some method of calculation in an  
override as well.

Not trying to be an architect here, just thinking out loud.

Dan

On Jul 7, 2005, at 5:45 PM, Dennis Brown wrote:

> On Jul 7, 2005, at 7:49 PM, Dan Shafer wrote:
>
>
>> And if we had an array OBJECT, we could subclass it to get what we  
>> really want.
>>
>
> Dan,
>
> This is an interesting concept.  Would an array object have a  
> spreadsheet as its visual representation?
>
> Dennis
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
>

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Marielle Lange
In reply to this post by see3d

No worries, Jon... I didn't either interpret Dennis suggestion as converting
revolution to c++ ;-) (I appreciate the value of his comments -- I even
recently voted for Dennis' last suggestion on stack scope variable).

I just wanted to voice the other perspective. Adding new array declarations and
new array manipulations features, I vote for. This would indeed be most useful
for image processing. However, revolution is primarily a 4GL and a 4GL (high
level programming language) as inherent limitations. With a 4GL, you loose in
sheer speed in terms of data crunching. But you win in speed in ease of use and
data manipulation of reasonably-sized projects. Add-ons, externals, new
functions in revolution, all that is welcome... provided that there is never an
attempt to go down from 4GL to 3GL or less.  I prefer a program that is slow
when asked to manipulate 1000x1000 matrices than a program that is slow and
dumb in all other tasks. Back to the idea that not single tool is best for all
jobs.

------------------

I did not interpret Dennis' suggestion as meaning that ALL arrays would
henceforth ONLY have integer indices; rather, I think he was suggesting
that a NEW type of array be created which had this attribute.  All of
the old code would still work.

:)

Jon
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

Marielle Lange
In reply to this post by see3d
>I also use the current array capabilities and do not want to lose
>anything we have.  In fact I am in favor of expanding them further.
>What I propose is a different array organization that is optimized
>for execution speed and space efficiency at the cost of a more
>restricted organizational flexibility.  The restrictions would only
>occur if you wanted to make this tradeoff for a particular array.
> And if we had an array OBJECT, we could subclass it to get what we
> really want.

I reacted on the fact you proposed a c/c++ solution....

>This is an interesting concept.  Would an array object have a
>spreadsheet as its visual representation?

When I pointed to SQL, I was hinting to the same idea. But I am only a hobbyist
programmer, I can only stir a debate and let others express things better than
I can ;-).

Marielle


Dennis

I would be interested to know if in you timing test, you also tried using a
variable rather than an array.

repeat for x in (trows)
 repeat for y in (tcolumns)
    do something on (variable & "_" & x & "_" & y).
 end repeat
end repeat

+ what if you store all rows in one array rowdata[1] and all columns in another
one columndata[1]. You then reduce the dimensionality and perhaps access time.

Marielle
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Making Revolution faster using dimensioned arrays

see3d

On Jul 8, 2005, at 9:09 AM, Marielle Lange wrote:

> Dennis
>
> I would be interested to know if in you timing test, you also tried  
> using a
> variable rather than an array.
>
> repeat for x in (trows)
>  repeat for y in (tcolumns)
>     do something on (variable & "_" & x & "_" & y).
>  end repeat
> end repeat

Marielle,

The do operator would be very slow, as it has to compile the line,  
then execute it.

+ what if you store all rows in one array rowdata[1] and all columns  
in another
one columndata[1]. You then reduce the dimensionality and perhaps  
access time.

Yes, I have organized data both ways at times to get the best access  
method

I do try to limit the number of accesses as much as possible.  
Whenever possible I use the form:

repeat for each line theLine in textArray
   repeat for each item theItem in theLine
     process theItem
   end repeat
end repeat

which is about as fast as you can get, but many times I need to index  
into the data so I might use:

split textArray by return
repeat with i-2 to 1000 step 5
   put textArray[i] into itemArray
   split itemArray by comma
   repeat with j=2 to 1000 step 3
     process itemArray[j]
   end repeat
end repeat

I might also mix the two forms to get the most speed.
I try to make the most of what Rev has to offer today in how I  
organize my data.  However, that requires a lot more thought, and you  
have to understand what is fast and what is slow in Rev.  It is just  
not the case that you can solve the problem in the most straight  
forward way --if you care about how long it takes to get the answer.  
For simple things, it usually does not matter, but for complex  
applications, the speed is important.  The crazy thing is that with  
Rev, doing more complex applications is more practical for the  
amateur programmer, but the results are unsatisfactory due to the  
speed of execution.  There is no reason in my mind that precludes  
having both ease of use and reasonably fast speed for the speed  
critical portions of an application.

Dennis
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution