valueDiff for arrays?

classic Classic list List threaded Threaded
135 messages Options
1234 ... 7
Reply | Threaded
Open this post in threaded view
|

valueDiff for arrays?

Knapp Martin via use-livecode
We have a command for obtaining the difference between arrays, where
"difference" is defined as having different keys.

What I could use at the moment is a means of obtaining an array which is
the difference is both keys present in one but not in the other, and
values that are different for those elements where they keys are present
in both.

I can write a loop for that, but it seems like it would be useful for
others so I thought I'd ask if anyone here knows of a feature request in
the DB for this.  If not I'll add one.

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
Not to beat a dead horse, but this would be a perfect application for moving the arrays to memory databases, and running queries against them, then converting back to arrays again. I can have a go at it if I have a couple JSON arrays to work with. Are  they typical 2 dimensional arrays like datagrids use, or are there embedded arrays?

Bob S

> On Aug 3, 2018, at 10:40 , Richard Gaskin via use-livecode <[hidden email]> wrote:
>
> We have a command for obtaining the difference between arrays, where "difference" is defined as having different keys.
>
> What I could use at the moment is a means of obtaining an array which is the difference is both keys present in one but not in the other, and values that are different for those elements where they keys are present in both.
>
> I can write a loop for that, but it seems like it would be useful for others so I thought I'd ask if anyone here knows of a feature request in the DB for this.  If not I'll add one.
>
> --
> Richard Gaskin


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On 08/03/2018 10:40 AM, Richard Gaskin via use-livecode wrote:
> We have a command for obtaining the difference between arrays, where
> "difference" is defined as having different keys.
>
> What I could use at the moment is a means of obtaining an array which is
> the difference is both keys present in one but not in the other, and
> values that are different for those elements where they keys are present
> in both.

Considering how easy the first part is to code, I doubt a feature
request would get a second glance.

function arrayDifferences pArray1, pArray2
    local tDiff1, tDiff2, tDiffs

    difference pArray1 with pArray2 into tDiff1
    difference pArray2 with pArray1 into tDiff2
    union tDiff1 with tDiff2 into tDiffs
    return tDiffs
end arrayDifferences

For the second part, I'm not sure what the output would look like... do
you want an array with duplicate keys? A multidimensional array with the
same keys in each but different values? What's the use case?

--
  Mark Wieder
  [hidden email]

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
"symmetric difference" will get you the first part (unique keys)

You could shorten the compare loop by using "intersect" (to get common keys)


function getKeys pLeft, pRight
   local tIntersect, tResult

   intersect pLeft with pRight into tIntersect
   repeat for each key tKey in tIntersect
      if pLeft[tKey] is not pRight[tKey] then
         delete variable pRight[tKey]
      end if
   end repeat

   symmetric difference pLeft with pRight into tResult
   return the keys of tResult
end getKeys

On Fri, Aug 3, 2018 at 5:54 PM, Mark Wieder via use-livecode <
[hidden email]> wrote:

> On 08/03/2018 10:40 AM, Richard Gaskin via use-livecode wrote:
>
>> We have a command for obtaining the difference between arrays, where
>> "difference" is defined as having different keys.
>>
>> What I could use at the moment is a means of obtaining an array which is
>> the difference is both keys present in one but not in the other, and values
>> that are different for those elements where they keys are present in both.
>>
>
> Considering how easy the first part is to code, I doubt a feature request
> would get a second glance.
>
> function arrayDifferences pArray1, pArray2
>    local tDiff1, tDiff2, tDiffs
>
>    difference pArray1 with pArray2 into tDiff1
>    difference pArray2 with pArray1 into tDiff2
>    union tDiff1 with tDiff2 into tDiffs
>    return tDiffs
> end arrayDifferences
>
> For the second part, I'm not sure what the output would look like... do
> you want an array with duplicate keys? A multidimensional array with the
> same keys in each but different values? What's the use case?
>
> --
>  Mark Wieder
>  [hidden email]
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
(As Mark said, if the values are different, which value would you want in
the array?)
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On 08/03/2018 04:26 PM, Brian Milby via use-livecode wrote:
> "symmetric difference" will get you the first part (unique keys)

Nice. One-stop shopping.

>
> You could shorten the compare loop by using "intersect" (to get common keys)
>
>
> function getKeys pLeft, pRight
>     local tIntersect, tResult
>
>     intersect pLeft with pRight into tIntersect
>     repeat for each key tKey in tIntersect
>        if pLeft[tKey] is not pRight[tKey] then
>           delete variable pRight[tKey]
>        end if
>     end repeat
>
>     symmetric difference pLeft with pRight into tResult
>     return the keys of tResult
> end getKeys

I think Richard was aiming for (if I can put words in his mouth here)
the set of just the elements with duplicate keys but different values.
Here's my take on that, returning an array with two subarrays:

function getKeys pLeft, pRight
    local tIntersect, tResult

    intersect pLeft with pRight into tIntersect
    repeat for each key tKey in tIntersect
       if pLeft[tKey] is pRight[tKey] then
          delete variable pRight[tKey]
       end if
    end repeat
    put tIntersect into tResult["left"]
    intersect pRight with tIntersect into tResult["right"]
    return tResult
end getKeys


--
  Mark Wieder
  [hidden email]

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
I was kind of thinking along those lines too - if the values were wanted.

Looking at "MCArraysDoUnion" in exec-array.cpp, it would be pretty simple
to add code that would do a modified union.  The only thing removed would
be matching keys with matching values.  That is only if you want to keep
the pLeft values though.  If you need to preserve both values or indicate
which keys had differing values, then it is more complicated.  The code to
add the syntax and other stuff isn't something I know off hand, but
wouldn't be too difficult.

Of course, then we have to figure out what to name the beast too...

On Fri, Aug 3, 2018 at 6:54 PM, Mark Wieder via use-livecode <
[hidden email]> wrote:

> On 08/03/2018 04:26 PM, Brian Milby via use-livecode wrote:
>
>> "symmetric difference" will get you the first part (unique keys)
>>
>
> Nice. One-stop shopping.
>
>
>> You could shorten the compare loop by using "intersect" (to get common
>> keys)
>>
>>
>> function getKeys pLeft, pRight
>>     local tIntersect, tResult
>>
>>     intersect pLeft with pRight into tIntersect
>>     repeat for each key tKey in tIntersect
>>        if pLeft[tKey] is not pRight[tKey] then
>>           delete variable pRight[tKey]
>>        end if
>>     end repeat
>>
>>     symmetric difference pLeft with pRight into tResult
>>     return the keys of tResult
>> end getKeys
>>
>
> I think Richard was aiming for (if I can put words in his mouth here) the
> set of just the elements with duplicate keys but different values. Here's
> my take on that, returning an array with two subarrays:
>
> function getKeys pLeft, pRight
>    local tIntersect, tResult
>
>    intersect pLeft with pRight into tIntersect
>    repeat for each key tKey in tIntersect
>       if pLeft[tKey] is pRight[tKey] then
>          delete variable pRight[tKey]
>       end if
>    end repeat
>    put tIntersect into tResult["left"]
>    intersect pRight with tIntersect into tResult["right"]
>    return tResult
> end getKeys
>
>
>
> --
>  Mark Wieder
>  [hidden email]
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
On 08/03/2018 05:16 PM, Brian Milby via use-livecode wrote:

> Of course, then we have to figure out what to name the beast too...

LOL.

But again, considering how easy it is to code this in LCS and how many
different use cases there might be to take care of, I doubt it's worth
fiddling with the engine code for this.

--
  Mark Wieder
  [hidden email]

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
Other than the fact that coding is fun :)

On Fri, Aug 3, 2018 at 8:15 PM, Mark Wieder via use-livecode <
[hidden email]> wrote:

> On 08/03/2018 05:16 PM, Brian Milby via use-livecode wrote:
>
> Of course, then we have to figure out what to name the beast too...
>>
>
> LOL.
>
> But again, considering how easy it is to code this in LCS and how many
> different use cases there might be to take care of, I doubt it's worth
> fiddling with the engine code for this.
>
>
> --
>  Mark Wieder
>  [hidden email]
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
On 2018-08-04 04:00, Brian Milby via use-livecode wrote:
> Other than the fact that coding is fun :)

Yes - but coding in LCS is much more fun, quicker (for me, anyway - and
I've been writing C/C++ for nigh-on 30 years) - more importantly it is
also MUCH MUCH more secure.

Additionally, it is also the *one* language that everyone on this list
understands and comprehends. and more secure than in C++

In general coding things in C++ in the engine should be strictly
reserved for things where it is *absolutely* required - and in this
case, I don't see an absolute requirement in this case (yet!).

This would be fundamental operations, and the interconnection with
things where doing it in LCS or LCB is not feasible or cannot produce
the fidelity required to do it.

All array operations in LiveCode (union, intersect etc.) can all be
written precisely (as in replicating exact functionality of the engine)
in LCS. Indeed, I'm pretty sure there are comments lurking around either
in tests, or in the engine source which show the LCS equivalent. In this
case, the C++ implementation is merely a 'hand-coded' to-C++ compilation
of the an LCS handler for all intents and purposes (and I'm not even
sure there is much performance advantage to it being so - perhaps an
interesting project to determine, for those that are interested in such
things!).

My suggestion here would be *first* write an LCS handler which does what
you want. Then rewrite it, trying to distill it down so it does a single
orthogonal operation (it might be what you actually want is a
composition of two or more). Get a 'perfect' LCS implementation written
which uses the engine's various 'internal' features (names,
copy-on-write) etc. to make sure it is as efficient as it can be
(centered around the use of lists / arrays - i.e. actual individual
values - rather than string-lists).

This will give rise to a name of the operation, or fundamental
operations it is composed of; the name(s) of which will lead to
potential syntax for it/them... For example, there is zero semantic
(i.e. runtime effect) difference between:

   arrayUnion @xArray, pOtherArray

and

   union xArray with pOtherArray

The difference is purely in our brains language-processing centers - the
abstract operation is the same.

As I've always said in the past, english-like syntax is like sugar (when
used as a condiment) - it makes many things easier to ingest (there are
many things which many people would not even think about wanting to
eat/drink without it), but is something you add after something exists,
not before it does.

On a related note, I get the feeling I might have irked Mr Wieder the
other day with regards the android permission thing - that wasn't my
intent - but it is a similar situation. Android and iOS permissions are
actually really quite different - yes they have the same user-level
effect - but internally iOS's are essentially limited and hard-coded,
Android's are not. If we had 'raced' to implement a uniform 'mobile'
abstraction form for them - then it would have most likely have been
wrong or at the very least very limited. Instead, we now have a simple
set of Android handlers (3 IIRC) which mean that an LC application can
utilise the full power (and it is quite extensive!) of the Android
permissions usage APIs - as if you were writing in Java (but a little
bit easier to grok!).

The 'mobileRequestPermission' handler which would be nice - can be done
in an LCS library - where it is easy to change / evolve and get right.
Indeed, people can write their own variants should they have the need
and then feed back if they want to.

If we had raced straight to C++ this wouldn't have been the case. You
all would have been stuck with what the people who poke around in the
engine had come up with and their understanding of that particular part
of the Android elephant and any future changes to them would have been
glacial.

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
On 08/04/2018 08:57 AM, Mark Waddingham via use-livecode wrote:

> On a related note, I get the feeling I might have irked Mr Wieder the
> other day with regards the android permission thing - that wasn't my

No, not irked, and sorry if I gave that impression.
My point was simply that the way I would approach this is (pseudocode
follows)

function mobilePermissions
   switch platform
     case "iOS"
       return iosPermissions
       break
     case "Android"
       return androidPermissions
       break
   end switch
end mobilePermissions

private function iosPermissions
   return_the_array_of_ios_permissions
end iosPermissions

private function androidPermissions
   throw "not yet implemented"
end androidPermissions

...document the fact that mobilePermissions is available *only* on iOS
for now, and then flesh out the Android function later on. My thinking
being that developers could use the mobileXXX functions when developing
for iOS and then not have to change their code for Android down the line.

--
  Mark Wieder
  [hidden email]

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
Mark Waddingham wrote:

 > In general coding things in C++ in the engine should be strictly
 > reserved for things where it is *absolutely* required - and in this
 > case, I don't see an absolute requirement in this case (yet!).
 >
 > This would be fundamental operations, and the interconnection with
 > things where doing it in LCS or LCB is not feasible or cannot produce
 > the fidelity required to do it.

Agreed.  This one's not at all hard to write, just floated the idea to
see if others saw it suitable to add to the "convenience" functions
already in the language, such as:

 > All array operations in LiveCode (union, intersect etc.) can all be
 > written precisely (as in replicating exact functionality of the
 > engine) in LCS.

LC has a few things like that. It was nice to see "difference" added to
that suite fairly recently.

Adding more such commands to operate on values instead of just keys
would be handy, but not at all necessary.  There are more pressing
concerns that only the engine team can implement, and scripting in
LiveCode is something every LiveCode scripter can do.

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On 2018-08-04 18:25, Mark Wieder via use-livecode wrote:

> On 08/04/2018 08:57 AM, Mark Waddingham via use-livecode wrote:
>
>> On a related note, I get the feeling I might have irked Mr Wieder the
>> other day with regards the android permission thing - that wasn't my
>
> No, not irked, and sorry if I gave that impression.
> My point was simply that the way I would approach this is (pseudocode
> follows)
>
> function mobilePermissions
>   switch platform
>     case "iOS"
>       return iosPermissions
>       break
>     case "Android"
>       return androidPermissions
>       break
>   end switch
> end mobilePermissions
>
> private function iosPermissions
>   return_the_array_of_ios_permissions
> end iosPermissions
>
> private function androidPermissions
>   throw "not yet implemented"
> end androidPermissions
>
> ...document the fact that mobilePermissions is available *only* on iOS
> for now, and then flesh out the Android function later on. My thinking
> being that developers could use the mobileXXX functions when
> developing for iOS and then not have to change their code for Android
> down the line.

Heh - well your pseudo-code just essentially made my point for me - why
does any of that need to be in the engine, if the engine provides the
underlying wrappers around the OS functionality (as it is on the
specific OS) to be able to write them? :)

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On Sat, Aug 4, 2018 at 10:57 AM, Mark Waddingham via use-livecode <
[hidden email]> wrote:

> On 2018-08-04 04:00, Brian Milby via use-livecode wrote:
>
>> Other than the fact that coding is fun :)
>>
>
> Yes - but coding in LCS is much more fun, quicker (for me, anyway - and
> I've been writing C/C++ for nigh-on 30 years) - more importantly it is also
> MUCH MUCH more secure.
>

@Mark,
I wasn't necessarily suggesting that it belonged in the engine, but
pointing out why I would look at the code.

I will say from some other things that I've looked at recently that speed
would be so much faster in the engine - if it was a feature that merited
being there.  I did kind of cheat in that I looked at how the engine did
the array union/intersect functions.  The source actually contains LCS
versions of the logic too.  What I'm thinking is that there will be an
array size where iterating the array 3 times (twice by the engine-
intersect and symmetric difference or union) will be faster than iterating
the array once in LCS (using modified union logic).

All of the current engine array code is based on comparing the key.  The
distinction here is that a value comparison is also required.

Thanks,
Brian
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
On 2018-08-04 18:37, Brian Milby via use-livecode wrote:
> @Mark,
> I wasn't necessarily suggesting that it belonged in the engine, but
> pointing out why I would look at the code.

:)

Understanding how the engine actually operates is important, because it
can help to design the LCS implementation to work with its
implementation rather than against it.

> I will say from some other things that I've looked at recently that
> speed
> would be so much faster in the engine - if it was a feature that
> merited
> being there.  I did kind of cheat in that I looked at how the engine
> did
> the array union/intersect functions.  The source actually contains LCS
> versions of the logic too.  What I'm thinking is that there will be an
> array size where iterating the array 3 times (twice by the engine-
> intersect and symmetric difference or union) will be faster than
> iterating
> the array once in LCS (using modified union logic).

Yes - which is why digging into the engine code and experimenting to
produce a good higher-level LCS implementation is the right approach.

Adding variant codepaths based on size of arrays and such is probably
inappropriate in the engine - because it will likely always be dependent
on the data set. (If you can prove that it is independent of dataset -
or is almost always what you might say is the amortized best-case - like
qsort - which is actually O(n^2) worst case, its just that you have to
work pretty darn hard to produce its worst-case behavior - then it makes
sense to do so).

Sometimes optimizations will be independent of all of that though. For
example:

All array operations are currently X, Y -> Z - i.e. not in-place. In the
cases where (for example):

   intersect X with Y

Is operating on an X which can safely be mutated directly - there is a
much more efficient implementation which does not require creating a new
array (i.e. you just loop through and delete the keys in X which aren't
in Y).

The reason the current implementations are in-place is actually due to
the syntax dispatch - that needs to be able to detect the 'correct'
situation where in-place can be used and then use it.

> All of the current engine array code is based on comparing the key.  
> The
> distinction here is that a value comparison is also required.

Indeed - which is why I wonder whether there are more fundamental
operations which do a basic 'key-ish' thing which would be appropriate
for being in the engine - as then any variation on what 'valueDiff'
might mean could be coded in LCS with little overhead compared to a
hard-coded C++ version.

i.e. I can certainly say that a patch which makes in-place array
operations work (in a way which doesn't cause a ripple effect elsewhere
in terms of locality and such) would be accepted.

I can also certainly forsee there being a variation on the
union/intersect etc. commands which are 'general' operations taking into
the key that would also make sense to be in the engine... However, I'm
not sure quite what that/they would be - but would be most interested to
see proposals / examples of (and implementations too) :)

So in no way was I intending to discourage you from your efforts - just
trying to focus on trying to find where the line is between engine/LCS
in terms of interface - that's the hardest part. At the end of the day,
the actual coding in one language or another when you know what you are
meaning to achieve is (communally) quite trivial.

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On 2018-08-04 18:33, Richard Gaskin via use-livecode wrote:
> Adding more such commands to operate on values instead of just keys
> would be handy, but not at all necessary.  There are more pressing
> concerns that only the engine team can implement, and scripting in
> LiveCode is something every LiveCode scripter can do.

Yes - so come up with LCS handlers which do that, share them, let's work
together to make them as efficient in LCS as possible and then see what
we can do in the engine to improve their performance (which I'm guessing
is as much the purpose for the proposition as the syntax).

If they show up more fundamental operations which they can be built out
of which can be made exceptionally efficient in C++ (because it has
greater fidelity at the bit/pointer/memory level), then we add those and
then the higher-level operations that use them remain in LCS.

The only difference between value manipulation operations in the engine
and in LCS is (at the end of day) the syntax, and the overhead of
maintenance. Maintenance of every single line of C++ code is at least an
order of magnitude mode expensive (in both time, man power and money)
than an equivalent line in LCS (otherwise we'd all be coding in C++,
wouldn't we?).

So - first - if you have an implementation of valueDiff - share it. It
means everyone can see directly in code, what your English description
actually means in effect. Then it means everyone can look to see how
efficient it can be made in LCS - and, also to see if it can be broken
down into more vastly more re-usable fundamental operations which allows
us all to build even more utility functions.

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
This caught my attention. The increased security is great, and I've heard
it said before, but would love to hear the reasons that make it more secure
than lower level code. That way I'd have a knowledgeable reason to be smug.

--
Jacqueline Landman Gay | [hidden email]
HyperActive Software | http://www.hyperactivesw.com
On August 4, 2018 10:59:23 AM Mark Waddingham via use-livecode
<[hidden email]> wrote:
>
> Yes - but coding in LCS is much more fun, quicker (for me, anyway - and
> I've been writing C/C++ for nigh-on 30 years) - more importantly it is
> also MUCH MUCH more secure.




_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On Sat, Aug 4, 2018 at 11:33 AM, Richard Gaskin via use-livecode <
[hidden email]> wrote:

> This one's not at all hard to write, just floated the idea to see if
> others saw it suitable to add to the "convenience" functions already in the
> language, such as:
>
>
@Richard,

What was your intent on the return value?  Did you want an array or the
keys.  If an array, which value for the differences (Left, Right,
Both-how?)  I'd like to compare how I solve it to how you did.

Thanks,
Brian
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On 2018-08-04 19:19, J. Landman Gay via use-livecode wrote:
> This caught my attention. The increased security is great, and I've
> heard it said before, but would love to hear the reasons that make it
> more secure than lower level code. That way I'd have a knowledgeable
> reason to be smug.

It is impossible to write LCS code which *just* manipulates values which
will crash/vulnerability because of the code you have written in LCS
*unless* there is actually a crash/vulnerability bug in the engine in
one of the core operations which you are using.

If you translate an operation to C++, though - it is far far easier to
introduce crashes and thus vulnerabilities - this is because C++ does
not insulate you from the fundamentals of memory management, pointers or
all that stuff which comes with writing in a language which is (in
reality) only a short step away from machine code.

Admittedly, if you are using C++ as modern C++ then it is also quite
hard to introduce such things (assuming you have built the abstractions
using templates and C++'s typing system to do so correctly and use them
uniformly and correctly) - but you still *can* write code that does as
you can always by-pass C++'s typing and such - also the engine is not
modern C++ - it is C which was transformed in C++ and due to various
requirements on usage (at the lower-level) of the engine - many APIs
used internally in the engine (e.g. libfoundation) are actually C APIs.

For example. Here is the core engine code for array intersection (which
actually covers intersection, recursive intersection and difference):

static void MCArraysDoIntersect(MCExecContext& ctxt, MCValueRef p_dst,
MCValueRef p_src, MCArrayDoIntersectOp p_op, MCValueRef& r_result)
{
     if (!MCValueIsArray(p_dst))
     {
         r_result = MCValueRetain(p_dst);
         return;

     }

     if (!MCValueIsArray(p_src))
     {
         if (p_op != kMCArrayDoIntersectOpDifference)
         {
             r_result = MCValueRetain(kMCEmptyString);
         }
         else
         {
             r_result = MCValueRetain(p_dst);
         }
         return;
     }

     MCArrayRef t_dst_array;
     t_dst_array = (MCArrayRef)p_dst;

     MCArrayRef t_src_array;
     t_src_array = (MCArrayRef)p_src;

     MCAutoArrayRef t_result;
     if (!MCArrayMutableCopy(t_dst_array, &t_result))
         return;

     MCNameRef t_key;
     MCValueRef t_src_value;
     MCValueRef t_dst_value;
     uintptr_t t_iterator;
     t_iterator = 0;

     while(MCArrayIterate(t_dst_array, t_iterator, t_key, t_dst_value))
     {
         bool t_key_exists;
         t_key_exists = MCArrayFetchValue(t_src_array, ctxt .
GetCaseSensitive(), t_key, t_src_value);

         if (t_key_exists == (p_op == kMCArrayDoIntersectOpDifference))
         {
             if (!MCArrayRemoveValue(*t_result, ctxt .
GetCaseSensitive(), t_key))
             {
                 ctxt . Throw();
                 return;
             }
         }
         else if (p_op == kMCArrayDoIntersectOpIntersectRecursively)
         {
             MCAutoValueRef t_recursive_result;
             MCArraysDoIntersect(ctxt, t_dst_value, t_src_value, p_op,
&t_recursive_result);

             if (ctxt . HasError())
                 return;

             if (!MCArrayStoreValue(*t_result, ctxt . GetCaseSensitive(),
t_key, *t_recursive_result))
                 return;
         }
     }

     r_result = MCValueRetain(*t_result);
}

This is, as far as I'm aware 'bug free' - in that the code respects all
the semantics required of using all the lower-level functions it is
built out of.

Here is a version which has a vulnerability/crash in it...

static void MCArraysDoIntersect(MCExecContext& ctxt, MCValueRef p_dst,
MCValueRef p_src, MCArrayDoIntersectOp p_op, MCValueRef& r_result)
{
     if (!MCValueIsArray(p_dst))
     {
         r_result = MCValueRetain(p_dst);
         return;

     }

     if (!MCValueIsArray(p_src))
     {
         if (p_op != kMCArrayDoIntersectOpDifference)
         {
             r_result = MCValueRetain(kMCEmptyString);
         }
         else
         {
             r_result = p_dst;
         }
         return;
     }

     MCArrayRef t_dst_array;
     t_dst_array = (MCArrayRef)p_dst;

     MCArrayRef t_src_array;
     t_src_array = (MCArrayRef)p_src;

     MCAutoArrayRef t_result;
     if (!MCArrayMutableCopy(t_dst_array, &t_result))
         return;

     MCNameRef t_key;
     MCValueRef t_src_value;
     MCValueRef t_dst_value;
     uintptr_t t_iterator;
     t_iterator = 0;

     while(MCArrayIterate(t_dst_array, t_iterator, t_key, t_dst_value))
     {
         bool t_key_exists;
         t_key_exists = MCArrayFetchValue(t_src_array, ctxt .
GetCaseSensitive(), t_key, t_src_value);

         if (t_key_exists == (p_op == kMCArrayDoIntersectOpDifference))
         {
             if (!MCArrayRemoveValue(*t_result, ctxt .
GetCaseSensitive(), t_key))
             {
                 ctxt . Throw();
                 return;
             }
         }
         else if (p_op == kMCArrayDoIntersectOpIntersectRecursively)
         {
             MCAutoValueRef t_recursive_result;
             MCArraysDoIntersect(ctxt, t_dst_value, t_src_value, p_op,
&t_recursive_result);

             if (ctxt . HasError())
                 return;

             if (!MCArrayStoreValue(*t_result, ctxt . GetCaseSensitive(),
t_key, *t_recursive_result))
                 return;
         }
     }

     r_result = MCValueRetain(*t_result);
}

Can you immediately see the error?

Of course I've given you the correct and incorrect versions, so its easy
to determine by line-by-line.

However, now imagine you don't know what the correct implementation is,
and you are reviewing the code for inclusion for the first time. Would
you have noticed the error (even assuming you actually designed/wrote
most of the lower level operations which are being used here - which I
did, in this case)?

Warmest Regards,

Mark.

P.S. The above bug I introduced actually causes one of the most
pernicious sort - the effect of the bug is unlikely to be seen at the
point of effect - it will only be seen later - potentially after many
many other operations are run. For those that are interested - it
introduces a reference-count error on one of the values - which would
cause a value to be released whilst there are still references to it.
They only have an effect *when* all true references are gone, *and* the
block of memory which the value uses hasn't been re-used for the same
value type (which frequently happens because the engine uses a small
pool-allocator for values of distinct types meaning repeated freeing and
creation of the same value type has almost zero cost).

P.P.S Fortunately we have tests and tools and such which help us to find
such things - and experience in knowing where to look when reviewing.
However, it is all too easy for such 'subtle' things to be missed.
Particularly when there is a large amount of code involved.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: valueDiff for arrays?

Knapp Martin via use-livecode
In reply to this post by Knapp Martin via use-livecode
On 08/04/2018 09:36 AM, Mark Waddingham via use-livecode wrote:
> On 2018-08-04 18:25, Mark Wieder via use-livecode wrote:

> Heh - well your pseudo-code just essentially made my point for me - why
> does any of that need to be in the engine, if the engine provides the
> underlying wrappers around the OS functionality (as it is on the
> specific OS) to be able to write them? :)

LOL. Well, OK. But by exposing *just* the mobileXXX function at the
scripting level, the onus is no longer on the developer to have to
script the utility function for each application. You have the
opportunity at this point to expose a single function rather than two or
three.

https://www.knowyourphrase.com/beating-a-dead-horse

--
  Mark Wieder
  [hidden email]

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
1234 ... 7