Issues with storage of data in stack

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Issues with storage of data in stack

Mike Kerner via use-livecode
I'm working on a session for next week's LCG.  We are all working on
fun super-advanced pushing-the-envelope things in LC, but I wanted to do
something for the beginners who are watching LCG, and handling data seemed
like an obvious choice.

I have to admit, I haven't stored data in a stack in a long time, since the
database options are so easy to use.  If you're currently storing data in a
stack, have you run into any issues on any platform with it, especially on
mobile, for instance, where the rules can be a little weird?

--
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Mike Kerner wrote:

 > I have to admit, I haven't stored data in a stack in a long time,
 > since the database options are so easy to use.  If you're currently
 > storing data in a stack, have you run into any issues on any platform
 > with it, especially on mobile, for instance, where the rules can be a
 > little weird?

In fields, or custom props?

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Hi.

I store data in both external files and within the stack itself. Never used
a DB.

But no issues of any kind ever.

Craig Newman



--
Sent from: http://runtime-revolution.278305.n4.nabble.com/Revolution-User-f278306.html

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Hi Mike,

I have 60+ machines running windows7  embedded  running a POS/TILL
system also a few  tablets running windows and android with the same
system - wth abouth 8 lines of code added to work on Android.

 In the first version of the system I save the arrays which held all
the Clerks orders/totals etc and items into an external stack every
time they added an item.

I then saved the whole thing and cleared the  "dirty" flag   on shutdown.

If the"till"  program started and the dirty flag was true then system
would know it was a powercut and load all the "open"  orders and other
status information  etc.

I have users who have not had a problem in over 2 and a half  years
others who have 3 "tills" with no problems and just a few maybe (4 who
have had a corrupt stack maybe 1 to 3 times a years - 2 or three times
is  too many. There is no rhyme or reason because others have had
powercuts (6 in an hour ) with no damage but others can can switch off
without exiting the till properly without a problem and yet I can have
a corrupt stack with everything shut down properly.  So Now I have
removed that stack and  just save the arrays to  "general" config
table called TABS in sqlite in a row with an idkey of '99' other
IDkeys can save other configuration information- NOT ONE PROBLEM since
I have done this - go figure.

I'll make a posting on the forum with the code - not trying to teach
my granny to suck eggs here, but maybe some beginners on the forum
might find it useful - would have done in the beginning.


http://forums.livecode.com/viewtopic.php?f=7&t=30717

Regards lagi

On 7 March 2018 at 20:30, Mike Kerner via use-livecode
<[hidden email]> wrote:

> I'm working on a session for next week's LCG.  We are all working on
> fun super-advanced pushing-the-envelope things in LC, but I wanted to do
> something for the beginners who are watching LCG, and handling data seemed
> like an obvious choice.
>
> I have to admit, I haven't stored data in a stack in a long time, since the
> database options are so easy to use.  If you're currently storing data in a
> stack, have you run into any issues on any platform with it, especially on
> mobile, for instance, where the rules can be a little weird?
>
> --
> On the first day, God created the heavens and the Earth
> On the second day, God created the oceans.
> On the third day, God put the animals on hold for a few hours,
>    and did a little diving.
> And God said, "This is good."
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
I've found that sending binary data over the internet is risky. A zip file
or base64 encoded file avoids most transit corruption.
--
Jacqueline Landman Gay         |     [hidden email]
HyperActive Software           |     http://www.hyperactivesw.com



On March 8, 2018 11:06:16 AM Lagi Pittas via use-livecode
<[hidden email]> wrote:

> Hi Mike,
>
> I have 60+ machines running windows7  embedded  running a POS/TILL
> system also a few  tablets running windows and android with the same
> system - wth abouth 8 lines of code added to work on Android.
>
>  In the first version of the system I save the arrays which held all
> the Clerks orders/totals etc and items into an external stack every
> time they added an item.
>
> I then saved the whole thing and cleared the  "dirty" flag   on shutdown.
>
> If the"till"  program started and the dirty flag was true then system
> would know it was a powercut and load all the "open"  orders and other
> status information  etc.
>
> I have users who have not had a problem in over 2 and a half  years
> others who have 3 "tills" with no problems and just a few maybe (4 who
> have had a corrupt stack maybe 1 to 3 times a years - 2 or three times
> is  too many. There is no rhyme or reason because others have had
> powercuts (6 in an hour ) with no damage but others can can switch off
> without exiting the till properly without a problem and yet I can have
> a corrupt stack with everything shut down properly.  So Now I have
> removed that stack and  just save the arrays to  "general" config
> table called TABS in sqlite in a row with an idkey of '99' other
> IDkeys can save other configuration information- NOT ONE PROBLEM since
> I have done this - go figure.
>
> I'll make a posting on the forum with the code - not trying to teach
> my granny to suck eggs here, but maybe some beginners on the forum
> might find it useful - would have done in the beginning.
>
>
> http://forums.livecode.com/viewtopic.php?f=7&t=30717
>
> Regards lagi
>
> On 7 March 2018 at 20:30, Mike Kerner via use-livecode
> <[hidden email]> wrote:
>> I'm working on a session for next week's LCG.  We are all working on
>> fun super-advanced pushing-the-envelope things in LC, but I wanted to do
>> something for the beginners who are watching LCG, and handling data seemed
>> like an obvious choice.
>>
>> I have to admit, I haven't stored data in a stack in a long time, since the
>> database options are so easy to use.  If you're currently storing data in a
>> stack, have you run into any issues on any platform with it, especially on
>> mobile, for instance, where the rules can be a little weird?
>>
>> --
>> On the first day, God created the heavens and the Earth
>> On the second day, God created the oceans.
>> On the third day, God put the animals on hold for a few hours,
>>    and did a little diving.
>> And God said, "This is good."
>> _______________________________________________
>> use-livecode mailing list
>> [hidden email]
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode <[hidden email]> wrote:
>
> In the first version of the system I save the arrays which held all
> the Clerks orders/totals etc and items into an external stack every
> time they added an item.

I too use arrays as a data store. You could save the arrays to disk directly with arrayEncode(). To save time and not store the entire array to disk, you could store only the portions that have been updated on every update. This will keep the data store safe and fast. It will also remove the occasional issue of stack corruption…since you will not store the data inside of stacks any longer.

All creates/reads/update/writes would be done to memory. Those calls that modify data will simply do a follow-up write to disk for that particular record. To defeat OS inode limitations, you could group your arrays logically into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped together. The loading and storing of arrays is very fast. Even with databases in the 100s of thousands to low millions of records counts.

Performance is amazing. The best part is you are speaking to your database using LiveCode commands and functions. The only limitation is the amount of RAM on a given device. Should you outgrow that limitation then you could move the data to the cloud. Or you could load/unload data accordingly from disk. This would also be a nice indicator that your business is growing. Of course, it could just be that you are storing a lot of data. :)

If you need any help or ideas, let me know.

Best regards,

Mark Talluto
livecloud.io <http://livecloud.io/>
nursenotes.net <http://nursenotes.net/>
canelasoftware.com <http://www.canelasoftware.com/>

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
You would need to arrayencode() before saving an array to a database as well.

Bob S


> On Mar 8, 2018, at 10:38 , Mark Talluto via use-livecode <[hidden email]> wrote:
>
> I too use arrays as a data store. You could save the arrays to disk directly with arrayEncode().


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Mark,

If the data is written to a file on disk, could someone else arrayDecode the file and gain access to it?

Peter Bogdanoff
ArtsInteractive

> On Mar 8, 2018, at 10:38 AM, Mark Talluto via use-livecode <[hidden email]> wrote:
>
>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode <[hidden email]> wrote:
>>
>> In the first version of the system I save the arrays which held all
>> the Clerks orders/totals etc and items into an external stack every
>> time they added an item.
>
> I too use arrays as a data store. You could save the arrays to disk directly with arrayEncode(). To save time and not store the entire array to disk, you could store only the portions that have been updated on every update. This will keep the data store safe and fast. It will also remove the occasional issue of stack corruption…since you will not store the data inside of stacks any longer.
>
> All creates/reads/update/writes would be done to memory. Those calls that modify data will simply do a follow-up write to disk for that particular record. To defeat OS inode limitations, you could group your arrays logically into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped together. The loading and storing of arrays is very fast. Even with databases in the 100s of thousands to low millions of records counts.
>
> Performance is amazing. The best part is you are speaking to your database using LiveCode commands and functions. The only limitation is the amount of RAM on a given device. Should you outgrow that limitation then you could move the data to the cloud. Or you could load/unload data accordingly from disk. This would also be a nice indicator that your business is growing. Of course, it could just be that you are storing a lot of data. :)
>
> If you need any help or ideas, let me know.
>
> Best regards,
>
> Mark Talluto
> livecloud.io <http://livecloud.io/>
> nursenotes.net <http://nursenotes.net/>
> canelasoftware.com <http://www.canelasoftware.com/>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Yes, if they had Livecode. In that case encrypt the arraydecode string using a seed that only you know. I use a special seed value that is different for every record I add to a database. Very effective and hard to guess.

Bob S


> On Mar 8, 2018, at 11:22 , Peter Bogdanoff via use-livecode <[hidden email]> wrote:
>
> Mark,
>
> If the data is written to a file on disk, could someone else arrayDecode the file and gain access to it?
>
> Peter Bogdanoff


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Hi Peter,

If your app could encrypt the arrayEncoded data before writing to disk,
and then decrypt it when reading it, that would solve the
unwanted-access problem. The amount of time used by encrypt/decrypt
processes is negligible, even for large files.

I love arrays. They have become one of my favorite things about LC.

Phil Davis


On 3/8/18 11:22 AM, Peter Bogdanoff via use-livecode wrote:

> Mark,
>
> If the data is written to a file on disk, could someone else arrayDecode the file and gain access to it?
>
> Peter Bogdanoff
> ArtsInteractive
>
>> On Mar 8, 2018, at 10:38 AM, Mark Talluto via use-livecode <[hidden email]> wrote:
>>
>>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode <[hidden email]> wrote:
>>>
>>> In the first version of the system I save the arrays which held all
>>> the Clerks orders/totals etc and items into an external stack every
>>> time they added an item.
>> I too use arrays as a data store. You could save the arrays to disk directly with arrayEncode(). To save time and not store the entire array to disk, you could store only the portions that have been updated on every update. This will keep the data store safe and fast. It will also remove the occasional issue of stack corruption…since you will not store the data inside of stacks any longer.
>>
>> All creates/reads/update/writes would be done to memory. Those calls that modify data will simply do a follow-up write to disk for that particular record. To defeat OS inode limitations, you could group your arrays logically into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped together. The loading and storing of arrays is very fast. Even with databases in the 100s of thousands to low millions of records counts.
>>
>> Performance is amazing. The best part is you are speaking to your database using LiveCode commands and functions. The only limitation is the amount of RAM on a given device. Should you outgrow that limitation then you could move the data to the cloud. Or you could load/unload data accordingly from disk. This would also be a nice indicator that your business is growing. Of course, it could just be that you are storing a lot of data. :)
>>
>> If you need any help or ideas, let me know.
>>
>> Best regards,
>>
>> Mark Talluto
>> livecloud.io <http://livecloud.io/>
>> nursenotes.net <http://nursenotes.net/>
>> canelasoftware.com <http://www.canelasoftware.com/>
>>
>> _______________________________________________
>> use-livecode mailing list
>> [hidden email]
>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

--
Phil Davis


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Hi Peter,

Yes. But, you can protect it with a password.

1. arrayEncode(yourDataInAVariable)
2. Encrypt the result above
3. Write out the encrypted value to disk

Best regards,

Mark Talluto
livecloud.io <http://livecloud.io/>
nursenotes.net <http://nursenotes.net/>
canelasoftware.com <http://www.canelasoftware.com/>


> On Mar 8, 2018, at 11:22 AM, Peter Bogdanoff via use-livecode <[hidden email]> wrote:
>
> Mark,
>
> If the data is written to a file on disk, could someone else arrayDecode the file and gain access to it?
>
> Peter Bogdanoff
> ArtsInteractive
>
>> On Mar 8, 2018, at 10:38 AM, Mark Talluto via use-livecode <[hidden email]> wrote:
>>
>>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode <[hidden email]> wrote:
>>>
>>> In the first version of the system I save the arrays which held all
>>> the Clerks orders/totals etc and items into an external stack every
>>> time they added an item.
>>
>> I too use arrays as a data store. You could save the arrays to disk directly with arrayEncode(). To save time and not store the entire array to disk, you could store only the portions that have been updated on every update. This will keep the data store safe and fast. It will also remove the occasional issue of stack corruption…since you will not store the data inside of stacks any longer.
>>
>> All creates/reads/update/writes would be done to memory. Those calls that modify data will simply do a follow-up write to disk for that particular record. To defeat OS inode limitations, you could group your arrays logically into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped together. The loading and storing of arrays is very fast. Even with databases in the 100s of thousands to low millions of records counts.
>>
>> Performance is amazing. The best part is you are speaking to your database using LiveCode commands and functions. The only limitation is the amount of RAM on a given device. Should you outgrow that limitation then you could move the data to the cloud. Or you could load/unload data accordingly from disk. This would also be a nice indicator that your business is growing. Of course, it could just be that you are storing a lot of data. :)
>>
>> If you need any help or ideas, let me know.
>>
>> Best regards,
>>
>> Mark Talluto
>> livecloud.io <http://livecloud.io/>
>> nursenotes.net <http://nursenotes.net/>
>> canelasoftware.com <http://www.canelasoftware.com/>
>>
>> _______________________________________________
>> use-livecode mailing list
>> [hidden email]
>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
HI Mark,

I am intrigued by your way of saving only whats changed and also when
you say save to disk after arrayencoded. Do you meanas a simple  save
as a binfile or in an sqlite BLOB?

I would really like to see some exampleish code on saving to disk - if
it's other than a single array in a single binfile - I think even I
can do that. But your other ideas sound brilliant.

Regards Lagi

On 8 March 2018 at 18:38, Mark Talluto via use-livecode
<[hidden email]> wrote:

>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode <[hidden email]> wrote:
>>
>> In the first version of the system I save the arrays which held all
>> the Clerks orders/totals etc and items into an external stack every
>> time they added an item.
>
> I too use arrays as a data store. You could save the arrays to disk directly with arrayEncode(). To save time and not store the entire array to disk, you could store only the portions that have been updated on every update. This will keep the data store safe and fast. It will also remove the occasional issue of stack corruption…since you will not store the data inside of stacks any longer.
>
> All creates/reads/update/writes would be done to memory. Those calls that modify data will simply do a follow-up write to disk for that particular record. To defeat OS inode limitations, you could group your arrays logically into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped together. The loading and storing of arrays is very fast. Even with databases in the 100s of thousands to low millions of records counts.
>
> Performance is amazing. The best part is you are speaking to your database using LiveCode commands and functions. The only limitation is the amount of RAM on a given device. Should you outgrow that limitation then you could move the data to the cloud. Or you could load/unload data accordingly from disk. This would also be a nice indicator that your business is growing. Of course, it could just be that you are storing a lot of data. :)
>
> If you need any help or ideas, let me know.
>
> Best regards,
>
> Mark Talluto
> livecloud.io <http://livecloud.io/>
> nursenotes.net <http://nursenotes.net/>
> canelasoftware.com <http://www.canelasoftware.com/>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Some notes from a while back on using LSON (LiveCode encoded arrays):

http://lists.runrev.com/pipermail/use-livecode/2016-May/227330.html

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Hi Lagi,

Our LiveCode array database does not use SQL or any other database. In terms of local only database, we do not rely on anything but LiveCode. It is purely a LiveCode derived system. Data is manipulated using familiar methodologies to other databases. The data is encrypted using ‘encrypt’ and stored using 'arrayEncode()’.  

The full array of the database is stored in memory. This method provides very quick access to your data thanks to the amazing performance provided by LiveCode.

This might get a little long. I am happy to take this off list for more details. I will try to be as succinct as possible.

-A little more explanation on storing data-
Each record is stored in array that looks like this: tableID/clusterID/recordID/recordData…
When a given record/s is updated, we cache the recordIDs that were touched. All data is updated first in memory then cached to disk. We then refer to the cached records and conclude which clusterIDs we affected. Thus, you can very quickly save only the clusters that have been modified. Each cluster will have one or more records associated with it. The clusters are the first ’n’ characters of the recordIDs. We use UUIDs as recordIDs. The cluster sizes are definable, giving us the power to decide where to apply the optimization. Clusters of only one or two characters will generate less clusters to be stored. This makes loading of tables from disk to RAM very fast. A cluster of 3 chars or more allows for less records per cluster thus improving saving from RAM to disk to be faster.

The pseudo code for this might looks like this:
-receive request for update in your API
-store the changes to your master array in RAM
-remember the recordIDs touched in a variable
-calculate the clusters touched by taking the first ’n’ characters of the records touched and make a new list of the clusters you need to write to disk
-write appropriate clusters to disk
-return the results of the action (any errors, recordIDs…)

You will find this method to be very performant and easy to manage. This is not particularly complicated to write. Once you get it all working you might add other niceties like:
-error checking the input before storing anything
-store metadata on each updated record: recordVersion, recordTime, updateTime
-add security using ‘encrypt’
-build simple APIs to do your CRUD first
-add other APIs as needed to make accessing your data easier

Here is an example API for storing data that you may find useful when making your own system.

-Input (array)-
put “Steve” into tInputA[“firstName”]
put “Jobs” into tInputA[“lastName”]
put “rolodex” into tInputA[“cdbTableName”]
put “local” into tInputA[“cdbTarget”] —We would use ‘cloud’ when we want to store offsite.

Your system might verify that the keys ‘firstName' and ‘lastName' are actual keys. This is very SQL in terms of error checking. Or, do not check the keys and feel more noSQL in nature.

From here we take the array and pass it to a function.
put cdb_create(tInputA) into tRecordID —returns the unique UUID representing the recordID

The ‘cdb_create()’ function basically runs the pseudo code proposed above. And voilà, you have your first record stored.

I hope this gives you some ideas. We have successfully used this in enterprise level scenarios. We nightly interchange data with our customer's database. We have not run into any issues with IT in terms of the type of database we are using in our backend. I fully support your experimentation with arrays as a data store.  :)

If you are interested in seeing other APIs we have found useful, here is a running list. Look under the API dropdown for more ideas.
http://canelasoftware.github.io/cassiaDB_docs/ <http://canelasoftware.github.io/cassiaDB_docs/>

-Mark


> On Mar 9, 2018, at 2:07 AM, Lagi Pittas <[hidden email]> wrote:
>
> HI Mark,
>
> I am intrigued by your way of saving only whats changed and also when
> you say save to disk after arrayencoded. Do you meanas a simple  save
> as a binfile or in an sqlite BLOB?
>
> I would really like to see some exampleish code on saving to disk - if
> it's other than a single array in a single binfile - I think even I
> can do that. But your other ideas sound brilliant.
>
> Regards Lagi
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Mark Talluto wrote:

    If you are interested in seeing other APIs we have found useful, here is a running list. Look under the API dropdown for more ideas.
    http://canelasoftware.github.io/cassiaDB_docs/ http://canelasoftware.github.io/cassiaDB_docs/

Where does one download the LiveCloud Manager application (duh, I am not seeing something)

BR



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Hi BR,

Sorry about that. I have updated the docs page to link back to the main site.  
https://livecloud.io <https://livecloud.io/>

Mark


> On Mar 11, 2018, at 7:18 PM, Sannyasin Brahmanathaswami via use-livecode <[hidden email]> wrote:
>
> Mark Talluto wrote:
>
>    If you are interested in seeing other APIs we have found useful, here is a running list. Look under the API dropdown for more ideas.
>    http://canelasoftware.github.io/cassiaDB_docs/ http://canelasoftware.github.io/cassiaDB_docs/
>
> Where does one download the LiveCloud Manager application (duh, I am not seeing something)
>
> BR
>
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Hi Mark,

Thanks for the detailed explanation but I have a few (ish) questions ...

Hope you don't mind me asking these questions because I did have to
write my own access routines in those bad old days before I started on
Clipper/Foxpro/Delphi/Btrieve  and I do enjoy learning from others on
the list and the forums - those AHA! moments when you finally get how
the Heapsort works the night before the exam.

Many moons ago I wrote a multi-way B-TREE based  on the explanation in
Wirth's Book "Algorithms + Data Structures = Programs" -  in UCSD
Pascal for the Apple 2,  I  had a 5MB hard Drives for the bigger
companies when I was lucky, for the smaller companies I made do with 2
143k floppy disks and Hashing for a "large" data set- oh the memories.
I used   the B-Trees  if the codes were alphanumeric. I also had my
own method where I kept the index in the first X Blocks of the file
and loaded the parts in memory as they were needed - a brain dead
version of yours I suppose.  I think we had about 40k of free ram to
Play with so couldn't always keep everything in RAM. I even made the
system multi-user and ran 20 Apple ][s on a network using a
proprietary Nestar/Zynar network using Ribbon Cables -  it worked but
am I glad we have Ethernet!

Anyway - I digress. I can understand the general idea of what you are
explaining but it's the low level code for writing to the
clusters/file on disk I'm not quite sure of.
Which way do you build your initial file? Is it "Sparse" or prebuilt,
or does each cluster  have a "pointer" to previous or next clusters?
Do you have records "spanning" clusters or do you leave any spare
space in a cluster empty. Do you mark a "record" as deleted but don't
remove the record until it's overwritten or do what Foxpro/Dbase does
and "PACK" them with a utility routine.
I also presume you use the "AT" option in the write command to write
the clusters randomly since you don't wriite the whole in memory table

Which brings me onto my final questions - I presume your system is
multi-user because you have a server program that receives calls and
executes them sequentially? And lastly what are the file size
limitations doing it this way - do You also virtualize the data in
memory?

Sorry for all the question but this is the interesting stuff

Regards Lagi

On 11 March 2018 at 20:02, Mark Talluto <[hidden email]> wrote:

> Hi Lagi,
>
> Our LiveCode array database does not use SQL or any other database. In terms
> of local only database, we do not rely on anything but LiveCode. It is
> purely a LiveCode derived system. Data is manipulated using familiar
> methodologies to other databases. The data is encrypted using ‘encrypt’ and
> stored using 'arrayEncode()’.
>
> The full array of the database is stored in memory. This method provides
> very quick access to your data thanks to the amazing performance provided by
> LiveCode.
>
> This might get a little long. I am happy to take this off list for more
> details. I will try to be as succinct as possible.
>
> -A little more explanation on storing data-
> Each record is stored in array that looks like this:
> tableID/clusterID/recordID/recordData…
> When a given record/s is updated, we cache the recordIDs that were touched.
> All data is updated first in memory then cached to disk. We then refer to
> the cached records and conclude which clusterIDs we affected. Thus, you can
> very quickly save only the clusters that have been modified. Each cluster
> will have one or more records associated with it. The clusters are the first
> ’n’ characters of the recordIDs. We use UUIDs as recordIDs. The cluster
> sizes are definable, giving us the power to decide where to apply the
> optimization. Clusters of only one or two characters will generate less
> clusters to be stored. This makes loading of tables from disk to RAM very
> fast. A cluster of 3 chars or more allows for less records per cluster thus
> improving saving from RAM to disk to be faster.
>
> The pseudo code for this might looks like this:
> -receive request for update in your API
> -store the changes to your master array in RAM
> -remember the recordIDs touched in a variable
> -calculate the clusters touched by taking the first ’n’ characters of the
> records touched and make a new list of the clusters you need to write to
> disk
> -write appropriate clusters to disk
> -return the results of the action (any errors, recordIDs…)
>
> You will find this method to be very performant and easy to manage. This is
> not particularly complicated to write. Once you get it all working you might
> add other niceties like:
> -error checking the input before storing anything
> -store metadata on each updated record: recordVersion, recordTime,
> updateTime
> -add security using ‘encrypt’
> -build simple APIs to do your CRUD first
> -add other APIs as needed to make accessing your data easier
>
> Here is an example API for storing data that you may find useful when making
> your own system.
>
> -Input (array)-
> put “Steve” into tInputA[“firstName”]
> put “Jobs” into tInputA[“lastName”]
> put “rolodex” into tInputA[“cdbTableName”]
> put “local” into tInputA[“cdbTarget”] —We would use ‘cloud’ when we want to
> store offsite.
>
> Your system might verify that the keys ‘firstName' and ‘lastName' are actual
> keys. This is very SQL in terms of error checking. Or, do not check the keys
> and feel more noSQL in nature.
>
> From here we take the array and pass it to a function.
> put cdb_create(tInputA) into tRecordID —returns the unique UUID representing
> the recordID
>
> The ‘cdb_create()’ function basically runs the pseudo code proposed above.
> And voilà, you have your first record stored.
>
> I hope this gives you some ideas. We have successfully used this in
> enterprise level scenarios. We nightly interchange data with our customer's
> database. We have not run into any issues with IT in terms of the type of
> database we are using in our backend. I fully support your experimentation
> with arrays as a data store.  :)
>
> If you are interested in seeing other APIs we have found useful, here is a
> running list. Look under the API dropdown for more ideas.
> http://canelasoftware.github.io/cassiaDB_docs/
>
> -Mark
>
>
> On Mar 9, 2018, at 2:07 AM, Lagi Pittas <[hidden email]> wrote:
>
> HI Mark,
>
> I am intrigued by your way of saving only whats changed and also when
> you say save to disk after arrayencoded. Do you meanas a simple  save
> as a binfile or in an sqlite BLOB?
>
> I would really like to see some exampleish code on saving to disk - if
> it's other than a single array in a single binfile - I think even I
> can do that. But your other ideas sound brilliant.
>
> Regards Lagi

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Hi Lagi,

Sorry about the delayed reply. I have been on a long business trip. Your early designs are far more sophisticated than what we put together here. Super impressive history you have.

LiveCode really is the champion here in that we are only using arrayEncode() and put myArrayA into url() to store the arrays. Selecting which array cluster to store might be easier to understand using a video.

http://canelasoftware.com/pub/canela/liveCode_array_clustering.mp4

Once you understand how the array is structured, I think the method will be clear.

We do not preallocate space. No appending. We overwrite a cluster when one or more records are saved to disk. The write happens at the end of the CRUD operation taking place. Thus, if you ‘create’ a single record, the record is first created in memory and then the cluster it belongs to is written to disk. I have toyed with the idea of making the write to disk feature controllable by the dev. Thus, you could define when the write is to take place. For example, you might like to write to disk after every 5 transactions or so. But, I have not found the write to affect performance in a noticeable way to need to add that feature.

-Multi User-
Yes, everything is processed sequentially in the cloud. There are no open sockets so you can have massive concurrent connections. All cloud calls are done via ‘post’. They are handled by PHP scripts to write the request to a cache area. One or more LiveCode standalones on the other end processes the request in the order they are received. Thus, should a process go down, no data is lost. When the process comes a back up, everything continues again as normal. Scale is handled by having more than one process be available. More scaling is handled by having data stored across multiple droplets/VMs (sharding). This can keep repeating itself as needed.

-File Size Limitations-
The OS iNode limitations are negated by not reaching its maximum limit. We found 40,000 files would really bring the performance down. Adding clustering of arrays lowers the file count to acceptable and controllable levels.

-Test Data-
100,000 records in table
Record size average: 45 chars
Keys in each record: last_name, first_name, date_of_birth, middle_name, student_number, gender, grade_level, active

A cluster size, clusters per table, time to load all clusters from disk to RAM, time to write all clusters from RAM to disk, time to write one cluster from RAM to disk:
1, 16, 1.46 secs, 1.5 secs, 91.4 ms
2, 256, 1.52 secs, 1.5 secs, 6.7 ms
3, 4096, 2.38 secs, 1.6 secs, 0.8 ms

I hope this information is helpful. Please let me know if you have any other questions.

Best regards,

Mark Talluto
livecloud.io <http://livecloud.io/>
nursenotes.net <http://nursenotes.net/>
canelasoftware.com <http://www.canelasoftware.com/>




> On Mar 12, 2018, at 10:31 AM, Lagi Pittas <[hidden email]> wrote:
>
> Hi Mark,
>
> Thanks for the detailed explanation but I have a few (ish) questions ...
>
> Hope you don't mind me asking these questions because I did have to
> write my own access routines in those bad old days before I started on
> Clipper/Foxpro/Delphi/Btrieve  and I do enjoy learning from others on
> the list and the forums - those AHA! moments when you finally get how
> the Heapsort works the night before the exam.
>
> Many moons ago I wrote a multi-way B-TREE based  on the explanation in
> Wirth's Book "Algorithms + Data Structures = Programs" -  in UCSD
> Pascal for the Apple 2,  I  had a 5MB hard Drives for the bigger
> companies when I was lucky, for the smaller companies I made do with 2
> 143k floppy disks and Hashing for a "large" data set- oh the memories.
> I used   the B-Trees  if the codes were alphanumeric. I also had my
> own method where I kept the index in the first X Blocks of the file
> and loaded the parts in memory as they were needed - a brain dead
> version of yours I suppose.  I think we had about 40k of free ram to
> Play with so couldn't always keep everything in RAM. I even made the
> system multi-user and ran 20 Apple ][s on a network using a
> proprietary Nestar/Zynar network using Ribbon Cables -  it worked but
> am I glad we have Ethernet!
>
> Anyway - I digress. I can understand the general idea of what you are
> explaining but it's the low level code for writing to the
> clusters/file on disk I'm not quite sure of.
> Which way do you build your initial file? Is it "Sparse" or prebuilt,
> or does each cluster  have a "pointer" to previous or next clusters?
> Do you have records "spanning" clusters or do you leave any spare
> space in a cluster empty. Do you mark a "record" as deleted but don't
> remove the record until it's overwritten or do what Foxpro/Dbase does
> and "PACK" them with a utility routine.
> I also presume you use the "AT" option in the write command to write
> the clusters randomly since you don't wriite the whole in memory table
>
> Which brings me onto my final questions - I presume your system is
> multi-user because you have a server program that receives calls and
> executes them sequentially? And lastly what are the file size
> limitations doing it this way - do You also virtualize the data in
> memory?
>
> Sorry for all the question but this is the interesting stuff
>
> Regards Lagi

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Hi Mark,

Thanks for that - I'd forgotten I'd asked ;-)

The times you gave are super impressive - who needs mysql - Im gonna
see if I get anytime in the next month - snowed under as usual.

I think I have got it - each cluster is in effect a file on the disk?

I will understand it more when I have had time to pause the video a
few times and try to implement a very simple system although to be
honest a few thousand records should be ample in most of my stuff and
I can archive data to a history file. Sqlite does fine for my single
user systems but needing routines to change between LC dates and
sqlite dates is a pain.

Thanks again

Lagi



On 22 March 2018 at 23:28, Mark Talluto <[hidden email]> wrote:

> Hi Lagi,
>
> Sorry about the delayed reply. I have been on a long business trip. Your
> early designs are far more sophisticated than what we put together here.
> Super impressive history you have.
>
> LiveCode really is the champion here in that we are only using arrayEncode()
> and put myArrayA into url() to store the arrays. Selecting which array
> cluster to store might be easier to understand using a video.
>
> http://canelasoftware.com/pub/canela/liveCode_array_clustering.mp4
>
> Once you understand how the array is structured, I think the method will be
> clear.
>
> We do not preallocate space. No appending. We overwrite a cluster when one
> or more records are saved to disk. The write happens at the end of the CRUD
> operation taking place. Thus, if you ‘create’ a single record, the record is
> first created in memory and then the cluster it belongs to is written to
> disk. I have toyed with the idea of making the write to disk feature
> controllable by the dev. Thus, you could define when the write is to take
> place. For example, you might like to write to disk after every 5
> transactions or so. But, I have not found the write to affect performance in
> a noticeable way to need to add that feature.
>
> -Multi User-
> Yes, everything is processed sequentially in the cloud. There are no open
> sockets so you can have massive concurrent connections. All cloud calls are
> done via ‘post’. They are handled by PHP scripts to write the request to a
> cache area. One or more LiveCode standalones on the other end processes the
> request in the order they are received. Thus, should a process go down, no
> data is lost. When the process comes a back up, everything continues again
> as normal. Scale is handled by having more than one process be available.
> More scaling is handled by having data stored across multiple droplets/VMs
> (sharding). This can keep repeating itself as needed.
>
> -File Size Limitations-
> The OS iNode limitations are negated by not reaching its maximum limit. We
> found 40,000 files would really bring the performance down. Adding
> clustering of arrays lowers the file count to acceptable and controllable
> levels.
>
> -Test Data-
> 100,000 records in table
> Record size average: 45 chars
> Keys in each record: last_name, first_name, date_of_birth, middle_name,
> student_number, gender, grade_level, active
>
> A cluster size, clusters per table, time to load all clusters from disk to
> RAM, time to write all clusters from RAM to disk, time to write one cluster
> from RAM to disk:
> 1, 16, 1.46 secs, 1.5 secs, 91.4 ms
> 2, 256, 1.52 secs, 1.5 secs, 6.7 ms
> 3, 4096, 2.38 secs, 1.6 secs, 0.8 ms
>
> I hope this information is helpful. Please let me know if you have any other
> questions.
>
> Best regards,
>
> Mark Talluto
> livecloud.io
> nursenotes.net
> canelasoftware.com
>
>
>
>
> On Mar 12, 2018, at 10:31 AM, Lagi Pittas <[hidden email]> wrote:
>
> Hi Mark,
>
> Thanks for the detailed explanation but I have a few (ish) questions ...
>
> Hope you don't mind me asking these questions because I did have to
> write my own access routines in those bad old days before I started on
> Clipper/Foxpro/Delphi/Btrieve  and I do enjoy learning from others on
> the list and the forums - those AHA! moments when you finally get how
> the Heapsort works the night before the exam.
>
> Many moons ago I wrote a multi-way B-TREE based  on the explanation in
> Wirth's Book "Algorithms + Data Structures = Programs" -  in UCSD
> Pascal for the Apple 2,  I  had a 5MB hard Drives for the bigger
> companies when I was lucky, for the smaller companies I made do with 2
> 143k floppy disks and Hashing for a "large" data set- oh the memories.
> I used   the B-Trees  if the codes were alphanumeric. I also had my
> own method where I kept the index in the first X Blocks of the file
> and loaded the parts in memory as they were needed - a brain dead
> version of yours I suppose.  I think we had about 40k of free ram to
> Play with so couldn't always keep everything in RAM. I even made the
> system multi-user and ran 20 Apple ][s on a network using a
> proprietary Nestar/Zynar network using Ribbon Cables -  it worked but
> am I glad we have Ethernet!
>
> Anyway - I digress. I can understand the general idea of what you are
> explaining but it's the low level code for writing to the
> clusters/file on disk I'm not quite sure of.
> Which way do you build your initial file? Is it "Sparse" or prebuilt,
> or does each cluster  have a "pointer" to previous or next clusters?
> Do you have records "spanning" clusters or do you leave any spare
> space in a cluster empty. Do you mark a "record" as deleted but don't
> remove the record until it's overwritten or do what Foxpro/Dbase does
> and "PACK" them with a utility routine.
> I also presume you use the "AT" option in the write command to write
> the clusters randomly since you don't wriite the whole in memory table
>
> Which brings me onto my final questions - I presume your system is
> multi-user because you have a server program that receives calls and
> executes them sequentially? And lastly what are the file size
> limitations doing it this way - do You also virtualize the data in
> memory?
>
> Sorry for all the question but this is the interesting stuff
>
> Regards Lagi
>
>

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Issues with storage of data in stack

Mike Kerner via use-livecode
Lagi Pittas write:

 > Sqlite does fine for my single user systems but needing routines to
 > change between LC dates and sqlite dates is a pain.

Indeed it is, or to any format not already handled.

There are some requests in the bug DB for other formats:
https://quality.livecode.com/show_bug.cgi?id=4636
https://quality.livecode.com/show_bug.cgi?id=19601

Most interesting is Comment #4 in bz# from Monte:
https://quality.livecode.com/show_bug.cgi?id=19601#c4

Apparently he submitted a pull request last year for what seems like a
very nice solution, using formatting strings:
https://github.com/livecode/livecode/pull/5433

This was per this BZ request:
https://quality.livecode.com/show_bug.cgi?id=18150

Not sure what happened to that.  Seems like it would be a great option.

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
12