LC 9 and Memory

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

LC 9 and Memory

Stephen MacLean via use-livecode
Hi All,

I’m running into an issue where LC 9.0.1 RC1 Business is running a process and is gobbling up memory and then not releasing it until I quit LC. Like 22GB of memory!

Let me describe what I’m doing: I’m getting a query back from a MYSQL DB that contains 32k records, all text. Using revQueryDatabase() to put the data into a record set and the move cursor to go through each record in a repeat loop.

During the loop, I’m calling multiple functions that use a variety of all the goodies LC has: tsNet directly to get some data and PUT and POST URL were they work fine (For some reason, they don’t work on some URL’s, but tsNet does). Processing that data and saving it back to the DB into a different table. While doing this I’m putting some text into a logging field, moving a progress bar and updating a counter field to show what record it’s on . I’m also running a timer to show the elapsed time.

When done I save the log fld to a text file, then clear it. I also close the record set.

The memory used for LC always goes up, but never goes down unless I quit.

What am I missing? LC does it’s own GC and memory management, but in this case, I don’t see it.

Anyone experience this before? Any tips for  controlling it?

TIA,

Steve MacLean



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
Is this on a Mac? I know that for Macs, memory can be "in use" by an app, but the app will release it on request. "wired" memory is that which an app has allocated and will not release. Right now I have 11+ of 16 gigs of memory in use, but only 2.32 of it is "wired". If you look at the Activity Monitor, the indicator for Memory Pressure is really the indication of available resources.

Windows these days probably uses a similar way of managing memory.

Bob S


> On Aug 13, 2018, at 10:42 , Stephen MacLean via use-livecode <[hidden email]> wrote:
>
> Hi All,
>
> I’m running into an issue where LC 9.0.1 RC1 Business is running a process and is gobbling up memory and then not releasing it until I quit LC. Like 22GB of memory!
>
> Let me describe what I’m doing: I’m getting a query back from a MYSQL DB that contains 32k records, all text. Using revQueryDatabase() to put the data into a record set and the move cursor to go through each record in a repeat loop.
>
> During the loop, I’m calling multiple functions that use a variety of all the goodies LC has: tsNet directly to get some data and PUT and POST URL were they work fine (For some reason, they don’t work on some URL’s, but tsNet does). Processing that data and saving it back to the DB into a different table. While doing this I’m putting some text into a logging field, moving a progress bar and updating a counter field to show what record it’s on . I’m also running a timer to show the elapsed time.
>
> When done I save the log fld to a text file, then clear it. I also close the record set.
>
> The memory used for LC always goes up, but never goes down unless I quit.
>
> What am I missing? LC does it’s own GC and memory management, but in this case, I don’t see it.
>
> Anyone experience this before? Any tips for  controlling it?
>
> TIA,
>
> Steve MacLean

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
In reply to this post by Stephen MacLean via use-livecode
On 2018-08-13 19:42, Stephen MacLean via use-livecode wrote:
> Anyone experience this before? Any tips for  controlling it?

Well it sounds like you are doing a fair bit there so its really hard to
say whether or not there is a memory leak.

The first thing to do is to find out whether that is actually 22Gb in
use, or just an 'illusion'.

Leave you application running so that you can observe the gradually
increase of memory. Then determine its PID (should be visible in
Activity Monitor) then do from Terminal:

   heap <pid>

And share the report which is emitted to stdout with us :)

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
In reply to this post by Stephen MacLean via use-livecode
Yes, it’s on MacOS 10.12.6. No, this is in use… and cause my open apps, including LC,  to be “paused” saying that I had run out of memory.

I have 32GB of real RAM and a 4TB SSD that the OS sits on, with half of that free. The swap file had grown to almost 30GB in size, with most of it used., which is probably why I paused everything…

> On Aug 13, 2018, at 1:47 PM, Bob Sneidar via use-livecode <[hidden email]> wrote:
>
> Is this on a Mac? I know that for Macs, memory can be "in use" by an app, but the app will release it on request. "wired" memory is that which an app has allocated and will not release. Right now I have 11+ of 16 gigs of memory in use, but only 2.32 of it is "wired". If you look at the Activity Monitor, the indicator for Memory Pressure is really the indication of available resources.
>
> Windows these days probably uses a similar way of managing memory.
>
> Bob S
>
>
>> On Aug 13, 2018, at 10:42 , Stephen MacLean via use-livecode <[hidden email]> wrote:
>>
>> Hi All,
>>
>> I’m running into an issue where LC 9.0.1 RC1 Business is running a process and is gobbling up memory and then not releasing it until I quit LC. Like 22GB of memory!
>>
>> Let me describe what I’m doing: I’m getting a query back from a MYSQL DB that contains 32k records, all text. Using revQueryDatabase() to put the data into a record set and the move cursor to go through each record in a repeat loop.
>>
>> During the loop, I’m calling multiple functions that use a variety of all the goodies LC has: tsNet directly to get some data and PUT and POST URL were they work fine (For some reason, they don’t work on some URL’s, but tsNet does). Processing that data and saving it back to the DB into a different table. While doing this I’m putting some text into a logging field, moving a progress bar and updating a counter field to show what record it’s on . I’m also running a timer to show the elapsed time.
>>
>> When done I save the log fld to a text file, then clear it. I also close the record set.
>>
>> The memory used for LC always goes up, but never goes down unless I quit.
>>
>> What am I missing? LC does it’s own GC and memory management, but in this case, I don’t see it.
>>
>> Anyone experience this before? Any tips for  controlling it?
>>
>> TIA,
>>
>> Steve MacLean
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
In reply to this post by Stephen MacLean via use-livecode
Hi Mark,

Yes, doing quite a bit, which LC handles just fine:) Except for these with big queries;) I do want to mention that I call plenty of functions, so it’s not a real tight loop, just a loop with a lot going on.

Ok, will do the next time I run one of these… These big batches take about 5 hours to run.

Thanks,

Steve

> On Aug 13, 2018, at 1:55 PM, Mark Waddingham via use-livecode <[hidden email]> wrote:
>
> On 2018-08-13 19:42, Stephen MacLean via use-livecode wrote:
>> Anyone experience this before? Any tips for  controlling it?
>
> Well it sounds like you are doing a fair bit there so its really hard to say whether or not there is a memory leak.
>
> The first thing to do is to find out whether that is actually 22Gb in use, or just an 'illusion'.
>
> Leave you application running so that you can observe the gradually increase of memory. Then determine its PID (should be visible in Activity Monitor) then do from Terminal:
>
>  heap <pid>
>
> And share the report which is emitted to stdout with us :)
>
> Warmest Regards,
>
> Mark.
>
> --
> Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
On 2018-08-13 20:01, Stephen MacLean via use-livecode wrote:
> Hi Mark,
>
> Yes, doing quite a bit, which LC handles just fine:) Except for these
> with big queries;) I do want to mention that I call plenty of
> functions, so it’s not a real tight loop, just a loop with a lot going
> on.
>
> Ok, will do the next time I run one of these… These big batches take
> about 5 hours to run.

 From what you said in your other message this could well be a
memory-leak...

However, the other potential cause is what's called
'heap-fragmentation'. Whilst memory is being freed correctly, sometimes
certain use patterns can cause the C heap to consist of 'mostly free
space', but having no contiguous block large enough to allocate some of
the blocks of memory you are using - so the heap gets extended. When you
do free a 'big block' it gets used up by lots of small blocks.

Basically, the heap becomes like a block of swiss-cheese - you can fit
small things in the holes, but nothing big.

The 'heap' report will help determine what's going on there (you don't
need to run it until you get to the exhaustion point - just for a while
until you see the gradually continual rise).

By looking at the report it is possible to tell the actual % of memory
in use in the heap overall.

Is there any way to split up the processing of a single batch into
pieces, each run in its own process (sequentially?) - that approach
actually gives you two things - (1) a fix to the exhaustion problem you
are having and (2) the ability to recover most of the batch if one piece
fails.

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode

> On Aug 13, 2018, at 2:07 PM, Mark Waddingham via use-livecode <[hidden email]> wrote:
>
> Is there any way to split up the processing of a single batch into pieces, each run in its own process (sequentially?) - that approach actually gives you two things - (1) a fix to the exhaustion problem you are having and (2) the ability to recover most of the batch if one piece fails.
>

It’s about a split up as I can make it… I like to do that myself because 1) Helps me follow the code better 2) As you said, a failure or error won’t botch the whole thing. The only way to really make it smaller would be to limit the query. But, after this initial “heavy lifting”, it will be run against MUCH smaller data sets, on a regular basis. At which point it probably won’t matter, but I can see a cumulative effect build up if it is a memory leak.

I will let you know on the dump.

Best,

Steve



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
In reply to this post by Stephen MacLean via use-livecode
Hi Mark,

Here is one from when the app hit the half way point in it’s processing run:

Process:         LiveCode-Business [799]
Path:            /Applications/LiveCode Business 9.0.1 (rc 2).app/Contents/MacOS/LiveCode-Business
Load Address:    0x10d387000
Identifier:      com.runrev.livecode
Version:         9.0.1.15101 [RC 2] (9.0.1.15101 [RC 2])
Code Type:       X86-64
Parent Process:  ??? [1]

Date/Time:       2018-08-26 01:24:06.956 -0400
Launch Time:     2018-08-25 22:42:23.095 -0400
OS Version:      Mac OS X 10.12.6 (16G1510)
Report Version:  7
Analysis Tool:   /Applications/Xcode.app/Contents/Developer/usr/bin/heap
Analysis Tool Version:  Xcode 9.2 (9C40b)
----

Process 799: 6 zones
Zone DefaultMallocZone_0x10ee3a000: Overall size: 149641216KB; 520214585 nodes malloced for 28100061KB (18% of capacity); largest unused: [0x6003c9060040-1664KB]
Zone DefaultPurgeableMallocZone_0x11264e000: Overall size: 49836KB; 75 nodes malloced for 49828KB (99% of capacity); largest unused: [0x11daad000-11352KB]
Zone GFXMallocZone_0x10e5d5000: Overall size: 10240KB; 46 nodes malloced for 8KB (0% of capacity); largest unused: [0x7fd853001400-8187KB]
Zone MallocHelperZone_0x10e4f0000: Overall size: 2976716KB; 508527 nodes malloced for 1998382KB (67% of capacity); largest unused: [0x119aa3000-12876KB]
Zone QuartzCore_0x111cac000: Overall size: 20480KB; 4 nodes malloced for 5KB (0% of capacity); largest unused: [0x7fd851800000-8192KB]
Zone x-alloc_0x7fd84b028000: Overall size: 20KB; 5 nodes malloced for 1KB (5% of capacity); largest unused: [0x111cde000-8KB]
All zones: 520723242 nodes malloced - 30148282KB

I’d love to send you that one from near the end… LC was taking up almost 59GB of memory, memory pressure was insane, over 30 GB of compressed memory, 15GB swap file, finder having me force quit the few apps I had open. Tried to run heap, but after 3 hours it caused a kernel panic and the computer rebooted.


I can send you the full file if needed. Any help is appreciated.

Thanks,

Steve MacLean

> On Aug 13, 2018, at 2:07 PM, Mark Waddingham via use-livecode <[hidden email]> wrote:
>
> On 2018-08-13 20:01, Stephen MacLean via use-livecode wrote:
>> Hi Mark,
>> Yes, doing quite a bit, which LC handles just fine:) Except for these
>> with big queries;) I do want to mention that I call plenty of
>> functions, so it’s not a real tight loop, just a loop with a lot going
>> on.
>> Ok, will do the next time I run one of these… These big batches take
>> about 5 hours to run.
>
> From what you said in your other message this could well be a memory-leak...
>
> However, the other potential cause is what's called 'heap-fragmentation'. Whilst memory is being freed correctly, sometimes certain use patterns can cause the C heap to consist of 'mostly free space', but having no contiguous block large enough to allocate some of the blocks of memory you are using - so the heap gets extended. When you do free a 'big block' it gets used up by lots of small blocks.
>
> Basically, the heap becomes like a block of swiss-cheese - you can fit small things in the holes, but nothing big.
>
> The 'heap' report will help determine what's going on there (you don't need to run it until you get to the exhaustion point - just for a while until you see the gradually continual rise).
>
> By looking at the report it is possible to tell the actual % of memory in use in the heap overall.
>
> Is there any way to split up the processing of a single batch into pieces, each run in its own process (sequentially?) - that approach actually gives you two things - (1) a fix to the exhaustion problem you are having and (2) the ability to recover most of the batch if one piece fails.
>
> Warmest Regards,
>
> Mark.
>
> --
> Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
just wondering whats the last version in which it didn't do this?

On Sun, Aug 26, 2018 at 1:52 PM Stephen MacLean via use-livecode <
[hidden email]> wrote:

> Hi Mark,
>
> Here is one from when the app hit the half way point in it’s processing
> run:
>
> Process:         LiveCode-Business [799]
> Path:            /Applications/LiveCode Business 9.0.1 (rc
> 2).app/Contents/MacOS/LiveCode-Business
> Load Address:    0x10d387000
> Identifier:      com.runrev.livecode
> Version:         9.0.1.15101 [RC 2] (9.0.1.15101 [RC 2])
> Code Type:       X86-64
> Parent Process:  ??? [1]
>
> Date/Time:       2018-08-26 01:24:06.956 -0400
> Launch Time:     2018-08-25 22:42:23.095 -0400
> OS Version:      Mac OS X 10.12.6 (16G1510)
> Report Version:  7
> Analysis Tool:   /Applications/Xcode.app/Contents/Developer/usr/bin/heap
> Analysis Tool Version:  Xcode 9.2 (9C40b)
> ----
>
> Process 799: 6 zones
> Zone DefaultMallocZone_0x10ee3a000: Overall size: 149641216KB; 520214585
> nodes malloced for 28100061KB (18% of capacity); largest unused:
> [0x6003c9060040-1664KB]
> Zone DefaultPurgeableMallocZone_0x11264e000: Overall size: 49836KB; 75
> nodes malloced for 49828KB (99% of capacity); largest unused:
> [0x11daad000-11352KB]
> Zone GFXMallocZone_0x10e5d5000: Overall size: 10240KB; 46 nodes malloced
> for 8KB (0% of capacity); largest unused: [0x7fd853001400-8187KB]
> Zone MallocHelperZone_0x10e4f0000: Overall size: 2976716KB; 508527 nodes
> malloced for 1998382KB (67% of capacity); largest unused:
> [0x119aa3000-12876KB]
> Zone QuartzCore_0x111cac000: Overall size: 20480KB; 4 nodes malloced for
> 5KB (0% of capacity); largest unused: [0x7fd851800000-8192KB]
> Zone x-alloc_0x7fd84b028000: Overall size: 20KB; 5 nodes malloced for 1KB
> (5% of capacity); largest unused: [0x111cde000-8KB]
> All zones: 520723242 nodes malloced - 30148282KB
>
> I’d love to send you that one from near the end… LC was taking up almost
> 59GB of memory, memory pressure was insane, over 30 GB of compressed
> memory, 15GB swap file, finder having me force quit the few apps I had
> open. Tried to run heap, but after 3 hours it caused a kernel panic and the
> computer rebooted.
>
>
> I can send you the full file if needed. Any help is appreciated.
>
> Thanks,
>
> Steve MacLean
>
> > On Aug 13, 2018, at 2:07 PM, Mark Waddingham via use-livecode <
> [hidden email]> wrote:
> >
> > On 2018-08-13 20:01, Stephen MacLean via use-livecode wrote:
> >> Hi Mark,
> >> Yes, doing quite a bit, which LC handles just fine:) Except for these
> >> with big queries;) I do want to mention that I call plenty of
> >> functions, so it’s not a real tight loop, just a loop with a lot going
> >> on.
> >> Ok, will do the next time I run one of these… These big batches take
> >> about 5 hours to run.
> >
> > From what you said in your other message this could well be a
> memory-leak...
> >
> > However, the other potential cause is what's called
> 'heap-fragmentation'. Whilst memory is being freed correctly, sometimes
> certain use patterns can cause the C heap to consist of 'mostly free
> space', but having no contiguous block large enough to allocate some of the
> blocks of memory you are using - so the heap gets extended. When you do
> free a 'big block' it gets used up by lots of small blocks.
> >
> > Basically, the heap becomes like a block of swiss-cheese - you can fit
> small things in the holes, but nothing big.
> >
> > The 'heap' report will help determine what's going on there (you don't
> need to run it until you get to the exhaustion point - just for a while
> until you see the gradually continual rise).
> >
> > By looking at the report it is possible to tell the actual % of memory
> in use in the heap overall.
> >
> > Is there any way to split up the processing of a single batch into
> pieces, each run in its own process (sequentially?) - that approach
> actually gives you two things - (1) a fix to the exhaustion problem you are
> having and (2) the ability to recover most of the batch if one piece fails.
> >
> > Warmest Regards,
> >
> > Mark.
> >
> > --
> > Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
> > LiveCode: Everyone can create apps
> >
> > _______________________________________________
> > use-livecode mailing list
> > [hidden email]
> > Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode

> On Aug 26, 2018, at 8:14 PM, Tom Glod via use-livecode <[hidden email]> wrote:
>
> just wondering whats the last version in which it didn't do this?
>

If this was to me, I wouldn’t know… this project was created in LC 9.

Best,

Steve



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: LC 9 and Memory

Stephen MacLean via use-livecode
answers my question. thanks. i keep eye on threads about bugs that could
affect my projects, and memory leaks are a big danger since once of my apps
is intended to never close.

Cheers...i hope you find the reason for the runaways mem usage.

On Sun, Aug 26, 2018 at 9:08 PM Stephen MacLean via use-livecode <
[hidden email]> wrote:

>
> > On Aug 26, 2018, at 8:14 PM, Tom Glod via use-livecode <
> [hidden email]> wrote:
> >
> > just wondering whats the last version in which it didn't do this?
> >
>
> If this was to me, I wouldn’t know… this project was created in LC 9.
>
> Best,
>
> Steve
>
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode