Processing Large Amounts Of Data

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Processing Large Amounts Of Data

Warren Kuhl-2
I have a variable that has approx 100,000 records (loaded from a text file).
I need to read through each record and extract items from each record.

I am currently using:
Repeat with x = 1 to tRecordCount
Put item 2 of line x of tData into tItem2
Put item 6 of line x of tData into tItem6
...process data
End Repeat

I find this very slow to process. Is there a fast way to process through the
data?

Thanks!

Warren
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Processing Large Amounts Of Data

masmit
Warren, a 'repeat for each' loop should do much better:

repeat for each line L in tData
  put item 2 of L into tItem2
  put item 6 of L into tItem6
  ...process data
end repeat

best,

Mark Smith

Warren Kuhl-2 wrote
I have a variable that has approx 100,000 records (loaded from a text file).
I need to read through each record and extract items from each record.

I am currently using:
Repeat with x = 1 to tRecordCount
Put item 2 of line x of tData into tItem2
Put item 6 of line x of tData into tItem6
...process data
End Repeat

I find this very slow to process. Is there a fast way to process through the
data?

Thanks!

Warren
_______________________________________________
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Processing Large Amounts Of Data

Richard Gaskin
In reply to this post by Warren Kuhl-2
Warren Kuhl wrote:

> I have a variable that has approx 100,000 records (loaded from a text file).
> I need to read through each record and extract items from each record.
>
> I am currently using:
> Repeat with x = 1 to tRecordCount
> Put item 2 of line x of tData into tItem2
> Put item 6 of line x of tData into tItem6
> ...process data
> End Repeat
>
> I find this very slow to process. Is there a fast way to process through the
> data?

The "repeat with..." construct will be slow when using the iteration
variable to access the data, because each time through the loop it needs
to count from 1 to x to find the line in question.

You should see at least an order of magnitude performance gain using
"repeat for each...", e.g.:

  repeat for each line tLine in tData
    put item 2 of tLine into tItem2
    put item 6 of tLine into tItem6
    ...process data
  end Repeat

With "repeat for each", the engine parses the data as it goes, putting
each line as it finds it into tLine, and keep track of where it is so
the next time through the loop it just picks up where it left off.
Much, much faster.

--
  Richard Gaskin
  Fourth World
  Revolution training and consulting: http://www.fourthworld.com
  Webzine for Rev developers: http://www.revjournal.com
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Processing Large Amounts Of Data

Warren Kuhl-2
Richard,

Thanks...that really helps!  I will give it a try.

Warren


On 7/29/09, Richard Gaskin <[hidden email]> wrote:

>
> Warren Kuhl wrote:
>
> I have a variable that has approx 100,000 records (loaded from a text
>> file).
>> I need to read through each record and extract items from each record.
>>
>> I am currently using:
>> Repeat with x = 1 to tRecordCount
>> Put item 2 of line x of tData into tItem2
>> Put item 6 of line x of tData into tItem6
>> ...process data
>> End Repeat
>>
>> I find this very slow to process. Is there a fast way to process through
>> the
>> data?
>>
>
> The "repeat with..." construct will be slow when using the iteration
> variable to access the data, because each time through the loop it needs to
> count from 1 to x to find the line in question.
>
> You should see at least an order of magnitude performance gain using
> "repeat for each...", e.g.:
>
>  repeat for each line tLine in tData
>   put item 2 of tLine into tItem2
>   put item 6 of tLine into tItem6
>   ...process data
>  end Repeat
>
> With "repeat for each", the engine parses the data as it goes, putting each
> line as it finds it into tLine, and keep track of where it is so the next
> time through the loop it just picks up where it left off. Much, much faster.
>
> --
>  Richard Gaskin
>  Fourth World
>  Revolution training and consulting: http://www.fourthworld.com
>  Webzine for Rev developers: http://www.revjournal.com
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution
>
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution