How can I scrape text from a web page that is generated by javascript?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
 There are three values at the top of this page that are updated by
javascript.  The $ amounts are not visible in the source.

https://www.coinbase.com/charts

What is the best approach to grab the text using LiveCode.  Is there a
browser widget command that will do it?

Thanks.
~Roger
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
If you need buy, sell and spot prices you can use this..
-- spot price
get url https://api.coinbase.com/v2/prices/spot?currency=USD
-- buy price
get url https://api.coinbase.com/v2/prices/buy?currency=USD
-- sell price
get url https://api.coinbase.com/v2/prices/sell?currency=USD

Data is returned as json

On Mon, Jul 24, 2017 at 6:48 AM, Roger Eller via use-livecode <
[hidden email]> wrote:

>  There are three values at the top of this page that are updated by
> javascript.  The $ amounts are not visible in the source.
>
> https://www.coinbase.com/charts
>
> What is the best approach to grab the text using LiveCode.  Is there a
> browser widget command that will do it?
>
> Thanks.
> ~Roger
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
Ah, nevermind. Only works for spot, but if you look at the api and get an
authentication key, then you should be able to request all kinds of
information through oath2

On Mon, Jul 24, 2017 at 7:05 AM, Mike Bonner <[hidden email]> wrote:

> If you need buy, sell and spot prices you can use this..
> -- spot price
> get url https://api.coinbase.com/v2/prices/spot?currency=USD
> -- buy price
> get url https://api.coinbase.com/v2/prices/buy?currency=USD
> -- sell price
> get url https://api.coinbase.com/v2/prices/sell?currency=USD
>
> Data is returned as json
>
> On Mon, Jul 24, 2017 at 6:48 AM, Roger Eller via use-livecode <
> [hidden email]> wrote:
>
>>  There are three values at the top of this page that are updated by
>> javascript.  The $ amounts are not visible in the source.
>>
>> https://www.coinbase.com/charts
>>
>> What is the best approach to grab the text using LiveCode.  Is there a
>> browser widget command that will do it?
>>
>> Thanks.
>> ~Roger
>> _______________________________________________
>> use-livecode mailing list
>> [hidden email]
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
Roger,
I use a chrome plugin called "Web Scraper", and then I process the CSV in
LC - then again, I'm usually scraping many many many pages, and this
automates the process.
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
In reply to this post by ** Clarence P Martin ** via use-livecode
> Roger E. wrote:
> Is there a browser widget command that will do it?

(Yet another option):

YES.

Set the URL of a browser widget to https://www.coinbase.com/charts
and get (via a timer) the htmltext of the browser widget. This is the
_generated_ source.

You can now search and find the needed offsets of the tags.
OR
Get the needed class-elements via javascript (more lengthy).

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
Mike, Mike, hh,

So many good ideas!  Thanks guys!

~Roger


On Mon, Jul 24, 2017 at 9:23 AM, hh via use-livecode <
[hidden email]> wrote:

> > Roger E. wrote:
> > Is there a browser widget command that will do it?
>
> (Yet another option):
>
> YES.
>
> Set the URL of a browser widget to https://www.coinbase.com/charts
> and get (via a timer) the htmltext of the browser widget. This is the
> _generated_ source.
>
> You can now search and find the needed offsets of the tags.
> OR
> Get the needed class-elements via javascript (more lengthy).
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
In reply to this post by ** Clarence P Martin ** via use-livecode
I have tried this before and it does not work without getting into Javascript. The contents of a java object are not revealed in the htmltext. I have a really great application for this if I could get it going. I would love to write an app  that gathered all the pertinent data from a web managed device (like the copiers I work with day to day) and save it to some kind of file so it could be one-click restored at some later point (like when we have to replace the hard drives).

Bob S


> On Jul 24, 2017, at 06:23 , hh via use-livecode <[hidden email]> wrote:
>
>> Roger E. wrote:
>> Is there a browser widget command that will do it?
>
> (Yet another option):
>
> YES.
>
> Set the URL of a browser widget to https://www.coinbase.com/charts
> and get (via a timer) the htmltext of the browser widget. This is the
> _generated_ source.
>
> You can now search and find the needed offsets of the tags.
> OR
> Get the needed class-elements via javascript (more lengthy).
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
In reply to this post by ** Clarence P Martin ** via use-livecode
No workie. I create a new sitemap with a URL of http://192.168.1.87 and it tells me it's not a valid URL!!! That is strike 3 for me when I encounter that kind of problem. Even if I load the full url default page for that device, then copy paste the URL it STILL tells me it's invalid.

Bob S


> On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <[hidden email]> wrote:
>
> Roger,
> I use a chrome plugin called "Web Scraper", and then I process the CSV in
> LC - then again, I'm usually scraping many many many pages, and this
> automates the process.
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
Don't know what to tell you.  I scrape a few hundred thousand pages at a
time from multiple sites with this plugin.

On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
[hidden email]> wrote:

> No workie. I create a new sitemap with a URL of http://192.168.1.87 and
> it tells me it's not a valid URL!!! That is strike 3 for me when I
> encounter that kind of problem. Even if I load the full url default page
> for that device, then copy paste the URL it STILL tells me it's invalid.
>
> Bob S
>
>
> > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> [hidden email]> wrote:
> >
> > Roger,
> > I use a chrome plugin called "Web Scraper", and then I process the CSV in
> > LC - then again, I'm usually scraping many many many pages, and this
> > automates the process.
> > _______________________________________________
> > use-livecode mailing list
> > [hidden email]
> > Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



--
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
I installed the plugin, but am unfamiliar with how to use it.  I didn't see
an obvious way to filter the page down to the current prices I want to grab.

~Roger

On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
[hidden email]> wrote:

> Don't know what to tell you.  I scrape a few hundred thousand pages at a
> time from multiple sites with this plugin.
>
> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> [hidden email]> wrote:
>
> > No workie. I create a new sitemap with a URL of http://192.168.1.87 and
> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > encounter that kind of problem. Even if I load the full url default page
> > for that device, then copy paste the URL it STILL tells me it's invalid.
> >
> > Bob S
> >
> >
> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > [hidden email]> wrote:
> > >
> > > Roger,
> > > I use a chrome plugin called "Web Scraper", and then I process the CSV
> in
> > > LC - then again, I'm usually scraping many many many pages, and this
> > > automates the process.
> > > _______________________________________________
> > > use-livecode mailing list
> > > [hidden email]
> > > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> >
> > _______________________________________________
> > use-livecode mailing list
> > [hidden email]
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
>
>
>
> --
> On the first day, God created the heavens and the Earth
> On the second day, God created the oceans.
> On the third day, God put the animals on hold for a few hours,
>    and did a little diving.
> And God said, "This is good."
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
Scratch that.  It's amazing what watching the video or reading the docs can
do to enlighten the unenlightened.


On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <[hidden email]>
wrote:

> I installed the plugin, but am unfamiliar with how to use it.  I didn't
> see an obvious way to filter the page down to the current prices I want to
> grab.
>
> ~Roger
>
> On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> [hidden email]> wrote:
>
>> Don't know what to tell you.  I scrape a few hundred thousand pages at a
>> time from multiple sites with this plugin.
>>
>> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
>> [hidden email]> wrote:
>>
>> > No workie. I create a new sitemap with a URL of http://192.168.1.87 and
>> > it tells me it's not a valid URL!!! That is strike 3 for me when I
>> > encounter that kind of problem. Even if I load the full url default page
>> > for that device, then copy paste the URL it STILL tells me it's invalid.
>> >
>> > Bob S
>> >
>> >
>> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
>> > [hidden email]> wrote:
>> > >
>> > > Roger,
>> > > I use a chrome plugin called "Web Scraper", and then I process the
>> CSV in
>> > > LC - then again, I'm usually scraping many many many pages, and this
>> > > automates the process.
>> > > _______________________________________________
>> > > use-livecode mailing list
>> > > [hidden email]
>> > > Please visit this url to subscribe, unsubscribe and manage your
>> > subscription preferences:
>> > > http://lists.runrev.com/mailman/listinfo/use-livecode
>> >
>> >
>> > _______________________________________________
>> > use-livecode mailing list
>> > [hidden email]
>> > Please visit this url to subscribe, unsubscribe and manage your
>> > subscription preferences:
>> > http://lists.runrev.com/mailman/listinfo/use-livecode
>> >
>>
>>
>>
>> --
>> On the first day, God created the heavens and the Earth
>> On the second day, God created the oceans.
>> On the third day, God put the animals on hold for a few hours,
>>    and did a little diving.
>> And God said, "This is good."
>> _______________________________________________
>> use-livecode mailing list
>> [hidden email]
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
The developer was also helpful when I was running into difficulty writing
some of the parameter sets.

On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
[hidden email]> wrote:

> Scratch that.  It's amazing what watching the video or reading the docs can
> do to enlighten the unenlightened.
>
>
> On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <[hidden email]>
> wrote:
>
> > I installed the plugin, but am unfamiliar with how to use it.  I didn't
> > see an obvious way to filter the page down to the current prices I want
> to
> > grab.
> >
> > ~Roger
> >
> > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > [hidden email]> wrote:
> >
> >> Don't know what to tell you.  I scrape a few hundred thousand pages at a
> >> time from multiple sites with this plugin.
> >>
> >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> >> [hidden email]> wrote:
> >>
> >> > No workie. I create a new sitemap with a URL of http://192.168.1.87
> and
> >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> >> > encounter that kind of problem. Even if I load the full url default
> page
> >> > for that device, then copy paste the URL it STILL tells me it's
> invalid.
> >> >
> >> > Bob S
> >> >
> >> >
> >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> >> > [hidden email]> wrote:
> >> > >
> >> > > Roger,
> >> > > I use a chrome plugin called "Web Scraper", and then I process the
> >> CSV in
> >> > > LC - then again, I'm usually scraping many many many pages, and this
> >> > > automates the process.
> >> > > _______________________________________________
> >> > > use-livecode mailing list
> >> > > [hidden email]
> >> > > Please visit this url to subscribe, unsubscribe and manage your
> >> > subscription preferences:
> >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> >> >
> >> >
> >> > _______________________________________________
> >> > use-livecode mailing list
> >> > [hidden email]
> >> > Please visit this url to subscribe, unsubscribe and manage your
> >> > subscription preferences:
> >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >> >
> >>
> >>
> >>
> >> --
> >> On the first day, God created the heavens and the Earth
> >> On the second day, God created the oceans.
> >> On the third day, God put the animals on hold for a few hours,
> >>    and did a little diving.
> >> And God said, "This is good."
> >> _______________________________________________
> >> use-livecode mailing list
> >> [hidden email]
> >> Please visit this url to subscribe, unsubscribe and manage your
> >> subscription preferences:
> >> http://lists.runrev.com/mailman/listinfo/use-livecode
> >>
> >
> >
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



--
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
So... I can make it scrape, then save to a CSV, then read the CSV into LC
and use the data.  But how to automate the scraping and CSV saving at a
given interval, I'm not there yet.

~Roger


On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
[hidden email]> wrote:

> The developer was also helpful when I was running into difficulty writing
> some of the parameter sets.
>
> On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> [hidden email]> wrote:
>
> > Scratch that.  It's amazing what watching the video or reading the docs
> can
> > do to enlighten the unenlightened.
> >
> >
> > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> [hidden email]>
> > wrote:
> >
> > > I installed the plugin, but am unfamiliar with how to use it.  I didn't
> > > see an obvious way to filter the page down to the current prices I want
> > to
> > > grab.
> > >
> > > ~Roger
> > >
> > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > [hidden email]> wrote:
> > >
> > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> at a
> > >> time from multiple sites with this plugin.
> > >>
> > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > >> [hidden email]> wrote:
> > >>
> > >> > No workie. I create a new sitemap with a URL of http://192.168.1.87
> > and
> > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > >> > encounter that kind of problem. Even if I load the full url default
> > page
> > >> > for that device, then copy paste the URL it STILL tells me it's
> > invalid.
> > >> >
> > >> > Bob S
> > >> >
> > >> >
> > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > >> > [hidden email]> wrote:
> > >> > >
> > >> > > Roger,
> > >> > > I use a chrome plugin called "Web Scraper", and then I process the
> > >> CSV in
> > >> > > LC - then again, I'm usually scraping many many many pages, and
> this
> > >> > > automates the process.
> > >> > > _______________________________________________
> > >> > > use-livecode mailing list
> > >> > > [hidden email]
> > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > >> > subscription preferences:
> > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >> >
> > >> >
> > >> > _______________________________________________
> > >> > use-livecode mailing list
> > >> > [hidden email]
> > >> > Please visit this url to subscribe, unsubscribe and manage your
> > >> > subscription preferences:
> > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> On the first day, God created the heavens and the Earth
> > >> On the second day, God created the oceans.
> > >> On the third day, God put the animals on hold for a few hours,
> > >>    and did a little diving.
> > >> And God said, "This is good."
> > >> _______________________________________________
> > >> use-livecode mailing list
> > >> [hidden email]
> > >> Please visit this url to subscribe, unsubscribe and manage your
> > >> subscription preferences:
> > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > >>
> > >
> > >
> > _______________________________________________
> > use-livecode mailing list
> > [hidden email]
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
>
>
>
> --
> On the first day, God created the heavens and the Earth
> On the second day, God created the oceans.
> On the third day, God put the animals on hold for a few hours,
>    and did a little diving.
> And God said, "This is good."
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
I always start my scrapes manually, so I am no help there with this
plugin.  They all take many hours, so I'm not starting them constantly.

On Mon, Jul 24, 2017 at 5:03 PM, Roger Eller via use-livecode <
[hidden email]> wrote:

> So... I can make it scrape, then save to a CSV, then read the CSV into LC
> and use the data.  But how to automate the scraping and CSV saving at a
> given interval, I'm not there yet.
>
> ~Roger
>
>
> On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
> [hidden email]> wrote:
>
> > The developer was also helpful when I was running into difficulty writing
> > some of the parameter sets.
> >
> > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> > [hidden email]> wrote:
> >
> > > Scratch that.  It's amazing what watching the video or reading the docs
> > can
> > > do to enlighten the unenlightened.
> > >
> > >
> > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> > [hidden email]>
> > > wrote:
> > >
> > > > I installed the plugin, but am unfamiliar with how to use it.  I
> didn't
> > > > see an obvious way to filter the page down to the current prices I
> want
> > > to
> > > > grab.
> > > >
> > > > ~Roger
> > > >
> > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > > [hidden email]> wrote:
> > > >
> > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> > at a
> > > >> time from multiple sites with this plugin.
> > > >>
> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > > >> [hidden email]> wrote:
> > > >>
> > > >> > No workie. I create a new sitemap with a URL of
> http://192.168.1.87
> > > and
> > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > > >> > encounter that kind of problem. Even if I load the full url
> default
> > > page
> > > >> > for that device, then copy paste the URL it STILL tells me it's
> > > invalid.
> > > >> >
> > > >> > Bob S
> > > >> >
> > > >> >
> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > > >> > [hidden email]> wrote:
> > > >> > >
> > > >> > > Roger,
> > > >> > > I use a chrome plugin called "Web Scraper", and then I process
> the
> > > >> CSV in
> > > >> > > LC - then again, I'm usually scraping many many many pages, and
> > this
> > > >> > > automates the process.
> > > >> > > _______________________________________________
> > > >> > > use-livecode mailing list
> > > >> > > [hidden email]
> > > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >> >
> > > >> > _______________________________________________
> > > >> > use-livecode mailing list
> > > >> > [hidden email]
> > > >> > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> On the first day, God created the heavens and the Earth
> > > >> On the second day, God created the oceans.
> > > >> On the third day, God put the animals on hold for a few hours,
> > > >>    and did a little diving.
> > > >> And God said, "This is good."
> > > >> _______________________________________________
> > > >> use-livecode mailing list
> > > >> [hidden email]
> > > >> Please visit this url to subscribe, unsubscribe and manage your
> > > >> subscription preferences:
> > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >>
> > > >
> > > >
> > > _______________________________________________
> > > use-livecode mailing list
> > > [hidden email]
> > > Please visit this url to subscribe, unsubscribe and manage your
> > > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >
> >
> >
> >
> > --
> > On the first day, God created the heavens and the Earth
> > On the second day, God created the oceans.
> > On the third day, God put the animals on hold for a few hours,
> >    and did a little diving.
> > And God said, "This is good."
> > _______________________________________________
> > use-livecode mailing list
> > [hidden email]
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



--
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
In reply to this post by ** Clarence P Martin ** via use-livecode
Something like autoit can probably be set up to automate things.  Set up an
autoit script that does the scrape, then use os scheduling tools to run the
script as often as you like.  I haven't done it in windows in a while, but
I think "at" is deprecated.  Can't recall whats replaced it.

On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode <
[hidden email]> wrote:

> So... I can make it scrape, then save to a CSV, then read the CSV into LC
> and use the data.  But how to automate the scraping and CSV saving at a
> given interval, I'm not there yet.
>
> ~Roger
>
>
> On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
> [hidden email]> wrote:
>
> > The developer was also helpful when I was running into difficulty writing
> > some of the parameter sets.
> >
> > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> > [hidden email]> wrote:
> >
> > > Scratch that.  It's amazing what watching the video or reading the docs
> > can
> > > do to enlighten the unenlightened.
> > >
> > >
> > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> > [hidden email]>
> > > wrote:
> > >
> > > > I installed the plugin, but am unfamiliar with how to use it.  I
> didn't
> > > > see an obvious way to filter the page down to the current prices I
> want
> > > to
> > > > grab.
> > > >
> > > > ~Roger
> > > >
> > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > > [hidden email]> wrote:
> > > >
> > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> > at a
> > > >> time from multiple sites with this plugin.
> > > >>
> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > > >> [hidden email]> wrote:
> > > >>
> > > >> > No workie. I create a new sitemap with a URL of
> http://192.168.1.87
> > > and
> > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > > >> > encounter that kind of problem. Even if I load the full url
> default
> > > page
> > > >> > for that device, then copy paste the URL it STILL tells me it's
> > > invalid.
> > > >> >
> > > >> > Bob S
> > > >> >
> > > >> >
> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > > >> > [hidden email]> wrote:
> > > >> > >
> > > >> > > Roger,
> > > >> > > I use a chrome plugin called "Web Scraper", and then I process
> the
> > > >> CSV in
> > > >> > > LC - then again, I'm usually scraping many many many pages, and
> > this
> > > >> > > automates the process.
> > > >> > > _______________________________________________
> > > >> > > use-livecode mailing list
> > > >> > > [hidden email]
> > > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >> >
> > > >> > _______________________________________________
> > > >> > use-livecode mailing list
> > > >> > [hidden email]
> > > >> > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> On the first day, God created the heavens and the Earth
> > > >> On the second day, God created the oceans.
> > > >> On the third day, God put the animals on hold for a few hours,
> > > >>    and did a little diving.
> > > >> And God said, "This is good."
> > > >> _______________________________________________
> > > >> use-livecode mailing list
> > > >> [hidden email]
> > > >> Please visit this url to subscribe, unsubscribe and manage your
> > > >> subscription preferences:
> > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >>
> > > >
> > > >
> > > _______________________________________________
> > > use-livecode mailing list
> > > [hidden email]
> > > Please visit this url to subscribe, unsubscribe and manage your
> > > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >
> >
> >
> >
> > --
> > On the first day, God created the heavens and the Earth
> > On the second day, God created the oceans.
> > On the third day, God put the animals on hold for a few hours,
> >    and did a little diving.
> > And God said, "This is good."
> > _______________________________________________
> > use-livecode mailing list
> > [hidden email]
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
FYI depending on what you are trying to get out you can also

get the htmltext of widget "browser"; set the htmltext of templateField(); put the text of templateField()

and see if that may serve in some use case.


 

On 7/24/17, 12:41 PM, "use-livecode on behalf of Mike Bonner via use-livecode" <[hidden email] on behalf of [hidden email]> wrote:

    Something like autoit can probably be set up to automate things.  Set up an
    autoit script that does the scrape, then use os scheduling tools to run the
    script as often as you like.  I haven't done it in windows in a while, but
    I think "at" is deprecated.  Can't recall whats replaced it.
   
    On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode <
    [hidden email]> wrote:
   
    > So... I can make it scrape, then save to a CSV, then read the CSV into LC
    > and use the data.  But how to automate the scraping and CSV saving at a
    > given interval, I'm not there yet.
    >
    > ~Roger
    >
    >
    > On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
    > [hidden email]> wrote:
    >
    > > The developer was also helpful when I was running into difficulty writing
    > > some of the parameter sets.
    > >
    > > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
    > > [hidden email]> wrote:
    > >
    > > > Scratch that.  It's amazing what watching the video or reading the docs
    > > can
    > > > do to enlighten the unenlightened.
    > > >
    > > >
    > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
    > > [hidden email]>
    > > > wrote:
    > > >
    > > > > I installed the plugin, but am unfamiliar with how to use it.  I
    > didn't
    > > > > see an obvious way to filter the page down to the current prices I
    > want
    > > > to
    > > > > grab.
    > > > >
    > > > > ~Roger
    > > > >
    > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
    > > > > [hidden email]> wrote:
    > > > >
    > > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
    > > at a
    > > > >> time from multiple sites with this plugin.
    > > > >>
    > > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
    > > > >> [hidden email]> wrote:
    > > > >>
    > > > >> > No workie. I create a new sitemap with a URL of
    > http://192.168.1.87
    > > > and
    > > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
    > > > >> > encounter that kind of problem. Even if I load the full url
    > default
    > > > page
    > > > >> > for that device, then copy paste the URL it STILL tells me it's
    > > > invalid.
    > > > >> >
    > > > >> > Bob S
    > > > >> >
    > > > >> >
    > > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
    > > > >> > [hidden email]> wrote:
    > > > >> > >
    > > > >> > > Roger,
    > > > >> > > I use a chrome plugin called "Web Scraper", and then I process
    > the
    > > > >> CSV in
    > > > >> > > LC - then again, I'm usually scraping many many many pages, and
    > > this
    > > > >> > > automates the process.
    > > > >> > > _______________________________________________
    > > > >> > > use-livecode mailing list
    > > > >> > > [hidden email]
    > > > >> > > Please visit this url to subscribe, unsubscribe and manage your
    > > > >> > subscription preferences:
    > > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
    > > > >> >
    > > > >> >
    > > > >> > _______________________________________________
    > > > >> > use-livecode mailing list
    > > > >> > [hidden email]
    > > > >> > Please visit this url to subscribe, unsubscribe and manage your
    > > > >> > subscription preferences:
    > > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
    > > > >> >
    > > > >>
    > > > >>
    > > > >>
    > > > >> --
    > > > >> On the first day, God created the heavens and the Earth
    > > > >> On the second day, God created the oceans.
    > > > >> On the third day, God put the animals on hold for a few hours,
    > > > >>    and did a little diving.
    > > > >> And God said, "This is good."
    > > > >> _______________________________________________
    > > > >> use-livecode mailing list
    > > > >> [hidden email]
    > > > >> Please visit this url to subscribe, unsubscribe and manage your
    > > > >> subscription preferences:
    > > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
    > > > >>
    > > > >
    > > > >
    > > > _______________________________________________
    > > > use-livecode mailing list
    > > > [hidden email]
    > > > Please visit this url to subscribe, unsubscribe and manage your
    > > > subscription preferences:
    > > > http://lists.runrev.com/mailman/listinfo/use-livecode
    > > >
    > >
    > >
    > >
    > > --
    > > On the first day, God created the heavens and the Earth
    > > On the second day, God created the oceans.
    > > On the third day, God put the animals on hold for a few hours,
    > >    and did a little diving.
    > > And God said, "This is good."
    > > _______________________________________________
    > > use-livecode mailing list
    > > [hidden email]
    > > Please visit this url to subscribe, unsubscribe and manage your
    > > subscription preferences:
    > > http://lists.runrev.com/mailman/listinfo/use-livecode
    > >
    > _______________________________________________
    > use-livecode mailing list
    > [hidden email]
    > Please visit this url to subscribe, unsubscribe and manage your
    > subscription preferences:
    > http://lists.runrev.com/mailman/listinfo/use-livecode
    >
    _______________________________________________
    use-livecode mailing list
    [hidden email]
    Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
    http://lists.runrev.com/mailman/listinfo/use-livecode
   

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I scrape text from a web page that is generated by javascript?

** Clarence P Martin ** via use-livecode
Indeed.  In fact, Thierry Douez solved this over the weekend with a mix of
that and some regex.


On Sat, Jul 29, 2017 at 3:56 PM, Sannyasin Brahmanathaswami via
use-livecode <[hidden email]> wrote:

> FYI depending on what you are trying to get out you can also
>
> get the htmltext of widget "browser"; set the htmltext of templateField();
> put the text of templateField()
>
> and see if that may serve in some use case.
>
>
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Loading...