Shell Command and cURL

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Shell Command and cURL

Gregory Lypny
Hello everyone,

I have never used LiveCode’s Shell command, and I was hoping that someone on the forum might be able to give me an example of how to use it with cURL to download a list of files from an FTP site. I have been using cURL to download many files at a time from a given directory by pasting the code into Terminal in OS X. The problem is that Terminal cannot handle a command that is too long. This limits me to pasting no more than about 500 file paths at a time, but I need to download about 145,000 files. That’s a fair bit of manual pasting. So if someone could give me an example of using the Shell command with cURL, I’d be much obliged.

Regards,

        Gregory
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Shell Command and cURL

Richard Gaskin
Gregory Lypny wrote:

 > I have never used LiveCode’s Shell command, and I was hoping that
 > someone on the forum might be able to give me an example of how to
 > use it with cURL to download a list of files from an FTP site. I have
 > been using cURL to download many files at a time from a given
 > directory by pasting the code into Terminal in OS X. The problem is
 > that Terminal cannot handle a command that is too long. This limits
 > me to pasting no more than about 500 file paths at a time, but I need
 > to download about 145,000 files. That’s a fair bit of manual pasting.
 > So if someone could give me an example of using the Shell command
 > with cURL, I’d be much obliged.

Why not use libURL?  If you pass it a valid folder path ending in "/"
it'll return the list of files there.

rsync may be faster for that; see the "--list-only" flag, e.g.:

   rsync --list-only username@servername:/directoryname/

And if you need to download those files rsync will definitely be much
faster.

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Shell Command and cURL

xtalkprogrammer
In reply to this post by Gregory Lypny
It doesn't make sense to create a list of 140000-something files and try
to paste it somewhere. Instead, write a shell script that gets the file
list and download all files in the list.

Your script might look like this:

#!/bin/bash
$ curl -s ftp://user:pass@IP/path/to/folder/ | \
     grep -e '^-' | awk '{ print $9 }' | \
         while read f; do \
             curl -O ftp://user:pass@IP/path/to/folder/$f; \
         done

Save the script in a text file and do a chmod 755 on it to make an
executable file. Now let LiveCode launch the executable file. I think
the files will be downloaded to the defaultFolder or perhaps to the
location of the executable file .

If this is too much trouble, you might also save your current shell
command to a file and have LiveCode launch it.

Kind regards,

Mark Schonewille
http://economy-x-talk.com
https://www.facebook.com/marksch

Buy the most extensive book on the
LiveCode language:
http://livecodebeginner.economy-x-talk.com

Op 22-Nov-16 om 23:29 schreef Gregory Lypny:

> Hello everyone,
>
> I have never used LiveCode’s Shell command, and I was hoping that someone on the forum might be able to give me an example of how to use it with cURL to download a list of files from an FTP site. I have been using cURL to download many files at a time from a given directory by pasting the code into Terminal in OS X. The problem is that Terminal cannot handle a command that is too long. This limits me to pasting no more than about 500 file paths at a time, but I need to download about 145,000 files. That’s a fair bit of manual pasting. So if someone could give me an example of using the Shell command with cURL, I’d be much obliged.
>
> Regards,
>
> Gregory
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Kind regards,

Drs. Mark Schonewille

Economy-x-Talk Consultancy and Software Engineering
Homepage: http://economy-x-talk.com
Twitter: http://twitter.com/xtalkprogrammer
Facebook: http://facebook.com/LiveCode.Beginner
KvK: 50277553
Reply | Threaded
Open this post in threaded view
|

Re: Shell Command and cURL

Gregory Lypny
In reply to this post by Gregory Lypny
Hi Richard and Mark,

Thanks for responding.

Richard, I had tinkered with Put and libURLDownloadToFile or whatever version of libURL was available in version 7.1 (I have only today updated to 8.1.2) but the problem with those commands is that they need to make a connection or call (not sure of the proper terminology) to the remote server for every file (say, by using a Repeat For Each loop), and the server cuts me off after a random number of files. cURL makes just one connection or keeps one connection open, and it has worked like a charm.

Thanks for the heads up about rsync. I hadn’t heard of it, so I’ll do my homework.

Thanks for the script, Mark. Shell scripts will definitely be a new venture for me.

Regards,

        Gregory

Richard Gaskin wrote:


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode