Put URL into variable and textEncoding

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Put URL into variable and textEncoding

Mike Kerner via use-livecode
Hi,

I am a little bit puzzled at the moment. When I am using

Put URL „http://www.some/service/endpoint <http://www.some/service/endpoint>“ into tVariable

and the content is UTF-8 encoded I vaguely recall I had to textDecode it to display it correctly in a field afterwards. Did this change?

I am not sure if the web service I use changed, or if it is the latest live code stable version…

Any pointers much appreciated.

Cheers,

Malte



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode
If you know it *is* utf-8 then (in LC 8/9) it "just works" whether put into
a variable or directly into a field.
But you have to be aware when working with the variable that you have to
use codepoint for conversions for "foreign chars" (e.g. using diacritics).

> Malte P-B. wrote:
> Hi, I am a little bit puzzled at the moment. When I am using
> Put URL "http://www.some/service/endpoint <http://www.some/service/endpoint>“ into tVariable
> and the content is UTF-8 encoded I vaguely recall I had to textDecode it to display it
> correctly in a field afterwards. Did this change?
> I am not sure if the web service I use changed, or if it is the latest live code stable
> version…
> Any pointers much appreciated.
> Cheers,
> Malte

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode
No, nothing has changed. The Dictionary entry for textEncode and
textDecode still contains this note:

It is highly recommended that any time you interface with things
outside LiveCode (files, network sockets, processes, etc) that you
explicitly textEncode any text you send outside LiveCode and
textDecode all text received into LiveCode. If this doesnt happen, a
platform-dependent encoding will be used (which normally does not
support Unicode text).

Of course, the URL you use today might work fine without textDecode,
but another URL and another day you might not be so lucky.

put textDecode(URL "http://www.some/service/endpoint","UTF8") into tVariable

is still only a single line of code and I very much doubt that you
will notice the extra cpu cycles.

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode
Given the nearly-universal use of UTF-8, it would seem ideal for a
language focusing on ease of use if that could be made a default.

I can understand why we can't use "open tFile for text..." given the
legacy, but maybe something like "open tFile for modernText..." or
something more sensible.

--
  Richard Gaskin
  Fourth World Systems


Kay C Lan wrote:

> No, nothing has changed. The Dictionary entry for textEncode and
> textDecode still contains this note:
>
> It is highly recommended that any time you interface with things
> outside LiveCode (files, network sockets, processes, etc) that you
> explicitly textEncode any text you send outside LiveCode and
> textDecode all text received into LiveCode. If this doesnt happen, a
> platform-dependent encoding will be used (which normally does not
> support Unicode text).
>
> Of course, the URL you use today might work fine without textDecode,
> but another URL and another day you might not be so lucky.
>
> put textDecode(URL "http://www.some/service/endpoint","UTF8") into tVariable
>
> is still only a single line of code and I very much doubt that you
> will notice the extra cpu cycles.


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode


> On 13 Feb 2018, at 11:56 am, Richard Gaskin via use-livecode <[hidden email]> wrote:
>
> something more sensible


We already have:

open file for “utf-8” text

There was some discussion of implementing “file+<encoding>:/path/to/file” style urls to avoid having to use binfile and textDecode and manual line ending wrangling.

Also if there is a BOM then the unicode encoding can and is worked for file: urls and open file for text. It’s when there’s no BOM and no explicit encoding specified that the engine defaults to legacy native encodings.

Cheers

Monte
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode
On 2/12/2018 8:49 PM, Monte Goulding via use-livecode wrote:

>
>> On 13 Feb 2018, at 11:56 am, Richard Gaskin via use-livecode <[hidden email]> wrote:
>>
>> something more sensible
> We already have:
>
> open file for “utf-8” text
>
> There was some discussion of implementing “file+<encoding>:/path/to/file” style urls to avoid having to use binfile and textDecode and manual line ending wrangling.
>
> Also if there is a BOM then the unicode encoding can and is worked for file: urls and open file for text. It’s when there’s no BOM and no explicit encoding specified that the engine defaults to legacy native encodings.

It is this last case (no BOM) that a 'guessEncoding(<file>)' function is
needed in LiveCode so those of us that make app that deal with end user
files (which can be of variable 'type' and whose encoding is generally
completely unknown to the user) do no have to each write our own.

I will happily offer the guessEncoding script we wrote from common
algorithms for this task if only LiveCode would add it to the engine. Or
any one else that has written a guessEncoding script could offer these
and LiveCode could pick the "best" one and add it to the engine.


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode
ICU has character set detection. Looks like it wouldn’t be that hard to add a function to return either the best match or an array of encoding -> confidence. Perhaps dd an enhancement request if there isn’t one already.

Cheers

Monte

> On 13 Feb 2018, at 1:04 pm, Paul Dupuis via use-livecode <[hidden email]> wrote:
>
> It is this last case (no BOM) that a 'guessEncoding(<file>)' function is
> needed in LiveCode so those of us that make app that deal with end user
> files (which can be of variable 'type' and whose encoding is generally
> completely unknown to the user) do no have to each write our own.
>
> I will happily offer the guessEncoding script we wrote from common
> algorithms for this task if only LiveCode would add it to the engine. Or
> any one else that has written a guessEncoding script could offer these
> and LiveCode could pick the "best" one and add it to the engine.

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Put URL into variable and textEncoding

Mike Kerner via use-livecode
In reply to this post by Mike Kerner via use-livecode
Monte Goulding wrote:

 >> On 13 Feb 2018, at 11:56 am, Richard Gaskin wrote:
 >> something more sensible
 >
 > We already have:
 >
 > open file for “utf-8” text
 >
 > There was some discussion of implementing “file+<encoding>:/path
 > /to/file” style urls to avoid having to use binfile and textDecode
 > and manual line ending wrangling.
 >
 > Also if there is a BOM then the unicode encoding can and is worked
 > for file: urls and open file for text. It’s when there’s no BOM and
 > no explicit encoding specified that the engine defaults to legacy
 > native encodings.

More reasonable indeed.  With all the complicated examples floating
around I'd forgotten about the "open file" addition.  Nice.

--
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  [hidden email]                http://www.FourthWorld.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode