Don't understand unicode handling

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Don't understand unicode handling

Tiemo Hollmann TB
Hello,

I have a file exported from an older genealogy program A, which I want to
import into another genealogy program B. Program B is pretty new and modern
and is supposed to support Unicode.

When directly importing that exchange file, all special chars get corrupted
in program B.

When reading the export file into a field of a LC 8.0 program and writing
the field with LC via: put fld "Out" into URL "file:myFile"

and importing that LC file into program B the special chars are still
corrupted.

My thinking was, that in the moment, you get data into a field in LC 8.0, LC
converts it automatically into Unicode and so it would be also Unicode, when
writing this field into a file.

 

I am also using UltraEdit as an editor. Opening the exported file I can
convert the data "from ASCII to UTF-8 (Unicode editing)". If I do that with
the exported file, all special chars are fine, when importing the file into
program B

 

Do I still have to convert data with LC 8? Where am I thinking wrong?

Thanks for a push into the right direction

Tiemo

 

 

 

 

 

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Don't understand unicode handling

Thierry Douez
Hallo Tiemo,


from: http://lessons.livecode.com/m/4071/l/12304-how-do-i-use-unicode-in-rev


To import text from a UTF-8 file and display it in our stack we just need
to read it in, uniEncode() it and then display it as we learnt earlier.

*put* url ("file:greek.txt") into tUnicodeText

*set* the unicodeText of field 1 to uniEncode(tUnicodeText,"UTF8")

If we want to export text from our stack we simply go in the other direction

*put* uniDecode(the unicodeText of field 1,"UTF8") into tUTF8Text

*put* tUTF8Text into url("file:greek2.txt")



Guess it's what you need....


Regards,

Thierry

------------------------------------------------
Thierry Douez - http://sunny-tdz.com
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

AW: Don't understand unicode handling

Tiemo Hollmann TB
Bonsoir Thierry,

thank you for the helpful link, with your help, I found the right
conversion, though I don't understand what is going on :)
For my issue the solution was not to use uniEncode() when importing the file
into LC, but just putting the file into a field.
But using uniDecode() when exporting it from LC.

I think my problem is that you don't see what type of coding a file has. If
you have to handle with unknown data, you just have to try the different
combinations of encode/decode/do nothing with the data, because a file has
no label "I am UTF-8".

Thanks
Tiemo

-----Ursprüngliche Nachricht-----
Von: use-livecode [mailto:[hidden email]] Im Auftrag
von Thierry Douez
Gesendet: Freitag, 20. Mai 2016 19:07
An: How to use LiveCode <[hidden email]>
Betreff: Re: Don't understand unicode handling

Hallo Tiemo,


from: http://lessons.livecode.com/m/4071/l/12304-how-do-i-use-unicode-in-rev


To import text from a UTF-8 file and display it in our stack we just need to
read it in, uniEncode() it and then display it as we learnt earlier.

*put* url ("file:greek.txt") into tUnicodeText

*set* the unicodeText of field 1 to uniEncode(tUnicodeText,"UTF8")

If we want to export text from our stack we simply go in the other direction

*put* uniDecode(the unicodeText of field 1,"UTF8") into tUTF8Text

*put* tUTF8Text into url("file:greek2.txt")



Guess it's what you need....


Regards,

Thierry

------------------------------------------------
Thierry Douez - http://sunny-tdz.com
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: AW: Don't understand unicode handling

J. Landman Gay
On 5/20/2016 12:53 PM, Tiemo Hollmann TB wrote:

> Bonsoir Thierry,
>
> thank you for the helpful link, with your help, I found the right
> conversion, though I don't understand what is going on :)
> For my issue the solution was not to use uniEncode() when importing the file
> into LC, but just putting the file into a field.
> But using uniDecode() when exporting it from LC.
>
> I think my problem is that you don't see what type of coding a file has. If
> you have to handle with unknown data, you just have to try the different
> combinations of encode/decode/do nothing with the data, because a file has
> no label "I am UTF-8".

In LC 7 and above, the old unicode syntax is deprecated and the new
encoding syntax is easier to use. Whenever you import or export text you
need to translate it using the new textEncode and textDecode functions.

To bring unicode text into LC:

   get textDecode(<filepath>,"UTF8")

and to export it:

   put textEncode(tText,"UTF8") into file <filepath>

If you are not sure what the unicode encoding is, it is almost always
safe to assume UTF-8. The dictionary has examples.

--
Jacqueline Landman Gay         |     [hidden email]
HyperActive Software           |     http://www.hyperactivesw.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

AW: AW: Don't understand unicode handling

Tiemo Hollmann TB
Hi Jacque,
thanks for the function update.
But if you don't know, what kind of encoding the source file is, it's still
trial and error, or is there any kind of "quickcheck" for the current source
encoding?
Tiemo

-----Ursprüngliche Nachricht-----
Von: use-livecode [mailto:[hidden email]] Im Auftrag
von J. Landman Gay
Gesendet: Freitag, 20. Mai 2016 20:05
An: How to use LiveCode <[hidden email]>
Betreff: Re: AW: Don't understand unicode handling

On 5/20/2016 12:53 PM, Tiemo Hollmann TB wrote:

> Bonsoir Thierry,
>
> thank you for the helpful link, with your help, I found the right
> conversion, though I don't understand what is going on :) For my issue
> the solution was not to use uniEncode() when importing the file into
> LC, but just putting the file into a field.
> But using uniDecode() when exporting it from LC.
>
> I think my problem is that you don't see what type of coding a file
> has. If you have to handle with unknown data, you just have to try the
> different combinations of encode/decode/do nothing with the data,
> because a file has no label "I am UTF-8".

In LC 7 and above, the old unicode syntax is deprecated and the new encoding
syntax is easier to use. Whenever you import or export text you need to
translate it using the new textEncode and textDecode functions.

To bring unicode text into LC:

   get textDecode(<filepath>,"UTF8")

and to export it:

   put textEncode(tText,"UTF8") into file <filepath>

If you are not sure what the unicode encoding is, it is almost always safe
to assume UTF-8. The dictionary has examples.

--
Jacqueline Landman Gay         |     [hidden email]
HyperActive Software           |     http://www.hyperactivesw.com

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: AW: AW: Don't understand unicode handling

Paul Dupuis
I spoke to Fraser when LC7 came out about a lack of a guessTextEncoding
function.

See http://quality.livecode.com/show_bug.cgi?id=14474



On 5/23/2016 3:08 AM, Tiemo Hollmann TB wrote:

> Hi Jacque,
> thanks for the function update.
> But if you don't know, what kind of encoding the source file is, it's still
> trial and error, or is there any kind of "quickcheck" for the current source
> encoding?
> Tiemo
>
> -----Ursprüngliche Nachricht-----
> Von: use-livecode [mailto:[hidden email]] Im Auftrag
> von J. Landman Gay
> Gesendet: Freitag, 20. Mai 2016 20:05
> An: How to use LiveCode <[hidden email]>
> Betreff: Re: AW: Don't understand unicode handling
>
> On 5/20/2016 12:53 PM, Tiemo Hollmann TB wrote:
>> Bonsoir Thierry,
>>
>> thank you for the helpful link, with your help, I found the right
>> conversion, though I don't understand what is going on :) For my issue
>> the solution was not to use uniEncode() when importing the file into
>> LC, but just putting the file into a field.
>> But using uniDecode() when exporting it from LC.
>>
>> I think my problem is that you don't see what type of coding a file
>> has. If you have to handle with unknown data, you just have to try the
>> different combinations of encode/decode/do nothing with the data,
>> because a file has no label "I am UTF-8".
> In LC 7 and above, the old unicode syntax is deprecated and the new encoding
> syntax is easier to use. Whenever you import or export text you need to
> translate it using the new textEncode and textDecode functions.
>
> To bring unicode text into LC:
>
>    get textDecode(<filepath>,"UTF8")
>
> and to export it:
>
>    put textEncode(tText,"UTF8") into file <filepath>
>
> If you are not sure what the unicode encoding is, it is almost always safe
> to assume UTF-8. The dictionary has examples.
>


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

AW: AW: AW: Don't understand unicode handling

Tiemo Hollmann TB
Yes, that function is exactly, what I was looking for.
Fraser told to provide such a function, but I didn't see it yet.
Tiemo


-----Ursprüngliche Nachricht-----
Von: use-livecode [mailto:[hidden email]] Im Auftrag
von Paul Dupuis
Gesendet: Montag, 23. Mai 2016 13:40
An: [hidden email]
Betreff: Re: AW: AW: Don't understand unicode handling

I spoke to Fraser when LC7 came out about a lack of a guessTextEncoding
function.

See http://quality.livecode.com/show_bug.cgi?id=14474



On 5/23/2016 3:08 AM, Tiemo Hollmann TB wrote:

> Hi Jacque,
> thanks for the function update.
> But if you don't know, what kind of encoding the source file is, it's
> still trial and error, or is there any kind of "quickcheck" for the
> current source encoding?
> Tiemo
>
> -----Ursprüngliche Nachricht-----
> Von: use-livecode [mailto:[hidden email]] Im
> Auftrag von J. Landman Gay
> Gesendet: Freitag, 20. Mai 2016 20:05
> An: How to use LiveCode <[hidden email]>
> Betreff: Re: AW: Don't understand unicode handling
>
> On 5/20/2016 12:53 PM, Tiemo Hollmann TB wrote:
>> Bonsoir Thierry,
>>
>> thank you for the helpful link, with your help, I found the right
>> conversion, though I don't understand what is going on :) For my
>> issue the solution was not to use uniEncode() when importing the file
>> into LC, but just putting the file into a field.
>> But using uniDecode() when exporting it from LC.
>>
>> I think my problem is that you don't see what type of coding a file
>> has. If you have to handle with unknown data, you just have to try
>> the different combinations of encode/decode/do nothing with the data,
>> because a file has no label "I am UTF-8".
> In LC 7 and above, the old unicode syntax is deprecated and the new
> encoding syntax is easier to use. Whenever you import or export text
> you need to translate it using the new textEncode and textDecode
functions.

>
> To bring unicode text into LC:
>
>    get textDecode(<filepath>,"UTF8")
>
> and to export it:
>
>    put textEncode(tText,"UTF8") into file <filepath>
>
> If you are not sure what the unicode encoding is, it is almost always
> safe to assume UTF-8. The dictionary has examples.
>


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: AW: AW: AW: Don't understand unicode handling

Paul Dupuis
Unfortunately, no, one was never posted.

We're writing our own (since we needed one). It is still being tested
and refined, but when "done" (perhaps another couple weeks and
acknowledging that such a routine will never be 100% accurate), I will
try to post it to the bug report.


On 5/23/2016 8:07 AM, Tiemo Hollmann TB wrote:

> Yes, that function is exactly, what I was looking for.
> Fraser told to provide such a function, but I didn't see it yet.
> Tiemo
>
>
> -----Ursprüngliche Nachricht-----
> Von: use-livecode [mailto:[hidden email]] Im Auftrag
> von Paul Dupuis
> Gesendet: Montag, 23. Mai 2016 13:40
> An: [hidden email]
> Betreff: Re: AW: AW: Don't understand unicode handling
>
> I spoke to Fraser when LC7 came out about a lack of a guessTextEncoding
> function.
>
> See http://quality.livecode.com/show_bug.cgi?id=14474
>
>
>
> On 5/23/2016 3:08 AM, Tiemo Hollmann TB wrote:
>> Hi Jacque,
>> thanks for the function update.
>> But if you don't know, what kind of encoding the source file is, it's
>> still trial and error, or is there any kind of "quickcheck" for the
>> current source encoding?
>> Tiemo
>>
>> -----Ursprüngliche Nachricht-----
>> Von: use-livecode [mailto:[hidden email]] Im
>> Auftrag von J. Landman Gay
>> Gesendet: Freitag, 20. Mai 2016 20:05
>> An: How to use LiveCode <[hidden email]>
>> Betreff: Re: AW: Don't understand unicode handling
>>
>> On 5/20/2016 12:53 PM, Tiemo Hollmann TB wrote:
>>> Bonsoir Thierry,
>>>
>>> thank you for the helpful link, with your help, I found the right
>>> conversion, though I don't understand what is going on :) For my
>>> issue the solution was not to use uniEncode() when importing the file
>>> into LC, but just putting the file into a field.
>>> But using uniDecode() when exporting it from LC.
>>>
>>> I think my problem is that you don't see what type of coding a file
>>> has. If you have to handle with unknown data, you just have to try
>>> the different combinations of encode/decode/do nothing with the data,
>>> because a file has no label "I am UTF-8".
>> In LC 7 and above, the old unicode syntax is deprecated and the new
>> encoding syntax is easier to use. Whenever you import or export text
>> you need to translate it using the new textEncode and textDecode
> functions.
>> To bring unicode text into LC:
>>
>>    get textDecode(<filepath>,"UTF8")
>>
>> and to export it:
>>
>>    put textEncode(tText,"UTF8") into file <filepath>
>>
>> If you are not sure what the unicode encoding is, it is almost always
>> safe to assume UTF-8. The dictionary has examples.
>>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

AW: AW: AW: AW: Don't understand unicode handling

Tiemo Hollmann TB
I would appreciate it!

-----Ursprüngliche Nachricht-----
Von: use-livecode [mailto:[hidden email]] Im Auftrag
von Paul Dupuis
Gesendet: Montag, 23. Mai 2016 14:25
An: [hidden email]
Betreff: Re: AW: AW: AW: Don't understand unicode handling

Unfortunately, no, one was never posted.

We're writing our own (since we needed one). It is still being tested and
refined, but when "done" (perhaps another couple weeks and acknowledging
that such a routine will never be 100% accurate), I will try to post it to
the bug report.


On 5/23/2016 8:07 AM, Tiemo Hollmann TB wrote:

> Yes, that function is exactly, what I was looking for.
> Fraser told to provide such a function, but I didn't see it yet.
> Tiemo
>
>
> -----Ursprüngliche Nachricht-----
> Von: use-livecode [mailto:[hidden email]] Im
> Auftrag von Paul Dupuis
> Gesendet: Montag, 23. Mai 2016 13:40
> An: [hidden email]
> Betreff: Re: AW: AW: Don't understand unicode handling
>
> I spoke to Fraser when LC7 came out about a lack of a
> guessTextEncoding function.
>
> See http://quality.livecode.com/show_bug.cgi?id=14474
>
>
>
> On 5/23/2016 3:08 AM, Tiemo Hollmann TB wrote:
>> Hi Jacque,
>> thanks for the function update.
>> But if you don't know, what kind of encoding the source file is, it's
>> still trial and error, or is there any kind of "quickcheck" for the
>> current source encoding?
>> Tiemo
>>
>> -----Ursprüngliche Nachricht-----
>> Von: use-livecode [mailto:[hidden email]] Im
>> Auftrag von J. Landman Gay
>> Gesendet: Freitag, 20. Mai 2016 20:05
>> An: How to use LiveCode <[hidden email]>
>> Betreff: Re: AW: Don't understand unicode handling
>>
>> On 5/20/2016 12:53 PM, Tiemo Hollmann TB wrote:
>>> Bonsoir Thierry,
>>>
>>> thank you for the helpful link, with your help, I found the right
>>> conversion, though I don't understand what is going on :) For my
>>> issue the solution was not to use uniEncode() when importing the
>>> file into LC, but just putting the file into a field.
>>> But using uniDecode() when exporting it from LC.
>>>
>>> I think my problem is that you don't see what type of coding a file
>>> has. If you have to handle with unknown data, you just have to try
>>> the different combinations of encode/decode/do nothing with the
>>> data, because a file has no label "I am UTF-8".
>> In LC 7 and above, the old unicode syntax is deprecated and the new
>> encoding syntax is easier to use. Whenever you import or export text
>> you need to translate it using the new textEncode and textDecode
> functions.
>> To bring unicode text into LC:
>>
>>    get textDecode(<filepath>,"UTF8")
>>
>> and to export it:
>>
>>    put textEncode(tText,"UTF8") into file <filepath>
>>
>> If you are not sure what the unicode encoding is, it is almost always
>> safe to assume UTF-8. The dictionary has examples.
>>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
> subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode