Unicode Store in JSON

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Unicode Store in JSON

Geoff Canyon via use-livecode
I need to store unicode in JSON on Mobile; and store that in SQLLite dBase

If you do a "direct" transfer

*put*(char1to35oftQuote)& "..."intosTruncFirstLine

        "Yea, jīva is actually Śiva."

# putting sTruncFirstLine into an SQLLite column
# when you get the data out, it becomes

        "Yea, jƒ´va is actually ≈öiva...."

"textEncode" won't work because it produces binary code, can't store in
JSON...

base64 should work, but I get similar results,

*        put*(char1to35oftQuote)& "..."intosTruncFirstLine

*        put*base64Encode(sTruncFirstLine) intosTruncFirstLine

                  # restore later

*        put*base64Decode(pEntryA["label"]) intotLab*el**
*

*        set*thetextoffieldtName totLabel

                # returns "garble"

What are my other options?

BR


 



 




_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode
I'd store the text normally, without alteration, in the JSON. TextEncode
it when you send it to the database and textDecode when you retrieve it
from the database.

On 10/14/18 9:26 AM, Sannyasin Brahmanathaswami via use-livecode wrote:

> I need to store unicode in JSON on Mobile; and store that in SQLLite dBase
>
> If you do a "direct" transfer
>
> *put*(char1to35oftQuote)& "..."intosTruncFirstLine
>
>          "Yea, jīva is actually Śiva."
>
> # putting sTruncFirstLine into an SQLLite column
> # when you get the data out, it becomes
>
>          "Yea, jƒ´va is actually ≈öiva...."
>
> "textEncode" won't work because it produces binary code, can't store in
> JSON...
>
> base64 should work, but I get similar results,
>
> *        put*(char1to35oftQuote)& "..."intosTruncFirstLine
>
> *        put*base64Encode(sTruncFirstLine) intosTruncFirstLine
>
>                    # restore later
>
> *        put*base64Decode(pEntryA["label"]) intotLab*el**
> *
>
> *        set*thetextoffieldtName totLabel
>
>                  # returns "garble"
>
> What are my other options?

--
Jacqueline Landman Gay         |     [hidden email]
HyperActive Software           |     http://www.hyperactivesw.com


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode
On 10/14/18 8:43 AM, J. Landman Gay via use-livecode wrote:
> I'd store the text normally, without alteration, in the JSON. TextEncode
> it when you send it to the database and textDecode when you retrieve it
> from the database.

Wow, that worked! And I thought that JSON could not be a container for
"binaryData" as described on the dictionary.

BR


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode


> On 15 Oct 2018, at 11:35 am, Sannyasin Brahmanathaswami via use-livecode <[hidden email]> wrote:
>
> Wow, that worked! And I thought that JSON could not be a container for
> "binaryData" as described on the dictionary.

Could you provide some detail where the docs are misleading. It is actually true that you can’t JSON encode binary data.

We do have different libraries that treat JSON text encoding differently. mergJSON and therefore ArrayToJSON and JSONToArray deal with UTF-8 encoded JSON only. ArrayToJSON and JSONToArray will do this encoding for you if you give it text strings. If you give it binary strings then they must be UTF-8 encoded already otherwise mergJSON will throw an error.

JSONImport and JSONExport on the other hand deals with JSON as text so you need to textDecode/textDecode as a separate operation. If you give any LCB API binary strings when it is expecting text strings then it will follow the engine’s standard conversion which will be native encoding (eg ISO/MacRoman).

Note while the previous JSON specs just say unicode encoding can be used RFC8259 specifies UTF-8 encoding of JSON between systems.

Cheers

Monte
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode
First can you clarify if both of these should work.? We see both in
dictionary...

*Example*

textEncode("A","UTF16")

*put*textEncode(field"output","UTF-8") intotOutput

will the 2nd param work with or without a hyphen?

-----------
unclarity --simply:

*Syntax*

textEncode(*stringToEncode*, encoding)

*Summary*

Converts from text to binary data.

========

doesn't stipulate an "exception" for

"different libraries that treat JSON text encoding differently. mergJSON [snip] "


BR




On 10/14/18 2:55 PM, Monte Goulding via use-livecode wrote:
> Could you provide some detail where the docs are misleading. It is actually true that you can’t JSON encode binary data.
>
> We do have different libraries that treat JSON text encoding differently. mergJSON and therefore ArrayToJSON and JSONToArray deal with UTF-8 encoded JSON only. ArrayToJSON and JSONToArray will do this encoding for you if you give it text strings. If you give it binary strings then they must be UTF-8 encoded already otherwise mergJSON will throw an error.
>
> JSONImport and JSONExport on the other hand deals with JSON as text so you need to textDecode/textDecode as a separate operation. If you give any LCB API binary strings when it is expecting text strings then it will follow the engine’s standard conversion which will be native encoding (eg ISO/MacRoman).
>
> Note while the previous JSON specs just say unicode encoding can be used RFC8259 specifies UTF-8 encoding of JSON between systems.
>
> Cheers


--
Svasti Astu, Be Well!
Brahmanathaswami

Get the SivaSiva app, it's free:
https://www.himalayanacademy.com/apps/sivasiva

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode


> On 15 Oct 2018, at 2:31 pm, Sannyasin Brahmanathaswami via use-livecode <[hidden email]> wrote:
>
> First can you clarify if both of these should work.? We see both in
> dictionary...
>
> *Example*
>
> textEncode("A","UTF16")
>
> *put*textEncode(field"output","UTF-8") intotOutput
>
> will the 2nd param work with or without a hyphen?

Yes

>
> -----------
> unclarity --simply:
>
> *Syntax*
>
> textEncode(*stringToEncode*, encoding)
>
> *Summary*
>
> Converts from text to binary data.
>
> ========
>
> doesn't stipulate an "exception" for
>
> "different libraries that treat JSON text encoding differently. mergJSON [snip] “

Well mergJSON is documented to return UTF-8 encoded JSON, however, I agree that JSONImport and JSONExport docs are a little ambiguous and should clarify that the JSON needs to be textDecoded before calling JSONImport and textEncoded after calling JSONExport.

Cheers

Monte


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode
On 10/14/18 5:41 PM, Monte Goulding via use-livecode wrote:
> Well mergJSON is documented to return UTF-8 encoded JSON, however, I agree that JSONImport and JSONExport docs are a little ambiguous and should clarify that the JSON needs to be textDecoded before calling JSONImport and textEncoded after calling JSONExport.
A "Related" entry under textDecode and textEncode might help.

The problem is when you get to a dictionary entry for those, there is no
mention "mergJSON".

So not matter how well "mergeJSON" is documented, as  a "newbie" or
someone who hasn't spend a lot of time in text encoding procedures, one
is "blindsided" to the "mergeJSON" feature.

There is a lacuna in the dictionary, not only in this space, but
throughout. I won't all it a "fault", because in some way "all livecode
is related" , so where to begin writing about "related, associations"?

It is probably best done on a use case basis, like this one.

I am getting better at GIT now. I could contribute to the dictionary if
I knew how.

BR

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode


> On 16 Oct 2018, at 12:09 am, Sannyasin Brahmanathaswami via use-livecode <[hidden email]> wrote:
>
> A "Related" entry under textDecode and textEncode might help.
>
> The problem is when you get to a dictionary entry for those, there is no
> mention "mergJSON".
>
> So not matter how well "mergeJSON" is documented, as  a "newbie" or
> someone who hasn't spend a lot of time in text encoding procedures, one
> is "blindsided" to the "mergeJSON" feature.

How so. Or do you mean that perhaps they both need to link to a general discussion on text encoding? There are a number of things that deal with text encoding/decoding in the docs so perhaps a general discussion they can all link to would be helpful. I presume until now it has been seen as beyond the scope of our docs but text encoding is definitely something that confuses many users. Something along the lines of the content of this article would probably be good:

https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
>
> I am getting better at GIT now. I could contribute to the dictionary if
> I knew how.

Well you can get a free t-shirt this month if you do! Check out Trevor’s recent post.

Cheers

Monte
_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode
In reply to this post by Geoff Canyon via use-livecode
Unicode is binary??

Bob S


> On Oct 14, 2018, at 17:35 , Sannyasin Brahmanathaswami via use-livecode <[hidden email]> wrote:
>
> On 10/14/18 8:43 AM, J. Landman Gay via use-livecode wrote:
>> I'd store the text normally, without alteration, in the JSON. TextEncode
>> it when you send it to the database and textDecode when you retrieve it
>> from the database.
>
> Wow, that worked! And I thought that JSON could not be a container for
> "binaryData" as described on the dictionary.
>
> BR


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|

Re: Unicode Store in JSON

Geoff Canyon via use-livecode
Unicode is binary??
> Bob S

not unicode per se Dictionary says

*Syntax*

textEncode(*stringToEncode*, encoding)

*Summary*

Converts from text to binary data.



_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode