Proposal: standardised text based stack file format

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal: standardised text based stack file format

david bovill
Hi everyone - and a belated Happy New Year to all!

I've spent the last few days, hacking and going over XML, array and text
based stack file formats for version control. I've now got 3 interchangeable
formats that I think should suite most needs, and would like to propose them
to the group.

First I thought I'd give a heads up, and outline the approach, and seek some
early advice. The approach I have taken can be broken down to the following:

   1. Start with an array based format - use this internally for fast and
   flexible processing
   2. No duplication of content - backgrounds listed only once.
   3. Be able to create arrays from individual controls, nested groups,
   cards or entire stacks
   4. Be able to compose new objects from different arrays easily and
   quickly
   5. A standard way to export and import the arrays to XML
   6. A standard way to export and import the arrays to nested folders and
   text files

This give 3 file formats - an encoded array, an XML based structure, and a
folder based structure (which can be zipped), all based on

I've just finished a first rewrite - so I've got working code for all of the
above, and have tested creating and recreatign stacks with substacks,
multiple cards and backgrounds, with custom props, and binary image, and
video data.

Limitations so far include issues with audioclip export, but videoclips
work. Not tested EPS objects.

Considering exporting audioclips by copying them to a new empty stack and
savign the binary for now.

Does anyone have any gotchas' - requirements or things to look out for, that
would be real handy. I'm going to present the work at the next LiveCode TV
session, and then publish the code and spec with the talk.

I see this text based format as a foundation for collaboration - so that
several people can be working on a single stack at the same time, and also
for being able to quantify how much different people have contributed to the
code base and designs. This can help in acknowledging contributors work or
even making revenue sharing agreements based on differential contributions.


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

mwieder
David-

Friday, January 7, 2011, 11:11:32 AM, you wrote:

> Does anyone have any gotchas' - requirements or things to look out for, that
> would be real handy. I'm going to present the work at the next LiveCode TV
> session, and then publish the code and spec with the talk.

A few: remember to take care of purely numeric custom props - I've
come across cases where folks use timestamps as keys of custom
propertysets, and used verbatim they don't make for valid xml tags.
Any xml parser will barf coming across one of these, and revxml is no
exception.

Also, watch for punctuation in custom prop names. This was probably
more prevalent before we had multidimensional arrays, but the
standalone builder, for one, is a notorious misuser of commas embedded
in custom property names.

And non-printable chars (cr and such) are also problems within xml
data (the htmltext, etc). I've taken to base64encoding all the above
with special tags to decode them on the way out. PITA.

--
-Mark
 [hidden email]

--
 Mark Wieder
 ahsoftware@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

david bovill
Thanks Mark - yes so it's the export of custom props to XML which is the
(AFIAK only) problem. It's one of the reasons that the XML stuff is a pain.
I think the solution is to simply create a binary for the custom properties
by placing all of them in an array and encoding it - then putting this
binary in the XML.

I'm not going to use the XML I think anyway, as the combination of the array
of folder based formats are much better for day-to-day use. The XML is for
interchange with other (non LiveCode) projects that may not be able to parse
the array format - which seems a pretty unlikely use case.

Seem reasonable?

The other way out I think is to use a pList type XML format - whcih when I
first looked at seemed so ugly, but I think would work for arrays with
unusual key names?

On 8 January 2011 02:23, Mark Wieder <[hidden email]> wrote:

> David-
>
> Friday, January 7, 2011, 11:11:32 AM, you wrote:
>
> > Does anyone have any gotchas' - requirements or things to look out for,
> that
> > would be real handy. I'm going to present the work at the next LiveCode
> TV
> > session, and then publish the code and spec with the talk.
>
> A few: remember to take care of purely numeric custom props - I've
> come across cases where folks use timestamps as keys of custom
> propertysets, and used verbatim they don't make for valid xml tags.
> Any xml parser will barf coming across one of these, and revxml is no
> exception.
>
> Also, watch for punctuation in custom prop names. This was probably
> more prevalent before we had multidimensional arrays, but the
> standalone builder, for one, is a notorious misuser of commas embedded
> in custom property names.
>
> And non-printable chars (cr and such) are also problems within xml
> data (the htmltext, etc). I've taken to base64encoding all the above
> with special tags to decode them on the way out. PITA.
>
> --
> -Mark
>  [hidden email]
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

mwieder
David-

Saturday, January 8, 2011, 2:46:35 AM, you wrote:

> Thanks Mark - yes so it's the export of custom props to XML which is the
> (AFIAK only) problem. It's one of the reasons that the XML stuff is a pain.

Or rather it's the other way around - exporting to xml is easy, but
reading it back in is where you run into problems. Yes, xml is a pain,
but its advantage is that it's a well-defined standard format that
even Microsoft's extensions can't mess up.

--
-Mark
 [hidden email]

--
 Mark Wieder
 ahsoftware@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

Monte Goulding
In reply to this post by david bovill

>
> I'm not going to use the XML I think anyway, as the combination of the array
> of folder based formats are much better for day-to-day use. The XML is for
> interchange with other (non LiveCode) projects that may not be able to parse
> the array format - which seems a pretty unlikely use case.
>
> Seem reasonable?
>
> No, why would a non livecode project need to parse any kind of stack file format? Why not drop it altogether?




Cheers

Monte
>
>


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

mwieder
Monte-

Saturday, January 8, 2011, 1:58:45 PM, you wrote:

>> No, why would a non livecode project need to parse any kind of
>> stack file format? Why not drop it altogether?

Being able to diff object properties with standard diff tools?

--
-Mark
 [hidden email]

--
 Mark Wieder
 ahsoftware@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

Monte Goulding


>
> >> No, why would a non livecode project need to parse any kind of
> >> stack file format? Why not drop it altogether?
>
> Being able to diff object properties with standard diff tools?
>
I was referring to the XML version which seems to be an uneccesary complication. I know the value of a text based stack file format. The one I like is just single file for each property and the control hierarchy represented by folders.

Cheers

Monte
>


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

david bovill
On 8 January 2011 23:52, Monte Goulding <[hidden email]> wrote:

> >
> > >> No, why would a non livecode project need to parse any kind of
> > >> stack file format? Why not drop it altogether?
> >
> > Being able to diff object properties with standard diff tools?
> >
> I was referring to the XML version which seems to be an uneccesary
> complication. I know the value of a text based stack file format. The one I
> like is just single file for each property and the control hierarchy
> represented by folders.
>

Yes - so we agree. I just include the XML as an option for those that seem
to think it useful. It's given for free as using a standard array_to_xml
function you always have that option (as long that is as you are a little
careful about the key names).

I could go back to the non-xml compatible structure - as there are a few
less key renaming hoops...


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

mwieder
In reply to this post by Monte Goulding
Monte-

Saturday, January 8, 2011, 3:52:28 PM, you wrote:

> I was referring to the XML version which seems to be an
> uneccesary complication. I know the value of a text based stack file
> format. The one I like is just single file for each property and the
> control hierarchy represented by folders.

OK - that isn't what you asked, though. In that case I don't see any
real advantage to xml over anything else. A file per property seems
like a bit of overkill - is there any advantage to that over a single
text file with a list of key/value pairs? It seems that a single file
would make diffing an easier task.

--
-Mark
 [hidden email]

--
 Mark Wieder
 ahsoftware@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

Monte Goulding

>
> > I was referring to the XML version which seems to be an
> > uneccesary complication. I know the value of a text based stack file
> > format. The one I like is just single file for each property and the
> > control hierarchy represented by folders.
>
> OK - that isn't what you asked, though. In that case I don't see any
> real advantage to xml over anything else. A file per property seems
> like a bit of overkill - is there any advantage to that over a single
> text file with a list of key/value pairs? It seems that a single file
> would make diffing an easier task.
>
I try not to be to pedantic but if you read the section of Davids post that i snipped you will see I was referring to the XML form and his statements about it.

Cheers

Monte

>


[Non-text portions of this message have been removed]

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

mwieder
Monte-

Saturday, January 8, 2011, 5:46:00 PM, you wrote:

> I try not to be to pedantic but if you read the section of Davids
> post that i snipped you will see I was referring to the XML form and
> his statements about it.

Having now gone back over this again, *no*, you asked "why would a non
livecode project need to parse any kind of stack file format?", and
that's what I replied to. So if the question is about why use a
non-native-livecode storage format, then I can see the case for
something that can be read by standard external tools. If the question
is about xml in particular then I don't have an argument for promoting
xml. For my own tools I originally went with straight text, then found
a problem with nesting groups so I went with xml for its hierarchical
format. But David seems to have sidestepped that nicely, and I don't
have any particular love of xml, given that it's slow and ugly and
bloated and can't have numeric or multi-word tags and has trouble with
embedded punctuation and unicode... did I mention that it's slow?

--
-Mark
 [hidden email]

--
 Mark Wieder
 ahsoftware@gmail.com
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: standardised text based stack file format

Monte Goulding
>
> Having now gone back over this again, *no*, you asked "why would a non
> livecode project need to parse any kind of stack file format?
>
>
Whatever, I knew what I was talking about, I guess I assumed others would understand the snip which was entirely about the pointlessness of an xml format provided the context for my question but obviously not. I think my last email cleared up the confusion so why are we still discussing this?

Cheers

Monte

[Non-text portions of this message have been removed]