OMG WTF detailed files BST?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

OMG WTF detailed files BST?

Monte Goulding via use-livecode
I have an interesting situation. The detailed files is reporting a LIE in
modification dates.

I have a standalone app which outputs a report including the modification date
of a bunch of files (and subsequently interprets and acts on it).

On Monday, a BAD THING happened at my client's side. On investigation it
turned out to be because on Sunday night my software (which runs every night)
decided that a very large number of files had been modified since the last
time it checked, and took some actions in response.

The files hadn't changed, but the clocks in the UK have - they went forward
one hour on Sunday morning, because of British Summer Time (BST).

However, a file that was last modified, for example, at 8:40am on July 27,
2015 should not therefore be reported as having been modified at 9:40am on
July 27, 2015. But that's essentially what happened.

The app dumps the mod date from the "detailed files" directly to file, i.e. a
value for the seconds since 1 Jan 1970. This isn't about how the time was
displayed for humans, but a direct comparison of the value returned by
'detailed files': on Sunday night, each value had increased by 3600.

The standalone, built from LC 6.7.11, is running on a Windows machine (VM
running Windows Server 2008 R2) in London, which has a volume mounted from a
server running an unknown operating system.

Essentially the same software has been running every night for several years
on a Windows VM in America, and does not appear to have encountered this issue
twice a year. I initially blamed STUPID WINDOWS for this screw-up, and guessed
it might be some issue to do with the respective file-servers, and an
operating system somewhere second-guessing the other one and messing up by
adjusting redundantly for time-zone issues.

However, when I actually logged into the machine and used Windows Explorer to
view the files on the mounted drive, it shows the example file as modified at
8:40am on that day. Which points the finger at LiveCode (and makes the fact
that the US install hasn't encountered this issue stranger).

Is there some error in the code by which LiveCode converts the file timestamp
that it gets from Windows, to the unix style seconds-since-1/1/1970 - taking
account of timezones and daylight saving times when it shouldn't?  Is there a
property somewhere - in LiveCode, or on an Windows server - which can be set
to avoid this issue?

Ben

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OMG WTF detailed files BST?

Monte Goulding via use-livecode
Hi Ben,

This might be of interest:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx

On 2017-03-31 12:48, Ben Rubinstein via use-livecode wrote:
> The standalone, built from LC 6.7.11, is running on a Windows machine
> (VM running Windows Server 2008 R2) in London, which has a volume
> mounted from a server running an unknown operating system.

What is the filesystem of the volume? The above document suggests that
FAT stores
filetimes in local time and *not* universal time which sounds like it
might be the problem...

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OMG WTF detailed files BST?

Monte Goulding via use-livecode
On 2017-03-31 13:20, Mark Waddingham via use-livecode wrote:

> Hi Ben,
>
> This might be of interest:
>
> https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx
>
> On 2017-03-31 12:48, Ben Rubinstein via use-livecode wrote:
>> The standalone, built from LC 6.7.11, is running on a Windows machine
>> (VM running Windows Server 2008 R2) in London, which has a volume
>> mounted from a server running an unknown operating system.
>
> What is the filesystem of the volume? The above document suggests that
> FAT stores
> filetimes in local time and *not* universal time which sounds like it
> might be the problem...

I just re-read the pertinent paragraph:

The FAT file system records times on disk in local time. GetFileTime
retrieves cached UTC times from the FAT file system. When it becomes
daylight saving time, the time retrieved by GetFileTime is off an hour,
because the cache is not updated. When you restart the computer, the
cached time that GetFileTime retrieves is correct. FindFirstFile
retrieves the local time from the FAT file system and converts it to UTC
by using the current settings for the time zone and daylight saving
time. Therefore, if it is daylight saving time, FindFirstFile takes
daylight saving time into account, even if the file time you are
converting is in standard time.

Sounds like a restart is needed to ensure that the APIs the engine has
to use are correct...

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OMG WTF detailed files BST?

Monte Goulding via use-livecode
Hi Mark,

Thanks for your response, very helpful.

Unfortunately I don't know what the filesystem of the volume with the files is
- this is a client's network, and they set up a VM in the DMZ for me to work
on, with this network share mounted - but I've no idea what the fileserver is.

The thing that puzzles me is why the Windows Explorer is reporting the mod
date of these files correctly, while LC isn't.

However, I've now restarted the VM - no change to how Windows Explorer reports
the dates; and then told it not to adjust for daylight saving time and
restarted it again: the clock on the machine went back by an hour, but the
time reported by Windows for the files on the network share didn't change.

I'll see what happens when the job runs overnight, whether one or both these
things fixes the timestamps that LC gets...

Ben

On 31/03/2017 12:22, Mark Waddingham via use-livecode wrote:

> On 2017-03-31 13:20, Mark Waddingham via use-livecode wrote:
>> Hi Ben,
>>
>> This might be of interest:
>>
>> https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx
>>
>> On 2017-03-31 12:48, Ben Rubinstein via use-livecode wrote:
>>> The standalone, built from LC 6.7.11, is running on a Windows machine
>>> (VM running Windows Server 2008 R2) in London, which has a volume
>>> mounted from a server running an unknown operating system.
>>
>> What is the filesystem of the volume? The above document suggests that
>> FAT stores
>> filetimes in local time and *not* universal time which sounds like it
>> might be the problem...
>
> I just re-read the pertinent paragraph:
>
> The FAT file system records times on disk in local time. GetFileTime retrieves
> cached UTC times from the FAT file system. When it becomes daylight saving
> time, the time retrieved by GetFileTime is off an hour, because the cache is
> not updated. When you restart the computer, the cached time that GetFileTime
> retrieves is correct. FindFirstFile retrieves the local time from the FAT file
> system and converts it to UTC by using the current settings for the time zone
> and daylight saving time. Therefore, if it is daylight saving time,
> FindFirstFile takes daylight saving time into account, even if the file time
> you are converting is in standard time.
>
> Sounds like a restart is needed to ensure that the APIs the engine has to use
> are correct...
>
> Mark.
>


_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OMG WTF detailed files BST?

Monte Goulding via use-livecode
On 2017-03-31 19:23, Local (BenR) wrote:
> Hi Mark,
>
> Thanks for your response, very helpful.
>
> Unfortunately I don't know what the filesystem of the volume with the
> files is - this is a client's network, and they set up a VM in the DMZ
> for me to work on, with this network share mounted - but I've no idea
> what the fileserver is.

Hmmm - I suspect something to do with the fileserver is at play here.

LiveCode uses FindFirstFileW and similar Win32 APIs to fetch the
detailed file
into. Beyond converting the raw 'filetime' value which you get to
'seconds
since 1970' (UTC) which is just (essentially) done by adding a constant
offset
it doesn't do anything suspect with such values.

I can't explain why Explorer shows the correct values unless:

   1) It 'knows' about some sort of DST drift problem and has hard-coded
handling of it

   2) It is using a different API to LiveCode to get the same info

Of course (2) requires there being a different file system API one could
use, but to
my knowledge all Win32 FS operations are implemented in terms of the
family we are
using.

Therefore, some more precise details of the setup your app is running on
is probably
the best thing to try and get now - i.e. the details of the network
share and how
it is configured in the VM.

Warmest Regards,

Mark.

--
Mark Waddingham ~ [hidden email] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OMG WTF detailed files BST?

Monte Goulding via use-livecode
The plot is slightly thickening. I restarted the VM, set the date+time to not
adjust for daylight saving, restarted again. This removed some 30,000 images
from the list with allegedly modified timestamps; the ones that were no longer
consider to have changed were - in some year or other - modified between late
march and the end of September.

I've now switched the 'adjust' setting back on, restarted the VM again, and
await tomorrow's report with interest.

(In the meantime I've asked the clients IT dept what the server in question is
running, but on the whole they tend to be very unresponsive... sigh.)

Ben




On 03/04/2017 09:06, Mark Waddingham via use-livecode wrote:

> On 2017-03-31 19:23, Local (BenR) wrote:
>> Hi Mark,
>>
>> Thanks for your response, very helpful.
>>
>> Unfortunately I don't know what the filesystem of the volume with the
>> files is - this is a client's network, and they set up a VM in the DMZ
>> for me to work on, with this network share mounted - but I've no idea
>> what the fileserver is.
>
> Hmmm - I suspect something to do with the fileserver is at play here.
>
> LiveCode uses FindFirstFileW and similar Win32 APIs to fetch the detailed file
> into. Beyond converting the raw 'filetime' value which you get to 'seconds
> since 1970' (UTC) which is just (essentially) done by adding a constant offset
> it doesn't do anything suspect with such values.
>
> I can't explain why Explorer shows the correct values unless:
>
>   1) It 'knows' about some sort of DST drift problem and has hard-coded
> handling of it
>
>   2) It is using a different API to LiveCode to get the same info
>
> Of course (2) requires there being a different file system API one could use,
> but to
> my knowledge all Win32 FS operations are implemented in terms of the family we
> are
> using.
>
> Therefore, some more precise details of the setup your app is running on is
> probably
> the best thing to try and get now - i.e. the details of the network share and how
> it is configured in the VM.
>
> Warmest Regards,
>
> Mark.
>

_______________________________________________
use-livecode mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Loading...