This is the same as the alarm clock problem. When a user creates an event, what does that user _mean_? In the alarm clock example, what should happen if I've set my alarm clock 7am and then travel to a different time zone? In the conference event example, what should happen if the local time zone rules change, or daylight savings time comes into force? Did the event organiser print marketing materials that said "starts at 1000 UTC" or "starts at 10am" with a "in the local time zone" implication?
Events generally need to be locked to _something_ - whether that is UTC or a local time zone depends on the application.
In situations with ambiguity like this, it's probably best to ask the user instead of make assumptions.
I would not be remotely perturbed if my phone had a pop up when I stepped off the plane asking me if I still wanted to wake up at 7AM local or would prefer to wake up at 10AM UTC.
I would be greatly perturbed if it assumed the latter and I missed my meeting, all because I forgot to set a timezone when I originally inserted it a month prior.
If, at 12:30 AM on a Tuesday, you ask Siri to remind you about something “tomorrow”, Siri will ask if you meant Wednesday or Tuesday. I always thought that was a nice touch.
In Gmail (and Inbox, RIP), the 12:30am "Snooze" action assumes "tomorrow" means 8am the same day. I'm not sure what time the switch-over becomes "8am the next day". I think around 4 or 5am. Once I learned that's what happens, it became just as useful as "later today".
I use both quite a bit. I'd definitely rather not be asked every time - especially since I work at night and do most of my email management in that timeframe
What do you suggest in situations where you can’t do that? Last summer, I travelled on a sleeper train from Lisbon to Madrid, which are in different time zones. At the time that I set my alarm, my phone had no way of knowing that it would automatically jump an hour ahead in the middle of the night.
In that specific case, it woke me up according to local time, which is what I expected. But what if I had crossed the time zone boundary less than an hour before I wanted to be woken up?
If I'm going from my personal experience when travelling across timezones, I generally set my phone to the local time, check it about 5 times, call the front desk and request a wake-up call, and set the bedside alarm. All three have failed me at one time or another and I'm a little paranoid about sleeping through meetings!
For the question of crossing a time zone boundary while using local time, I'd say the answer is obviously to go off the moment local time passes the alarm time.
Going the other way is a more fun problem, but a few hours of debouncing is fine for anyone moving at suborbital speeds.
In my experience assuming I think of my calendar as anything other than local time is always wrong. 10+ years of using calendars in different timezones and I have never once meant anything other than “9am when I’m there”
It’s my calendar, why would I set it to events in a way that’s not relative to my time reference?
I agree with this as you've stated it, but there's a difference with a personal event, like an alarm, and a calendar event.
For instance, say there's an 1pm CT meeting every Tuesday for a remote team. When I head to the East Coast for a week, that _should_ mean 2pm ET, but if the assumption is "1pm when I'm there", since the "there" has changed if my calendar doesn't adjust accordingly I'm going to be an hour early to the meeting.
This has happened to me numerous times. I set up an appointment in Outlook assuming I'll be there remotely via webconference and then end up having to travel there in person. I've been bitten enough times to double-check the timezone now, but that was a hard-earned lesson that most calendars tend to use the timezone the appointment was created in unless explicitly told otherwise.
I'm a swede watching a lot of NFL and college games. Often the only thing I see are American times and need to input them easily in my calendar. Then I'm more interested in inputing "1pm EST" into my calendar and let the calendar figure out when to alert me of the game.
This is something we (startup insurance company) have thought about a lot actually. Since the certificates we issue are relevant to a particular jurisdiction, if the document states e.g. "23:59:59", it means the "wall clock" time in that jurisdiction - whatever point-in-time our DB contains isn't really that relevant.
So the likely problematic situ is if we've got a policy end date/time more than a year in advance, then the country changes their TZ offsets, we need to make sure our point-in-time records get updated (and then of course the duration of the policy changes). It's a bit of a pain in a system built around immutable events!
On a kinda related note, we also took the decision to clearly define that our start/end date-times have a resolution of one second and are inclusive. So if a policy starts at 00:00:00 and ends at 23:59:59, that's the full day, all the way up to midnight. It also means then it's important that we render the full time (incl seconds) on our docs.
Exactly. It's usually best to specify end times exclusively, and not just for leap seconds; it's easier to program in general. If your clock has sub-second granularity, then you can end up with a time value like 23:59:59.123. Which will require some truncation before it can be compared like t ≤ 23:59:59. If you use exclusive end times, you can do the comparison directly; 2016-12-31 23:59:59.123 < 2017-01-01 00:00:00. The key is that now we have an end date-time that actually specifies a true instant in time, and isn't an interval in and of itself.
For that very reason, for start/end policy times, we store them aligned to the second - no fractional seconds allowed. There'd be no legal basis for that anyway.
Obviously for actual timestamps - e.g. created at/updated at times - we do store a point-in-time with fractional seconds.
From an operational perspective, if there was an incident in the second between 23:59:60 and midnight, it would of course be covered - irrespective of what's in any system, written on any doc, displayed in the app, etc.
We don't worry too much about leap seconds - particularly as very few people are even likely to realise it was a factor, and the national Motor Insurance Database (MID) does not support them.
The MID doesn't even allow us to specify timezones, so every year we sell around 20-30 one-hour car insurance policies which appear to run for 1 second - displaying the same start and end time on police computers etc. (Due to the clocks going back from BST to GMT.)
From a legal perspective, the time written on the policy doc is inclusive - we can't just decide that we want it to be exclusive because it suits our purposes.
> From a legal perspective, the time written on the policy doc is inclusive - we can't just decide that we want it to be exclusive because it suits our purposes.
Well, apparently it's not if it's understood that it covers leap seconds as well. With regard to the internal modeling, I always go for half-open intervals, where the open side depends on the data being modeled, they have much nicer mathematical properties than closed intervals.
What if the loss was caused by and at the leap second? e.g. imagine a self-driving car that ignored input during the leap second and crashed into a lane divider? You may have proof that the accident occurred at exactly 23:59:60.
What if the car is flying through the air aside a suspension bridge at 23:59:59 and :60, and crashes into the water a second or two later? Has the loss occurred the moment the impact with terrain becomes inevitable, or not until impact?
I bet you already know this but for the reader: the solution is to not think of times occurring with an offset but to rely on the IANA timezone database to figure out what a timezone means at a particular UTC time: https://www.iana.org/time-zones
That doesn't solve the problem. Suppose Massachusetts were to decide to move from the Eastern to the Atlantic time zone and New York were to decide to remain in the Eastern time zone. The time zone you would need for people in Massachusetts would be America/Boston, not America/New_York--which doesn't exist now.
Representing future times correctly is very hard, especially because time zone changes (including creation of new ones!) aren't always announced years in advance.
None of these tricks will help if I’m insuring myself for international travel. Only the trip to and from the airport is in the time zone of the policy.
Any incidents along the way or after that are going to need an interpretation relative to UTC.
A question to both parent of this and this specifically.
How would you deal with the current EU situation? They have recently voted to stop yearly transitions to summer time. But there must still be a timezone assigned to an event.
Are you assigning timezones (in the certificates themselves) as the timezones of a particular geographic location,, rather than as time offsets?
You should definitely use geographic timezone like America/Los_Angeles instead of just offset. Offset actually isn't a real timezone it is just an offset. Timezone usually means a lookup table of what offset it has at what time and that can also change due to different laws at different time. So I assume the moment EU countries will change the rules there will be one more row for each timezone that will say "from this time the offset is such and such". That means it is also important to use the right timezone in the sense that even though Bratislava/SK and Prague/CZ have the same rules (DST time and offsets) at the moment they can change in different ways after the DST is removed (as whether to use winter or summer time afterwards is in jurisdiction of each country I think).
I agree. In the only project I worked on where time was really important I stored a triple (original datetime, UTC timestamp, original time zone). I used UTC to globally sort events and sent the original datetime back to clients for display.
On the other side, sometimes we don't know where an event happens. Think of GPS trackers close to a border which is also a timezone border. This is a more complex problem which requires at least a model of country borders and maybe roads.
Finally, on some systems I'm using as a user, they have some EU timezones but not even all the major countries. It doesn't matter now but it will. I expect they'll add the missing timezones and make us confirm where we are.
On the certificates, we'd state e.g. "this policy runs from 27th March 2019 at 16:52:12 (GMT) to 3rd April 2019 at 17:52:12 (BST)" for a 168-hour (~7 day) policy.
Logically, we think of it as the "wall clock" time, plus the location ("Europe/London" - we don't operate outside the UK yet), plus the local zone (BST/GMT) to handle ambiguity when clocks go back.
Practically, right now the start/end date-times are stored using a Postgres timestamp with no timezone. As we've never issued an individual policy of more than 28 days' duration or more than 7 days in advance, we've never actually had to deal with any situation where the point-in-time is no longer correct. And this case is not likely to happen without a decent amount of notice.
We're very much in favour of abolishing seasonal transitions though - causes lots of confusion.
> And this case is not likely to happen without a decent amount of notice.
In the jurisdictions you care about. Last-minute timezone changes do happen, and the main time zone database has been updated retroactively in the past. It's actually possible that a timezone change has even been enacted retroactively (thanks to countries that like doing very last-minute DST starts).
Yeah indeed - for now we're fortunate that it's not something we need to worry about actually handling perfectly. It's more something we like to think about at this point.
One day we will need to deal with it and conversations like these are super helpful to support that.
If a time is designated as "local" it is most flexible to store the location in a way that will handle the dynamism. e.g. I would not record a time as "16:00 BST" but rather "16:00 Europe/London" and use the olson timezone db to resolve it to BST or GMT downstream.
That’s what annoys me about google calendar and most others. When I schedule an appointment most of the times it will be for the time zone i will be in at the time of the appointment. So when I travel and agree to meet people at 10 I want to enter it that way. But in most calendar apps everything gets shifted around as soon as I arrive at the new location. I would much prefer to have apps not shift stuff around unless I explicitly tell the app to do it.
I think what he wants is that the software defers asking for a time zone until the item relates to two users who have different timezone.
And that if the event arrives without having touched users in different timezones, then the software should infer that the event's timezone is the local timezone of all users at that point in time and space.
What I would like is to to be able to set my "local time schedule" and be able to do it in advance. So for example if I am traveling next week I can indicate that my local time will change.
This is great not only for me but also for other people trying to schedule events in my calendar.
With Gmail picking up flight and hotel confirmations this could even be mostly automatic.
Its not that you want no time zone, you want to mark the appointment as "local" time. That could be useful, though I expect it would create as many problems as it solves.
The explicit timezone is crucial for conference calls that cross timezones. That is at least half of the appointments on my calendar (I coordinate projects between the US, Europe and Asia).
Right. When I travel I usually have a combination of physical meetings in local time and phone calls back home or spanning multiple time zones. I do mess up sometimes but I’m not sure how much you can automate.
The iCalendar standard does actually support that, but perhaps Google Calendar's UI does not. I suspect it does work internally, because it interoperates with iCalendar fairly well.
The standard calls it a "floating" DATE-TIME:
"The recipient of an iCalendar object with a property value consisting of a local time, without any relative time zone information, SHOULD interpret the value as being fixed to whatever time zone the "ATTENDEE" is in at any given moment."
There are two use cases: one is for meetings with participants in different timezones. Gcal does this well. There other use case is travel. For this the current system is not very good.
Well, you know the timezone: it's of wherever in Germany the meeting will take place.
I think the problem is the display: since you know you'll be in Germany from date X to Y, you should be able to configure the presentation timezone for those days (which would also show you those meetings in the local time when you'll have to make them).
Most people have more events that happen at a particular instant but different local times (e.g. phone calls between people in different timezones) than events that happen at a fixed local time but different instants. When I'm planning my trip to Germany I certainly don't want it showing all my calls at the wrong time.
I think what you want, at least on the mobile Google Calendar app is under:
Settings - General - Use device time zone
If you disable this, you'll keep everything pinned to your preferred timezone.
But I'm pretty sure a better approach is to select the timezone of the event. If it is at 10am in Germany, why not tell your calendar about this? Sure, this is an extra step, but it is part of the event's parameters (as pointed out in the OP).
For that you would like to have an explicit "local time zone" option. I had recently similar in application I am developing, with datetime of which we are not sure in which time zone they are. That information might arrive later.
On my Palm Pilot there was a “floating” time zone that behaved like this. But I can’t find this feature in any modern app. Outlook doesn’t have it either.
And yet iOS calendar is also more powerful in that you can set different time zones on the start and end of a single event, which macOS calendar will show but not let you edit.
In UI cases like this, I usually wish tech would err on the side of being transparent and predictable rather than trying to be smart. For example, if my alarm has been set to 10:00 (with no attached time zone), it should go off when the clock face reaches 10:00.
I think python3's datetime has something like what you're pointing at -- objects are either timezone-aware or not. My complaint probably has to do with trying to hide this distinction from the user, when it would be more helpful to make it clear.
iCalendar (the Mac app, not the underlying representation) gets this badly wrong. It assumes that all events on the calendar are bound to the time zone that the computer was in when the event was entered. This is decidedly not the Right Thing when I travel. If I schedule an event for 7 AM, what I invariably mean is 7 AM in the local time zone where I am going to be on the day of the event, not on the day that I enter the event.
The upshot is that if I travel and change my Mac's clock to local time, all of my events get screwed up. The first time this happened I nearly had a heart attack because I have years of history in my calendar. So I always leave my Mac's clock set to my home time zone and do the conversion in my head.
With Google calendar that's very easy, there's an option to specify different time zones for a single event's start and end times.
It's also very useful that you can set the left margin (in the web app) to show two different time zones in a "ruler" format. So even if your overseas appointments for later in the week are displayed in terms of your current timezone, a quick glance across will confirm what their "local" time will be once you're on the ground.
Sort of a summarization of most of the ideas here is:
* for clear instants like a system log UTC makes sense
* for times in the past (far enough in the past that you know your current TZ rules are correct) UTC makes sense.
* for the immediate past (near enough where your rules might not be correct) you might want to store zone as well.
For all cases of times in the past, remember that you're trying to record an instant. An exact unambiguous point in time that is the same irrespective of who is looking at it or where they are. For this UTC makes sense - people who see that instant converted to their local timezone are noting what their clocks would have shown at that instant, but everyone who will ever see it is always referencing the same instant.
FUTURE times are very different. There are times at which you want to reference a future instant, at which point you can just use UTC, but most often what you really have is "this is what I expect to see on my wall clock when this thing happens". That is a very different beast, because nobody knows what the timezone rules or geopolitical situation will be when that thing does happen.
You need to recognize the use cases where what you mean is " I expect to see XX:YY:ZZ on my wall clock when this happens" and just record that information as is, including as much information as you can about that wall clock - possibly which timezone it (currently) uses, where it is (in case the timezone itself changes) and then convert to an instant at the last possible moment, or keep updating your instant based on changing rules.
Ran into the "fun" around future dates big time at my previous company.
We allowed users to define schedules which would control light, temperature, co2, irrigation, etc for greenhouses. Those schedules would load into on premise controllers and run for weeks or months.
They might be built by a user in one timezone and deployed to multiple facilities in different timezones. They often ran across DST boundaries, and some users wanted them to follow "wall clock time" while others wanted to ignore DST to match the behavior of legacy control systems.
There is one thing that I ran into with past times that I think falls outside of this. When you record things like “first day of employment”, you really do mean the date and not the datetime, and the date should be interpreted in whatever the local timezone is. Theoretically your first datetime if employment is what, midnight of the day you show up? Or is it 8am or whatever when you actually first step through the door? Each situation will be interpreted differently by the parties involved so best you can do is to store the date and the location/timezone of where they work, even though it’s in the past.
Yeah, a date and datetime / instant are two very different things, but I just renewed an insurance policy, and while it's been all dates on the forms so far the actual policy documents is clear about midnight to midnight on those dates.
In the particular case of employment I'd assume there was as implicit time as well, because employment would be a legal contract (whatever the precedent is in that jurisdiction would probably apply). You couldn't do something that's against your contract on the way to work on your first day, for example, and claim that it didn't count because it was before office hours.
The answer to your examples is that the conversion from date to datetime sometimes is ill-defined and you shouldn't assume that this can be properly done.
Legally, employment usually has a well defined start date and end date - and it often simply doesn't have a well defined start datetime and an end datetime, you might as well assume that it's undefined unless/until a particular situation forces you to get a court ruling for that.
How would you for example run payroll then for people who work 4pm-2am? You have to know whether the last two hours of the shift fall into one payroll period or the next, which necessitates that you be able to convert to a datetime.
Time is so messy, and that messiness becomes apparent when the yes/no, true/false world of computers tries to interpret things.
The blog posts brings up a lot of good points about time zones changing, and the comments make even better points about people moving across time zones. Something that confounds me occasionally is the concept of "local human time" because there are all sorts of ways that humans tamper with time.
Lawmakers, for example, that are required to pass certain laws by a particular date will sometimes unplug clocks in the legislative chambers to stop the legal passage of time while negotiations continue. You end up in a situation where, on paper the next day or week, 134 laws were negotiated, voted on, ratified, re-voted, passed, and signed all at exactly $date:23:59:59. A physical impossibility, but a legal fact.
Or, as I explore the more interesting corners of America more, there are entire towns and regions that ignore their official time zone and every house, business, church, pharmacy, etc... is on a more convenient time for the locals. The only place you see the official time is at the post office.
People are messy. Time is messy. People * time = a complete mess.
> Lawmakers, for example, that are required to pass certain laws by a particular date will sometimes unplug clocks in the legislative chambers to stop the legal passage of time while negotiations continue. You end up in a situation where, on paper the next day or week, 134 laws were negotiated, voted on, ratified, re-voted, passed, and signed all at exactly $date:23:59:59. A physical impossibility, but a legal fact.
Is this internationally accepted or a USA-only thing? It seems like fraud to me.
I don't know about that particular description, but it kind of makes sense - two bills passed on 27th March parliament session are passed on the same "business day" even if that day gets long and one of them was voted on five minutes before midnight and the other was voted on five minutes past midnight; and if there are some mandated legal delays (e.g. you have x days to before they'll be in force or something) then both bills will be treated as if they were passed on the same date.
This also matches common usage - if I tell you what I did on a friday night, it'll include both things before and after midnight.
There are also parallels in banking where business days don't align with calendar days, and you may have (depending on regulations and agreements) either a "long friday" or "long monday" where an action that's physically performed on a sunday will be treated with an effective date of friday or monday respectively; or possibly events after the closure of a business day may be legally treated as happened in the day after.
Offtopic: One time the Cardinals of the Catholic Church couldn't come to an agreement over the next Pope. After a few months of that the Holy Roman Emperor or some such had workmen remove the roof and locked them inside. A day and a half later, new Pope!
> entire towns and regions that ignore their official time zone and every house, business, church, pharmacy, etc... is on a more convenient time for the locals.
Fascinating, and I've never heard of such a thing. Are these towns near a time zone line, that find it more convenient to be in the time zone of a nearby large city on the other side?
I did a full switch to Postgres like 2 years ago, and a decent chunk of the decision was `timestamp with timezone` saved me a huge amount of headaches working on multinational services
Use timestamp with tz, drop Unix timestamps and use ISO8601 DateTime format across the application ezpz never look back
Unfortunately `timestamp with time zone` is misleading and crippled.
When you store something as timestamptz, postgres converts it to UTC and "loses" the timezone part.
The only difference between timestamptz and timestamp is that when postgres is returning the timestamptz to you, it converts it back to your local time zone defined by SET TIME ZONE (and not the time zone it was originally stored in)
So basically if you store a timestamptz, you cannot convert it back to the local time, as the local time is never even stored.
I personally had to add a second `timezone` column to my table to store the timezone of the date.
I wouldn't go so far as to say the timestamp types are broken, just confusingly named. `timestamp with time zone` signifies an absolute point in time; `timestamp without time zone` signifies a timestamp relative to some unspecified time zone (which you must store separately, as you point out). The builtin functions universally adhere to these semantics.
E.g., when you do store the timezone separately, use `timestamp without time zone`: `time AT TIME ZONE timezone` [1] only does the right thing (i.e., interpreting `time` as relative to `timezone`, and producing an absolute time corresponding to it) if `time` is `without time zone`. (If `time` is `with time zone`, it produces the local time corresponding to the absolute time `time`, which is mathematically exactly the opposite of what you want.)
I think PG will exhibit the problems that skeet was referring to because PG stores the UTC time and not the local time. If someone makes a change to the timezone rules then the times your columns are referring to will stay the same in UTC time which might not be what you want.
"For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time, traditionally known as Greenwich Mean Time, GMT). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's timezone parameter, and is converted to UTC using the offset for the timezone zone."
'timestamp with timezone' is kind of a gotcha. i guess it combines two things into one type that are logically one type but it also might mislead someone into believing the type is something else.
timstamptz is fine for historical data and certain kinds of “this will (need to )happen at this time” situations, but as others have said it just converts to/from UTC based on your session time zone - this makes it inadequate for calendaring.
For calendaring or other applications where time zone’s shifting offsets can cause issues you need to store the UTC time along with the timezone and then do the conversion on read.
Additionally, you have to be careful with the Postgres JDBC driver. When you store or load to a timestamp column, it compares the JVM local time zone with Postgres' session time zone and translates appropriately. Even when you have a test that makes sure that you get back the same timestamp that you put into it, it might be the wrong timestamp if you use psql to look at it.
Storing UTC (or rather TAI) is a silver bullet for "system" times - things that need to happen on a regular schedule, where we care how much time has passed between one instant and another, or things where we want to record at what instant something happened, and correlate system events with each other.
Indeed it doesn't solve the problem of handling user-inputted "human" times correctly. But that is a much rarer case than needing to handle "system" times. Many - probably most - applications never need to handle a "human" time at all.
I prefer TIA over UTC and wish it was widely used.
Leap seconds mean the same UTC time can happen twice if we fall back a second, or a UTC time may never happen when we skip ahead a second. Leap seconds also mean that epoch milliseconds derived from UTC do not represent milliseconds between the timestamp and Jan 1 1970 unless the conversion is aware of all leap seconds. Leap seconds are prescribed every six months with six month notice. There are many time keeping devices that are not updated as often as UTC leap seconds are introduced.
I'm pretty happy with ISO-8601, but dates after the year 10,000 will not sort lexicographical. A 64 bit signed int storing milliseconds since Jan 1 1970 covers 29225733 B.C to 29229672 A.D
Honestly, I think that leap seconds were simply a mistake. We could have phase-locked UTC to UT1R via frequency changes instead of via phase changes and much of that hassle would have been avoided.
UT1R is a time of day measurement that is based on making astronomical observations to establish the Earth's angular velocity and phase very precisely with respect to TAI. Leap seconds are created from time to time to keep the offset between UTC and UT1R to less than a half-second.
In effect, UTC is slaved to UT1R via a PLL that is only capable of making phase jumps. Instead, we could have slaved UTC to UT1R via a PLL that only ever made frequency changes. There would be no phase jumps (leap seconds), just very subtle adjustments in clock frequency. Those adjustments are fine enough that the majority of users don't care. The remaining users were already using TAI anyway.
Leap seconds are nothing to worry about when you're dealing with calendar times and dates in business. They are just a real-time clock adjustment which disappears from the calendar, which uses calendar seconds.
We can't phase lock real seconds to calendar seconds, because calendar seconds are a moving target. As the Earth's rotation slows, calendar seconds get longer. Seconds that get longer are not acceptable in science.
> We can't phase lock real seconds to calendar seconds, because calendar seconds are a moving target. As the Earth's rotation slows, calendar seconds get longer. Seconds that get longer are not acceptable in science.
Sure you can, and yes they are. There can be a difference between calendar seconds and atomic seconds. For most applications the difference is irrelevant. People doing science can explicitly account for the differences. And since most time-keeping in science is much more inaccurate than the difference between the speeds of TAI and UT1R, those folks won't care.
Most of the applications that do require very precise timekeeping actually only care about having a very consistent timebase, they don't necessarily care as much about the speed of time. So even if we steered GPS time by enough to use it as a broadcast source of UT1R, the majority of users who use GNSS to synchronize their measurements also don't care.
The remaining handful that do care are going to be purchasing atomic clocks anyway, and there are dedicated mechanisms available to synchronize those to TAI.
>There are many time keeping devices that are not updated as often as UTC leap seconds are introduced.
Isn't this an argument against using TAI in certain applications though? i.e. if you're designing a system which relies on non-leapsecond-updated devices to be in sync with devices that are capable of being updated, you should always design as such to accommodate the former rather than the latter?
No? It's an argument for using TAI,because devices that use TAI do not need leap second updates - a clock running TAI that was started in 1970 and never updated will show the same time as a clock running TAI that's started today (or one that was started in 1970 and updated continually), whereas a clock running UTC that wasn't updated will show a different time from one that was updated.
The united states will experience this soon. California and Washington are independently seeking to end daylight saving time. There is no America/Portland or America/Seattle so Washingtonians and Oregonians are using America/Los_Angeles. Unless Washington, California, and Oregon end Daylight savings on the same schedule we'll have to add new id's and users will have to change their configuration, and we'll still have issues described in the link.
The distinction between different kinds of time is really pretty key to having a clear understanding.
Civil time is different from atomic time and different from sidereal time or solar time.
Civil time is how people, businesses, and governments organize and plan their activities.
Atomic time is monotonic, physical, and "more objective" / less dependent on legislatures :)
Sidereal time is about the orientation of Earth relative to stars. It's really important if you're an astronomer.
UTC is a civil time standard based closely on TAI, an atomic time standard, except it gets corrected to be within 1 second of UT1, a sidereal time standard. It is basically _the_ civil time standard and it is a little compromise to make a civil time that advances like atomic time but also does not desync from sidereal time.
Different administrative regions use offsets from UTC to determine their local civil time.
If you're writing an application where people plan future activities and you really want to "get things right / not surprise users", you probably need to store times in the appropriate civil time standard, make it apparent _which_ civil time standard is canonical per activity if people are being coordinated across administrative regions that do not synchronize their standards, and ensure that your application can find out about planned changes to civil time in any of the administrative regions of interest.
Funny you should say that, here's an excerpt from the article:
> And then someone will calmly quote this passage in response, quietly pleased with themselves that the initial commenter was rude and certainly didn’t read the post at all. Then a third person will chime in on the thread saying the author was playing you all like a fiddle anyway, and the real problem is that the post was way too long to start with.
It's funny that it predicts my comment as well, touché.
I noticed another use case regarding photos on the filesystem. I like to see them in their local datetimes, rather than the absolute time in my current location, but don't believe linux filesystems supports that.
For example if I travel and take photos I don't want their mod times to be of those back home, which can be in the middle of the night or wrong day.
Have been moving them to my local timezone via a time shift, but not sure that is the right thing to do. Now they are in my timezone, but if I move one day, will I have to update all my photo times? Even with exiftool and DateTimeOriginal, that's more file updates than I'd like to do.
The filesystem is doing the right thing here by accurately storing the original point in time when the photo was taken (or at least it was before you shifted it). If two files were modified simultaneously then they should have the same mod times, even if the changes occurred on opposite sides of the planet. The problem is all in the presentation layer; you want your file explorer (or 'ls') to display the times in their original timezone, not the timezone you're currently in. There is an EXIF tag for the time zone offset but support for it appears to be limited. The original timezone could be inferred from the EXIF location data, however, if you have that enabled.
Things get really fun when you are asked to be able to sort these ambiguous dates.
Let's say Event A happens in New York (Eastern Time Zone) at 2018-12-08 02:00 UTC, and is recorded as occurring on 2018-12-07 (with no time) local time. Then let's say Event B happens in London (GMT) at 2018-12-08 01:00 UTC, and is recorded as occurring on 2018-12-08 (with no time) local time. Which event happened first? If we know the UTC times, a reasonable person would say Event B happened first. But if we just have the local dates with no times, we either say that Event A happened first, or we say we don't know.
This is a real problem I have to deal with and it is a pain. One user says A happened first, another says B happened first, another says A and B happened at the same time, and I say "give me the UTC time or you don't get this feature" :)
Exactly, timezone-aware dates do not have a total ordering but they do have a partial ordering. Most
of the time you can say if they are greater than or
equal to but sometimes you can't.
Not in Russia that wasn't the calendar in September of 1752! Apparently the `cal` command assumes certain geographic areas when it shows you calendars...
I believe the problems described here are with future dates, and future timezone definition changes you don't know about 'right now' when you converting a local time to UTC for storage.
The nature of the problem is that when you convert from a local timestamp to UTC, you might have been "wrong" in the conversion. Or, more specifically, it may have been as correct today, but it may become wrong some time in between now and when the timestamp occurs, because of future time zone definition changes. A future timezone definition change may mean you should have converted to UTC differently, which you couldn't have known because the timezone definition change happened after you converted. (Usually statutorily; sometimes maybe because your timezone data was wrong and was later corrected?)
I don't believe these problems apply to _past_ dates or "right now" timestamps. Unless your timezone data is wrong I guess.
As the article explains, though, present and past timestamps can be impacted too - your timezone library may be out of date at the moment you save the timestamp, or in some cases the legal body defining the timezone may reach into the past with a timezone change.
How can a legal body possibly change the timezone retroactively?
All those people on Monday that _thought_ it was 1pm, it turns on Tuesday we decided it was actually 2pm when they thought it was 1pm. But nobody could have known that on Monday. So they've got to... go back and change any times they put on documents on Monday? How does that make any sense?
But incorrect timezone data or implementation is too real, true.
I should also clarify that conceptually, even if you keep your copy of the tzinfo database obsessively up-to-date, you could still experience what appear to be retroactive updates, per the "Theory and Pragmatics" document from the tzinfo project [1].
That's because some governments are pretty sloppy about announcing their timezone changes in advance. It's compounded by the tzinfo DB being maintained by volunteers (who can't spend their lives obsessively gathering timezone change announcements the instant they're made), and the lead time necessary for software projects to update their tzinfo instances [2].
So, while retroactive changes may not be consciously made often or ever, it's certainly an effect that happens in practice.
Governments can do what they like, by and large. Just because it's crazy doesn't mean they won't, and nothing about physical reality prevents them from doing this.
I'm not aware of any specific examples, so perhaps I'm being too paranoid in my first post, but it would not surprise me in the least to hear of, for example, a bill to eliminate DST in a state that somehow manages not to pass until a day or three after the DST change takes effect.
There have at least been some some bizarre calendar changes that many software systems fail to accommodate, which probably feeds into my above paranoia:
- store future dates in the format that makes sense, based on what the truth is. Scheduling event at 9AM local time? Use local time. Determining when the spacecrafts meet? Use UTC (or Federation Standard Time if you prefer ;) ).
When a user inputs a time, let's say for an alarm, this is always "wall clock", and this is what you store. The timezone can be stored too, but this is also dynamic. For example, a user put a conference in a calendar at 7 am, this is dependent of the user location the day of the conference.
For app following the user, this is simple (like a mobile app), you just use the current timezone of the system the app is running on, but for a distributed system (client server), it might be complicated to pick a timezone and is dependent of the application.
For computer times (like logging, cron...), just use UTC.
Birthdays are a bit tricky too, say you want to show if it’s someone’s birthday today. In UTC you can be off by a day, so you just have to store the date without the TZ and treat it local in every TZ.
I don't see how the countdown timer can fail to reach zero when the conference starts, if we have calculated the correct number of seconds between now and the conference, and do not change the conference time based on whether or not daylight savings time is canceled, to preserve it apparent local time.
If we decide that the conference's start may be a moving target based on changing DST rules, then our coutdown timer cannot smoothly count down. At every tick, it has to evaluate the amount of time left to the conference start, and show that. The conference start's local time is always converted to UTC based on the current rules; when the rules change, it jumps by one hour and so the countdown timer adjusts itself at the next tick to show an extra hour.
You can't smoothly count-down toward something that starts at X or at X + 1h, and you don't know which until later.
Which is why storing the time data in UTC isn't a correct approach, because it just become countdown timer. A user may need to be reminded at specific time in a future, on a specific time zone. And it can't be reached by merely put a countdown timer.
That doesn't follow. If the DST rules change, then, effectively, any event whose local time stays the same has actually moved. The sun will be in a different spot of the sky when that meeting is held than it would have been.
All the dates stored in UTC which move due to a DST rule change have to be traversed and updated, as do any dependent countdown timers. That doesn't mean it wasn't "correct" to store them that way.
"Correct" means that we handle all the cases in our specifications and design, and they are translated to code without mistakes.
People like countdown timers. If there is going to be a countdown timer to an event whose time is uncertain, that countdown timer has to update itself when the exact time is pinned down. Until then, it should probably use the shorter countdown so that it errs on the side of too early. It has to be updated whether the time is stored as local time or as UTC. The behavior of the countdown timer cannot depend on how the time is stored in some database.
Store absolutely everything in UTC, do all calculations on UTC, and show in whatever timeunit the user wants. The one edge-case where user inputs a local time thats is not unique (repeated hour due to daylight savings time) can easily be solved consistently, and often is not even a problem at all.
This is bad advice if you need to store future events/appointments. Because in the majority of cases the meaning of the future datetime is "the date/times on the majority of the calendars and clocks on the wall in the vicinity of the event". The implication is that if the local timezone rules change between now and the future event, the user wants the local datetime of the appointment to stay unchanged, which implies that the UTC/epoch version of the appointment has to be changed. Generating the correct datetimes from UTC/epoch+local timezone requires a lot more administration and logic than storing just the local datetime. (You need at least both the current TZ database and the version when the appointment was made/last changed to have a hope to do this correctly.)
I used to think that, when I was young and naive, which was maybe a month or so ago.
Here's a use case: You and I are agreeing to meet at a given landmark. This could be any landmark in the United States, just to keep it simple. The user experience is: you propose that we meet at <landmark> at 4 PM tomorrow, and I agree. This happens on a website.
As far as you, me, and most of the system is concerned, the only sensible way to store that time is as a timezone-agnostic, UTC-agnostic local time: "2019-03-28T16:00". That's because, for the system to store that time as UTC, we have to know the UTC offset of every landmark in the United States at any point in time (which isn't an easy problem!) and consistently, correctly convert between UTC and that local time every time it gets displayed anyway. If we screwed up the UTC offset when we stored it the first time, we have to change the UTC time.
Now, if there's an additional constraint that you can only schedule meeting times that are actually in the future, you have to know what times constitute "the future" relative to the UTC offset of the landmark, so you don't get out of needing a comprehensive UTC offset map of the United States. But if you screw that up, you just end up putting an irrelevant or impossible meeting time in a dropdown and maybe dealing with some users attempting to time travel. Even that is a failure mode that users will somewhat understand, but most of the time it won't matter unless users are scheduling within the same time window as your UTC error.
If you stored it in UTC, you'd be screwing it up all the damn time until you fixed your UTC offsets, at which point previously-working data would be incorrect because someone would schedule a meeting 8 days out in one of those geographic regions for 6 PM and suddenly wonder how it got shifted to 5 PM for no reason.
Now, if we're scheduling a meeting online, and videoconferencing instead of traveling to the Navajo Reservation in Arizona--of course we store it in UTC. And if some of us are traveling to the Navajo Reservation[1] and some of us are dialing in from Newfoundland[2], we're all fucked anyway.
[1] The Navajo Indian Reservation spans multiple states, including Arizona. Arizona does not observe DST. The Navajo Indian Reservation does. However, inside the Arizona portion of the Navajo reservation, there is a Hopi Indian Reservation, which does not observe DST. Inside of the Hopi reservation, there is another part of the Navajo reservation that does observe DST.
[2] Newfoundland's UTC offset isn't an integer number of hours, there's an extra half hour in there. Enjoy!
Your examples indeed show common problems with several approaches when place is coupled to a time and then proceed to ignore the place (of meeting or user) when converting times.
I should add that where we store UTC that concerns a place, we also know/store that place in lat,long (or anything that resolves to this) and use that to show it in any timezone, to any user in any timezone, ..even in any era (yes places can change timezone).
I think a more correct title would have been "date storage antipatterns"
storing in UTC is fine as long as your clients are time zone aware and can convert to local and send timestamps with time zone so that the server can convert back to UTC.
the "options" provided are just examples of how not to do it.
We simply use the google timezone API to find the event’s local timezone, for physical events, then store UTC. For non-physical events, we store UTC based on the timezone of the user agent that made the request, taking into account daylight savings rules. On the extremely rare occasion that rules change between when the event was scheduled and when it takes place, we don’t have a system in place, so the time would have to be updated manually.
NOW... it gets even more interesting for recurring events. Does 11AM on Tue mean 11AM on Tue after daylight savings? How about events like Shabbat candle lighting times that depend on when the sun sets?
It took us a while to do all these. Does anyone know a good non-API solution to timezone calculations / shapefiles feed?
I don't understand the described result for Option 1 or how having stored in UTC creates a problem there. It would seem to me like when the government changes the rules for local time, they are effectively moving the (real, not clock) time for the conference. It would seem like there is enough information stored in the database that installing the new time zone rules should cause the timer to move to accommodate this, as long as addresses don't get redefined (a different issue with that storage model). I don't see why the timer would not instantaneously jump an hour to correct itself when it becomes aware of the new rules or why it wouldn't hit the zero at the right time.
The rule effective time represents when the start time was last changed, converted to UTC.
Next, create a time engine that can convert between events (time + location). The engine can convert events to different time zones in different epochs, to UTC, or the other way around, because it knows all the rules, and when the rules changed throughout history. A simple API will handle things:
time time_engine.parse_time(rule_effective_time_event, string_representation)
Where time_event is a tuple consisting of the rule effective time, and the timezone whose rules to apply.
When the venue changes, you update TimeZoneId. When the start time changes, you change StartTime and RuleEffectiveTime. The time engine would only be invoked to present the start time in current time + zone format for the user to modify in their own temporal space, then to parse the changed time back to UTC using the user location + new RuleEffectiveTime (which is now).
Timezone rules are fixed as events in time and space, so every rule effective time + time zone + utc_time will uniquely point to a specific time zone rule, which means that the rule itself doesn't need to be stored in the database.
Timestamps are also fixed, and should never change (unless the user changes the time for the meeting, of course).
You could build the time engine to preserve absolute time (14:00 becomes 15:00 after the rule change), or to preserve local time (14:00 remains 14:00 after the rule change, even though it's technically a different time). You could even change how the engine behaves after data has been created, since it's only affecting how things are interpreted locally to the user. The data remains the same no matter what.
The database's role is to store data, not to decide rules. Store the data, not the rules.
Storing UTC is still usually right because in many cases rather than dealing with calendar times, developers are dealing with instants in time that have already passed/are passing (creation/update timestamps, logs, etc.) I think this basically is only important when you might store a date in the future. If you have timezone rules incorrect for present or near past dates, well you're going to have a hell of a time dealing with that no matter what you do, really.
Handling calendar times and recurring times have their own complications.
In our application we can compare charts of timeseries data that may originate in many different places in the world.
If you want to compare e.g. two charts of temperature over a 24-hour period between two places, you want noon for both to be in the same place on the X-axis. So you want to use local time of both places.
But if you only store UTC, you don't know local time.
No need for tzinfo changes or travelling through time zones.
If you have ever done anything seriously with time zones you store everything UTC and must use a library to translate. The problem is it is geopolitical, so the time zone conversion result can vary based on the date itself as well as the physical location. For example DST was changed in 2007 in the US, there is no option but to programmatically translate.
Even if you store in UTC and use a library you have to know what the user mean when they create a recurring appointment.
I currently have several recurring alerts in my calendar app. One of them is an alarm everyday at 10 p.m. This must be in local time everywhere I go. Another is a conference call at 10H30 a.m on Wednesdays, Pacific Time (not my timezone). If you treat them the same way one of them will be wrong when I travel.
An interesting case was time stamps for photos on a global photo sharing site. UTC and conversion to local for display was the normal pattern. We decided to change this to display the time of the photo local to where the photo was taken. This where-taken and when-taken were paired. It made new years' and other holiday or time of day shots make sense.
The "time zone version" issue mentioned in the article isn't quite as hard as it looks, as time zone-based libraries will (if done right) include the whole history of time zone changes too. For example, java.time and the JS lib moment-timezone both use a full time zone history to convert to/from UTC, not just current offsets.
I think you've misunderstood what's changing here. It's not a matter of time zones not knowing that rules vary across time. It's a matter of knowing that "the rules for 2022" vary over time as well - in that the rules I might predict for 2022 may well not be the case now.
"Time zone history" (in every platform I've seen) does not mean "knowing what the rules for 2022 were in 2019". They mean "getting things right according to all the currently-known rules, across all of history". That's very different.
I like unix time. But its not perfect either. Soon we will have a new y2k problem where unix time can no longer fit into an integer. Anyone remember the y2k problem ? People predicted major catastrophes as the year was encoded with two digits. But no one is talking about unix time.
I wonder what kind of fallout will come from the year 2038 problem [1]. It appears the Deep Impact space probe was lost to a 32-bit clock rollover bug, which portends some level of disaster come 2038.
This will certainly be a problem for older apps, but anything I've done in the last 10 years I've been dev'ing, literally every library I've used for Unix time uses a 64 bit integer, which should last for a few million years.
Am I right in thinking that if you always store in UTC, when the clocks go back you will get overlapping data?
e.g. if you store your logs with a UTC date you'll see interleaving logs for the extra hour, or if you read sensor data or something like that you'll get double readings?
Well kind of. At least with how it works in the US. The first 2am is PST, the second 2am is PDT. so while they look the same, they're actually different times still. That said, rendering this for a user is still a pain depending on the UI.
On a related note, I do all of my personal time in UTC. All of my phones, clocks, watches, computers, calendars are set to UTC. It just makes everything easier for my personal projects.
I don't understand. Your first example is the one where people need to do offset maths. The second example requires no offset maths as it's 0400 for both parties and they both already know what they're doing at 0400 if we didn't have timezones.
Because for future events whose time is fixed in their local timezone, you can't be certain that the timezone's offset from UTC on that future date is going to be the same as what you currently think it will be.
For most use cases when a users enter a datetime in the future, they do not care about how many seconds from 1970-01-01 that point is, they care that it occurs at exactly 12:30 on that particular day.
Well, yes, but you really need to add some information to it. The last system I worked on that really needed dates for transactions and reporting stored: local time, local time zone, local offset from UTC at time of event, and UTC time stamp.
Its amazing how complicated your life can get when reporting across time zones and then mixing that with legal requirements for some reporting.
Also, time zone id and local time is insufficient unless you keep reference tables that can calculate what offset was in effect at the time of the event. Save the trouble and just record it along with the time stamp.
Events generally need to be locked to _something_ - whether that is UTC or a local time zone depends on the application.
The iCalendar standard (https://tools.ietf.org/html/rfc5545) generally gets this right.