ChatGPT解决这个技术问题 Extra ChatGPT

How do I parse an ISO 8601-formatted date?

I need to parse RFC 3339 strings like "2008-09-03T20:56:35.450686Z" into Python's datetime type.

I have found strptime in the Python standard library, but it is not very convenient.

What is the best way to do this?

To be clear: ISO 8601 is the main standard. RFC 3339 is a self-proclaimed “profile” of ISO 8601 that makes some unwise overrides of ISO 8601 rules.
Don't miss the python3.7+ solution below for inverting isoformat()
This question should not be closed as dupe to the linked post. Since this one is asking to parse an ISO 8601 time string (which wasn’t supported natively by python pre to 3.7) and the other is to format a datetime object into an epoch string using an obsolete method.

J
Josh Correia

isoparse function from python-dateutil

The python-dateutil package has dateutil.parser.isoparse to parse not only RFC 3339 datetime strings like the one in the question, but also other ISO 8601 date and time strings that don't comply with RFC 3339 (such as ones with no UTC offset, or ones that represent only a date).

>>> import dateutil.parser
>>> dateutil.parser.isoparse('2008-09-03T20:56:35.450686Z') # RFC 3339 format
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686, tzinfo=tzutc())
>>> dateutil.parser.isoparse('2008-09-03T20:56:35.450686') # ISO 8601 extended format
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686)
>>> dateutil.parser.isoparse('20080903T205635.450686') # ISO 8601 basic format
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686)
>>> dateutil.parser.isoparse('20080903') # ISO 8601 basic format, date only
datetime.datetime(2008, 9, 3, 0, 0)

The python-dateutil package also has dateutil.parser.parse. Compared with isoparse, it is presumably less strict, but both of them are quite forgiving and will attempt to interpret the string that you pass in. If you want to eliminate the possibility of any misreads, you need to use something stricter than either of these functions.

Comparison with Python 3.7+’s built-in datetime.datetime.fromisoformat

dateutil.parser.isoparse is a full ISO-8601 format parser, but fromisoformat is deliberately not. Please see the latter function's docs for this cautionary caveat. (See this answer).


For the lazy, it's installed via python-dateutil not dateutil, so: pip install python-dateutil.
Be warned that the dateutil.parser is intentionally hacky: it tries to guess the format and makes inevitable assumptions (customizable by hand only) in ambiguous cases. So ONLY use it if you need to parse input of unknown format and are okay to tolerate occasional misreads.
Agreed. An example is passing a "date" of 9999. This will return the same as datetime(9999, current month, current day). Not a valid date in my view.
@ivan_pozdeev what package would you recommend for non-guessing parsing?
@ivan_pozdeev there's an update to the module that reads iso8601 dates: dateutil.readthedocs.io/en/stable/…
D
Demetris

The datetime standard library has, since Python 3.7, a function for inverting datetime.isoformat().

classmethod datetime.fromisoformat(date_string): Return a datetime corresponding to a date_string in one of the formats emitted by date.isoformat() and datetime.isoformat(). Specifically, this function supports strings in the format(s): YYYY-MM-DD[*HH[:MM[:SS[.mmm[mmm]]]][+HH:MM[:SS[.ffffff]]]] where * can match any single character. Caution: This does not support parsing arbitrary ISO 8601 strings - it is only intended as the inverse operation of datetime.isoformat(). Examples: >>> from datetime import datetime >>> datetime.fromisoformat('2011-11-04') datetime.datetime(2011, 11, 4, 0, 0) …

Be sure to read the caution from the docs!


That's weird. Because a datetime may contain a tzinfo, and thus output a timezone, but datetime.fromisoformat() doesn't parse the tzinfo ? seems like a bug ..
Don't miss that note in the documentation, this doesn't accept all valid ISO 8601 strings, only ones generated by isoformat. It doesn't accept the example in the question "2008-09-03T20:56:35.450686Z" because of the trailing Z, but it does accept "2008-09-03T20:56:35.450686".
To properly support the Z the input script can be modified with date_string.replace("Z", "+00:00").
Note that for seconds it only handles either exactly 0, 3 or 6 decimal places. If the input data has 1, 2, 4, 5, 7 or more decimal places, parsing will fail!
@JDOaktown This example uses native Python’s datetime library, not dateutil’s parser. It actually will fail if the decimal places are not 0, 3, or 6 with this approach.
n
nofinator

Note in Python 2.6+ and Py3K, the %f character catches microseconds.

>>> datetime.datetime.strptime("2008-09-03T20:56:35.450686Z", "%Y-%m-%dT%H:%M:%S.%fZ")

See issue here


Note - if using Naive datetimes - I think you get no TZ at all - Z may not match anything.
This answer (in its current, edited form) relies upon hard-coding a particular UTC offset (namely "Z", which means +00:00) into the format string. This is a bad idea because it will fail to parse any datetime with a different UTC offset and raise an exception. See my answer that describes how parsing RFC 3339 with strptime is in fact impossible.
in my case %f caught microseconds rather than Z, datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f') so this did the trick
Does Py3K mean Python 3000?!?
@Robino IIRC, "Python 3000" is an old name for what is now known as Python 3.
C
Community

Several answers here suggest using datetime.datetime.strptime to parse RFC 3339 or ISO 8601 datetimes with timezones, like the one exhibited in the question:

2008-09-03T20:56:35.450686Z

This is a bad idea.

Assuming that you want to support the full RFC 3339 format, including support for UTC offsets other than zero, then the code these answers suggest does not work. Indeed, it cannot work, because parsing RFC 3339 syntax using strptime is impossible. The format strings used by Python's datetime module are incapable of describing RFC 3339 syntax.

The problem is UTC offsets. The RFC 3339 Internet Date/Time Format requires that every date-time includes a UTC offset, and that those offsets can either be Z (short for "Zulu time") or in +HH:MM or -HH:MM format, like +05:00 or -10:30.

Consequently, these are all valid RFC 3339 datetimes:

2008-09-03T20:56:35.450686Z

2008-09-03T20:56:35.450686+05:00

2008-09-03T20:56:35.450686-10:30

Alas, the format strings used by strptime and strftime have no directive that corresponds to UTC offsets in RFC 3339 format. A complete list of the directives they support can be found at https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior, and the only UTC offset directive included in the list is %z:

%z UTC offset in the form +HHMM or -HHMM (empty string if the the object is naive). Example: (empty), +0000, -0400, +1030

This doesn't match the format of an RFC 3339 offset, and indeed if we try to use %z in the format string and parse an RFC 3339 date, we'll fail:

>>> from datetime import datetime
>>> datetime.strptime("2008-09-03T20:56:35.450686Z", "%Y-%m-%dT%H:%M:%S.%f%z")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.4/_strptime.py", line 500, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/lib/python3.4/_strptime.py", line 337, in _strptime
    (data_string, format))
ValueError: time data '2008-09-03T20:56:35.450686Z' does not match format '%Y-%m-%dT%H:%M:%S.%f%z'
>>> datetime.strptime("2008-09-03T20:56:35.450686+05:00", "%Y-%m-%dT%H:%M:%S.%f%z")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.4/_strptime.py", line 500, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/lib/python3.4/_strptime.py", line 337, in _strptime
    (data_string, format))
ValueError: time data '2008-09-03T20:56:35.450686+05:00' does not match format '%Y-%m-%dT%H:%M:%S.%f%z'

(Actually, the above is just what you'll see in Python 3. In Python 2 we'll fail for an even simpler reason, which is that strptime does not implement the %z directive at all in Python 2.)

The multiple answers here that recommend strptime all work around this by including a literal Z in their format string, which matches the Z from the question asker's example datetime string (and discards it, producing a datetime object without a timezone):

>>> datetime.strptime("2008-09-03T20:56:35.450686Z", "%Y-%m-%dT%H:%M:%S.%fZ")
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686)

Since this discards timezone information that was included in the original datetime string, it's questionable whether we should regard even this result as correct. But more importantly, because this approach involves hard-coding a particular UTC offset into the format string, it will choke the moment it tries to parse any RFC 3339 datetime with a different UTC offset:

>>> datetime.strptime("2008-09-03T20:56:35.450686+05:00", "%Y-%m-%dT%H:%M:%S.%fZ")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.4/_strptime.py", line 500, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/lib/python3.4/_strptime.py", line 337, in _strptime
    (data_string, format))
ValueError: time data '2008-09-03T20:56:35.450686+05:00' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'

Unless you're certain that you only need to support RFC 3339 datetimes in Zulu time, and not ones with other timezone offsets, don't use strptime. Use one of the many other approaches described in answers here instead.


It's mind bogging why strptime doesn't have a directive for ISO format timezone info, and why it cannot be parsed. Incredible.
@CsabaToth Entirely agreed - if I have some time to kill, perhaps I'll try to add it into the language. Or you could do so, if you were so inclined - I see you have some C experience, unlike me.
@CsabaToth - Why incredible? It works good enough for most people, or they found easy enough workaround. If you need the feature, it is opensource and you can add it. Or pay someone to do it for you. Why someone should volunteer his own free time to solve your specific problems? Let source be with you.
@PeterMasiar Incredible because usually one discovers that things in python have been implemented thoughtfully and fully. We have been spoilt by this attention to detail and so when we stumble across something in the language that is "unpythonic" we throw our toys out the pram, as I am about to do so right now. Whaaaaaaaaaa Whaa wahaaaaa :-(
strptime() in Python 3.7 now supports everything described as impossible in this answer ('Z' literal and ':' in the timezone offset). Unfortunately, there is another corner case that makes RFC 3339 fundamentally incompatible with ISO 8601, namely, the former allows a negative null timezone offset -00:00 and the later not.
F
Flimm

Try the iso8601 module; it does exactly this.

There are several other options mentioned on the WorkingWithTime page on the python.org wiki.


Simple as iso8601.parse_date("2008-09-03T20:56:35.450686Z")
The question wasn't "how do I parse ISO 8601 dates", it was "how do I parse this exact date format."
@tiktak The OP asked "I need to parse strings like X" and my reply to that, having tried both libraries, is to use another one, because iso8601 has important issues still open. My involvement or lack thereof in such a project is completely unrelated to the answer.
iso8601, a.k.a. pyiso8601, has been updated as recently as Feb 2014. The latest version supports a much broader set of ISO 8601 strings. I've been using to good effect in some of my projects.
Sadly that lib called "iso8601" on pypi is trivially incomplete. It clearly states it doesn't handle dates based on week numbers just to pick one example.
F
FObersteiner

Starting from Python 3.7, strptime supports colon delimiters in UTC offsets (source). So you can then use:

import datetime
datetime.datetime.strptime('2018-01-31T09:24:31.488670+00:00', '%Y-%m-%dT%H:%M:%S.%f%z')

EDIT:

As pointed out by Martijn, if you created the datetime object using isoformat(), you can simply use datetime.fromisoformat().


But in 3.7, you also have datetime.fromisoformat() which handles strings like your input automatically: datetime.datetime.isoformat('2018-01-31T09:24:31.488670+00:00').
Good point. I agree, I recommend to use datetime.fromisoformat() and datetime.isoformat()
This is the only answer that actually meets the question criteria. If you have to use strptime this is the correct answer
You example fails on Python 3.6 with: ValueError: time data '2018-01-31T09:24:31.488670+00:00' does not match format '%Y-%m-%dT%H:%M:%S.%f%z' that's due to %z not matching +00:00. However +0000 matches %z see python doc docs.python.org/3.6/library/…
@Eric Yes, this answer only works in Python 3.7 or newer.
P
Peter Mortensen

What is the exact error you get? Is it like the following?

>>> datetime.datetime.strptime("2008-08-12T12:20:30.656234Z", "%Y-%m-%dT%H:%M:%S.Z")
ValueError: time data did not match format:  data=2008-08-12T12:20:30.656234Z  fmt=%Y-%m-%dT%H:%M:%S.Z

If yes, you can split your input string on ".", and then add the microseconds to the datetime you got.

Try this:

>>> def gt(dt_str):
        dt, _, us= dt_str.partition(".")
        dt= datetime.datetime.strptime(dt, "%Y-%m-%dT%H:%M:%S")
        us= int(us.rstrip("Z"), 10)
        return dt + datetime.timedelta(microseconds=us)

>>> gt("2008-08-12T12:20:30.656234Z")
datetime.datetime(2008, 8, 12, 12, 20, 30, 656234)

You can't just strip .Z because it means timezone and can be different. I need to convert date to the UTC timezone.
A plain datetime object has no concept of timezone. If all your times are ending in "Z", all the datetimes you get are UTC (Zulu time).
if the timezone is anything other than "" or "Z", then it must be an offset in hours/minutes, which can be directly added to/subtracted from the datetime object. you could create a tzinfo subclass to handle it, but that's probably not reccomended.
Additionally, "%f" is the microsecond specifier, so a (timezone-naive) strptime string looks like: "%Y-%m-%dT%H:%M:%S.%f" .
This will raise an exception if the given datetime string has a UTC offset other than "Z". It does not support the entire RFC 3339 format and is an inferior answer to others that handle UTC offsets properly.
F
FObersteiner

A simple option from one of the comments: replace 'Z' with '+00:00' - and use Python 3.7+'s fromisoformat:

from datetime import datetime

s = "2008-09-03T20:56:35.450686Z"

datetime.fromisoformat(s.replace('Z', '+00:00'))
# datetime.datetime(2008, 9, 3, 20, 56, 35, 450686, tzinfo=datetime.timezone.utc)

Although strptime can parse the 'Z' character to UTC, fromisoformat is faster by ~ x40 (see also: A faster strptime):

%timeit datetime.fromisoformat(s.replace('Z', '+00:00'))
388 ns ± 48.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%timeit dateutil.parser.isoparse(s)
11 µs ± 1.05 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit datetime.strptime(s, '%Y-%m-%dT%H:%M:%S.%f%z')
15.8 µs ± 1.32 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit dateutil.parser.parse(s)
87.8 µs ± 8.54 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

(Python 3.9.12 x64 on Windows 10)


@mikerodent: the point is that fromisoformat parses +00:00 but not Z to aware datetime with tzinfo being UTC. If your input e.g. ends with Z+00:00, you can just remove the Z before feeding it into fromisoformat. Other UTC offsets like e.g. +05:30 will then be parsed to a static UTC offset (not an actual time zone).
F
Flimm
import re
import datetime
s = "2008-09-03T20:56:35.450686Z"
d = datetime.datetime(*map(int, re.split(r'[^\d]', s)[:-1]))

I disagree, this is practically unreadable and as far as I can tell does not take into account the Zulu (Z) which makes this datetime naive even though time zone data was provided.
I find it quite readable. In fact, it's probably the easiest and most performing way to do the conversion without installing additional packages.
This is equivalent of d=datetime.datetime(*map(int, re.split('\D', s)[:-1])) i suppose.
a variation: datetime.datetime(*map(int, re.findall('\d+', s))
This results in a naive datetime object without timezone, right? So the UTC bit gets lost in translation?
P
Peter Mortensen

In these days, Arrow also can be used as a third-party solution:

>>> import arrow
>>> date = arrow.get("2008-09-03T20:56:35.450686Z")
>>> date.datetime
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686, tzinfo=tzutc())

Just use python-dateutil - arrow requires python-dateutil.
Arrow now supports ISO8601. The issues referenced are now closed.
B
Blairg23

Just use the python-dateutil module:

>>> import dateutil.parser as dp
>>> t = '1984-06-02T19:05:00.000Z'
>>> parsed_t = dp.parse(t)
>>> print(parsed_t)
datetime.datetime(1984, 6, 2, 19, 5, tzinfo=tzutc())

Documentation


Isn't this exactly @Flimms answer above?
Where do you see him parsing in seconds? I found this article by trying to get epoch time so I figured someone else would be as well.
This not UTC on my system. Rather, the output in seconds is the unix epoch time as if the date was in my local time zone.
This answer is buggy and should not be accepted. Probably the whole question should be marked as a duplicate of stackoverflow.com/questions/11743019/…
@tripleee Actually I just checked the code and it does appear to return the correct answer: 455051100 (checked at epochconverter.com),,, unless I'm missing something?
m
movermeyer

I have found ciso8601 to be the fastest way to parse ISO 8601 timestamps. As the name suggests, it is implemented in C.

import ciso8601
ciso8601.parse_datetime('2014-01-09T21:48:00.921000+05:30')

The GitHub Repo README shows their >10x speedup versus all of the other libraries listed in the other answers.

My personal project involved a lot of ISO 8601 parsing. It was nice to be able to just switch the call and go 10x faster. :)

Edit: I have since become a maintainer of ciso8601. It's now faster than ever!


This looks like a great library! For those wanting to optimize ISO8601 parsing on Google App Engine, sadly, we can't use it since it's a C library, but your benchmarks were insightful to show that native datetime.strptime() is the next fastest solution. Thanks for putting all that info together!
@hamx0r, be aware that datetime.strptime() is not a full ISO 8601 parsing library. If you are on Python 3.7, you can use the datetime.fromisoformat() method, which is a little more flexible. You might be interested in this more complete list of parsers which should be merged into the ciso8601 README soon.
ciso8601 works quite nice, but one have to first do "pip install pytz", because one cannot parse a timestamp with time zone information without the pytz dependency. Example would look like: dob = ciso8601.parse_datetime(result['dob']['date'])
@Dirk, only in Python 2. But even that should be removed in the next release.
e
enchanter

If you don't want to use dateutil, you can try this function:

def from_utc(utcTime,fmt="%Y-%m-%dT%H:%M:%S.%fZ"):
    """
    Convert UTC time string to time.struct_time
    """
    # change datetime.datetime to time, return time.struct_time type
    return datetime.datetime.strptime(utcTime, fmt)

Test:

from_utc("2007-03-04T21:08:12.123Z")

Result:

datetime.datetime(2007, 3, 4, 21, 8, 12, 123000)

This answer relies upon hard-coding a particular UTC offset (namely "Z", which means +00:00) into the format string passed to strptime. This is a bad idea because it will fail to parse any datetime with a different UTC offset and raise an exception. See my answer that describes how parsing RFC 3339 with strptime is in fact impossible.
It's hard-coded but its sufficient for case when you need to parse zulu only.
@alexander yes - which may be the case if, for instance, you know that your date string was generated with JavaScript's toISOString method. But there's no mention of the limitation to Zulu time dates in this answer, nor did the question indicate that that's all that's needed, and just using dateutil is usually equally convenient and less narrow in what it can parse.
D
Don Kirkby

If you are working with Django, it provides the dateparse module that accepts a bunch of formats similar to ISO format, including the time zone.

If you are not using Django and you don't want to use one of the other libraries mentioned here, you could probably adapt the Django source code for dateparse to your project.


Django's DateTimeField uses this when you set a string value.
P
Peter Mortensen

I've coded up a parser for the ISO 8601 standard and put it on GitHub: https://github.com/boxed/iso8601. This implementation supports everything in the specification except for durations, intervals, periodic intervals, and dates outside the supported date range of Python's datetime module.

Tests are included! :P


P
Peter Mortensen

This works for stdlib on Python 3.2 onwards (assuming all the timestamps are UTC):

from datetime import datetime, timezone, timedelta
datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%fZ").replace(
    tzinfo=timezone(timedelta(0)))

For example,

>>> datetime.utcnow().replace(tzinfo=timezone(timedelta(0)))
... datetime.datetime(2015, 3, 11, 6, 2, 47, 879129, tzinfo=datetime.timezone.utc)

This answer relies upon hard-coding a particular UTC offset (namely "Z", which means +00:00) into the format string passed to strptime. This is a bad idea because it will fail to parse any datetime with a different UTC offset and raise an exception. See my answer that describes how parsing RFC 3339 with strptime is in fact impossible.
In theory, yes, this fails. In practice, I've never encountered an ISO 8601-formatted date that wasn't in Zulu time. For my very-occasional need, this works great and isn't reliant on some external library.
you could use timezone.utc instead of timezone(timedelta(0)). Also, the code works in Python 2.6+ (at least) if you supply utc tzinfo object
Doesn't matter if you've encountered it, it doesn't match the spec.
You can use the %Z for timezone in the most recent versions of Python.
M
Matthew Moisen

I'm the author of iso8601utils. It can be found on GitHub or on PyPI. Here's how you can parse your example:

>>> from iso8601utils import parsers
>>> parsers.datetime('2008-09-03T20:56:35.450686Z')
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686)

D
Damian Yerrick

One straightforward way to convert an ISO 8601-like date string to a UNIX timestamp or datetime.datetime object in all supported Python versions without installing third-party modules is to use the date parser of SQLite.

#!/usr/bin/env python
from __future__ import with_statement, division, print_function
import sqlite3
import datetime

testtimes = [
    "2016-08-25T16:01:26.123456Z",
    "2016-08-25T16:01:29",
]
db = sqlite3.connect(":memory:")
c = db.cursor()
for timestring in testtimes:
    c.execute("SELECT strftime('%s', ?)", (timestring,))
    converted = c.fetchone()[0]
    print("%s is %s after epoch" % (timestring, converted))
    dt = datetime.datetime.fromtimestamp(int(converted))
    print("datetime is %s" % dt)

Output:

2016-08-25T16:01:26.123456Z is 1472140886 after epoch
datetime is 2016-08-25 12:01:26
2016-08-25T16:01:29 is 1472140889 after epoch
datetime is 2016-08-25 12:01:29

Thanks. This is disgusting. I love it.
What an incredible, awesome, beautiful hack! Thanks!
Welcome to the Bad and the Ugly section.
P
Peter Mortensen

Django's parse_datetime() function supports dates with UTC offsets:

parse_datetime('2016-08-09T15:12:03.65478Z') =
datetime.datetime(2016, 8, 9, 15, 12, 3, 654780, tzinfo=<UTC>)

So it could be used for parsing ISO 8601 dates in fields within entire project:

from django.utils import formats
from django.forms.fields import DateTimeField
from django.utils.dateparse import parse_datetime

class DateTimeFieldFixed(DateTimeField):
    def strptime(self, value, format):
        if format == 'iso-8601':
            return parse_datetime(value)
        return super().strptime(value, format)

DateTimeField.strptime = DateTimeFieldFixed.strptime
formats.ISO_INPUT_FORMATS['DATETIME_INPUT_FORMATS'].insert(0, 'iso-8601')

z
zawuza

An another way is to use specialized parser for ISO-8601 is to use isoparse function of dateutil parser:

from dateutil import parser

date = parser.isoparse("2008-09-03T20:56:35.450686+01:00")
print(date)

Output:

2008-09-03 20:56:35.450686+01:00

This function is also mentioned in the documentation for the standard Python function datetime.fromisoformat:

A more full-featured ISO 8601 parser, dateutil.parser.isoparse is available in the third-party package dateutil.


M
Michael Dorner

If pandas is used anyway, I can recommend Timestamp from pandas. There you can

ts_1 = pd.Timestamp('2020-02-18T04:27:58.000Z')    
ts_2 = pd.Timestamp('2020-02-18T04:27:58.000')

Rant: It is just unbelievable that we still need to worry about things like date string parsing in 2021.


pandas is strongly discouraged for this simple case: It depends on pytz, which violates the python standard, and pd.Timestamp is subtly not a compatible datetime object.
Thanks for your comment. Do you have some pointers for me? I was not able to find pytz: github.com/pandas-dev/pandas/blob/… and I’m not sure what Python standard and its violation you are referring to.
See the rant by Paul Ganssle. As for incompatibility, execute both datetime.fromisoformat('2021-01-01T00:00:00+01:00').tzinfo.utc and pandas.Timestamp('2021-01-01T00:00:00+01:00').tzinfo.utc : Not the same at all.
Thank you for pointers to this ongoing work. I didn’t know about that issue, but I really hope they fix it soon! But again: I can’t believe that time parsing is still an issue. :-)
t
theannouncer

Because ISO 8601 allows many variations of optional colons and dashes being present, basically CCYY-MM-DDThh:mm:ss[Z|(+|-)hh:mm]. If you want to use strptime, you need to strip out those variations first.

The goal is to generate a utc datetime object.

2016-06-29T19:36:29.3453Z

datetime.datetime.strptime(timestamp.translate(None, ':-'), "%Y%m%dT%H%M%S.%fZ")

2016-06-29T19:36:29.3453-0400

2008-09-03T20:56:35.450686+05:00

20080903T205635.450686+0500

import re
# this regex removes all colons and all 
# dashes EXCEPT for the dash indicating + or - utc offset for the timezone
conformed_timestamp = re.sub(r"[:]|([-](?!((\d{2}[:]\d{2})|(\d{4}))$))", '', timestamp)
datetime.datetime.strptime(conformed_timestamp, "%Y%m%dT%H%M%S.%f%z" )

%z

ValueError: 'z' is a bad directive in format '%Y%m%dT%H%M%S.%f%z'

Z

%z

import re
import datetime

# this regex removes all colons and all 
# dashes EXCEPT for the dash indicating + or - utc offset for the timezone
conformed_timestamp = re.sub(r"[:]|([-](?!((\d{2}[:]\d{2})|(\d{4}))$))", '', timestamp)

# split on the offset to remove it. use a capture group to keep the delimiter
split_timestamp = re.split(r"[+|-]",conformed_timestamp)
main_timestamp = split_timestamp[0]
if len(split_timestamp) == 3:
    sign = split_timestamp[1]
    offset = split_timestamp[2]
else:
    sign = None
    offset = None

# generate the datetime object without the offset at UTC time
output_datetime = datetime.datetime.strptime(main_timestamp +"Z", "%Y%m%dT%H%M%S.%fZ" )
if offset:
    # create timedelta based on offset
    offset_delta = datetime.timedelta(hours=int(sign+offset[:-2]), minutes=int(sign+offset[-2:]))
    # offset datetime with timedelta
    output_datetime = output_datetime + offset_delta

G
Gordon Wrigley

For something that works with the 2.X standard library try:

calendar.timegm(time.strptime(date.split(".")[0]+"UTC", "%Y-%m-%dT%H:%M:%S%Z"))

calendar.timegm is the missing gm version of time.mktime.


This just ignores the timezone '2013-01-28T14:01:01.335612-08:00' --> parsed as UTC, not PDT
u
user2646026

The python-dateutil will throw an exception if parsing invalid date strings, so you may want to catch the exception.

from dateutil import parser
ds = '2012-60-31'
try:
  dt = parser.parse(ds)
except ValueError, e:
  print '"%s" is an invalid date' % ds

I think it will throw an exception sometimes, it is not guaranteed to throw an exception if it can make a best effort at guessing what the datetime is.
Error hiding is in top three of anti-patterns: Don't.
j
jrc

Nowadays there's Maya: Datetimes for Humans™, from the author of the popular Requests: HTTP for Humans™ package:

>>> import maya
>>> str = '2008-09-03T20:56:35.450686Z'
>>> maya.MayaDT.from_rfc3339(str).datetime()
datetime.datetime(2008, 9, 3, 20, 56, 35, 450686, tzinfo=<UTC>)

C
Community

Thanks to great Mark Amery's answer I devised function to account for all possible ISO formats of datetime:

class FixedOffset(tzinfo):
    """Fixed offset in minutes: `time = utc_time + utc_offset`."""
    def __init__(self, offset):
        self.__offset = timedelta(minutes=offset)
        hours, minutes = divmod(offset, 60)
        #NOTE: the last part is to remind about deprecated POSIX GMT+h timezones
        #  that have the opposite sign in the name;
        #  the corresponding numeric value is not used e.g., no minutes
        self.__name = '<%+03d%02d>%+d' % (hours, minutes, -hours)
    def utcoffset(self, dt=None):
        return self.__offset
    def tzname(self, dt=None):
        return self.__name
    def dst(self, dt=None):
        return timedelta(0)
    def __repr__(self):
        return 'FixedOffset(%d)' % (self.utcoffset().total_seconds() / 60)
    def __getinitargs__(self):
        return (self.__offset.total_seconds()/60,)

def parse_isoformat_datetime(isodatetime):
    try:
        return datetime.strptime(isodatetime, '%Y-%m-%dT%H:%M:%S.%f')
    except ValueError:
        pass
    try:
        return datetime.strptime(isodatetime, '%Y-%m-%dT%H:%M:%S')
    except ValueError:
        pass
    pat = r'(.*?[+-]\d{2}):(\d{2})'
    temp = re.sub(pat, r'\1\2', isodatetime)
    naive_date_str = temp[:-5]
    offset_str = temp[-5:]
    naive_dt = datetime.strptime(naive_date_str, '%Y-%m-%dT%H:%M:%S.%f')
    offset = int(offset_str[-4:-2])*60 + int(offset_str[-2:])
    if offset_str[0] == "-":
        offset = -offset
    return naive_dt.replace(tzinfo=FixedOffset(offset))

A
A T

Initially I tried with:

from operator import neg, pos
from time import strptime, mktime
from datetime import datetime, tzinfo, timedelta

class MyUTCOffsetTimezone(tzinfo):
    @staticmethod
    def with_offset(offset_no_signal, signal):  # type: (str, str) -> MyUTCOffsetTimezone
        return MyUTCOffsetTimezone((pos if signal == '+' else neg)(
            (datetime.strptime(offset_no_signal, '%H:%M') - datetime(1900, 1, 1))
          .total_seconds()))

    def __init__(self, offset, name=None):
        self.offset = timedelta(seconds=offset)
        self.name = name or self.__class__.__name__

    def utcoffset(self, dt):
        return self.offset

    def tzname(self, dt):
        return self.name

    def dst(self, dt):
        return timedelta(0)


def to_datetime_tz(dt):  # type: (str) -> datetime
    fmt = '%Y-%m-%dT%H:%M:%S.%f'
    if dt[-6] in frozenset(('+', '-')):
        dt, sign, offset = strptime(dt[:-6], fmt), dt[-6], dt[-5:]
        return datetime.fromtimestamp(mktime(dt),
                                      tz=MyUTCOffsetTimezone.with_offset(offset, sign))
    elif dt[-1] == 'Z':
        return datetime.strptime(dt, fmt + 'Z')
    return datetime.strptime(dt, fmt)

But that didn't work on negative timezones. This however I got working fine, in Python 3.7.3:

from datetime import datetime


def to_datetime_tz(dt):  # type: (str) -> datetime
    fmt = '%Y-%m-%dT%H:%M:%S.%f'
    if dt[-6] in frozenset(('+', '-')):
        return datetime.strptime(dt, fmt + '%z')
    elif dt[-1] == 'Z':
        return datetime.strptime(dt, fmt + 'Z')
    return datetime.strptime(dt, fmt)

Some tests, note that the out only differs by precision of microseconds. Got to 6 digits of precision on my machine, but YMMV:

for dt_in, dt_out in (
        ('2019-03-11T08:00:00.000Z', '2019-03-11T08:00:00'),
        ('2019-03-11T08:00:00.000+11:00', '2019-03-11T08:00:00+11:00'),
        ('2019-03-11T08:00:00.000-11:00', '2019-03-11T08:00:00-11:00')
    ):
    isoformat = to_datetime_tz(dt_in).isoformat()
    assert isoformat == dt_out, '{} != {}'.format(isoformat, dt_out)

May I ask why did you do frozenset(('+', '-'))? Shouldn't a normal tuple like ('+', '-') be able to accomplish the same thing?
Sure, but isn't that a linear scan rather than a perfectly hashed lookup?
D
Denny Weinberg
def parseISO8601DateTime(datetimeStr):
    import time
    from datetime import datetime, timedelta

    def log_date_string(when):
        gmt = time.gmtime(when)
        if time.daylight and gmt[8]:
            tz = time.altzone
        else:
            tz = time.timezone
        if tz > 0:
            neg = 1
        else:
            neg = 0
            tz = -tz
        h, rem = divmod(tz, 3600)
        m, rem = divmod(rem, 60)
        if neg:
            offset = '-%02d%02d' % (h, m)
        else:
            offset = '+%02d%02d' % (h, m)

        return time.strftime('%d/%b/%Y:%H:%M:%S ', gmt) + offset

    dt = datetime.strptime(datetimeStr, '%Y-%m-%dT%H:%M:%S.%fZ')
    timestamp = dt.timestamp()
    return dt + timedelta(hours=dt.hour-time.gmtime(timestamp).tm_hour)

Note that we should look if the string doesn't ends with Z, we could parse using %z.


90% dead code, 10% buggy: do not use!