I am trying to read a CSV file with accented characters with Python (only French and/or Spanish characters). Based on the Python 2.5 documentation for the csvreader (http://docs.python.org/library/csv.html), I came up with the following code to read the CSV file since the csvreader supports only ASCII.
def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs):
# csv.py doesn't do Unicode; encode temporarily as UTF-8:
csv_reader = csv.reader(utf_8_encoder(unicode_csv_data),
dialect=dialect, **kwargs)
for row in csv_reader:
# decode UTF-8 back to Unicode, cell by cell:
yield [unicode(cell, 'utf-8') for cell in row]
def utf_8_encoder(unicode_csv_data):
for line in unicode_csv_data:
yield line.encode('utf-8')
filename = 'output.csv'
reader = unicode_csv_reader(open(filename))
try:
products = []
for field1, field2, field3 in reader:
...
Below is an extract of the CSV file I am trying to read:
0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu
0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris
0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert
...
Even though I try to encode/decode to UTF-8, I am still getting the following exception:
Traceback (most recent call last):
File ".\Test.py", line 53, in <module>
for field1, field2, field3 in reader:
File ".\Test.py", line 40, in unicode_csv_reader
for row in csv_reader:
File ".\Test.py", line 46, in utf_8_encoder
yield line.encode('utf-8', 'ignore')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 68: ordinal not in range(128)
How do I fix this?
The .encode
method gets applied to a Unicode string to make a byte-string; but you're calling it on a byte-string instead... the wrong way 'round! Look at the codecs
module in the standard library and codecs.open
in particular for better general solutions for reading UTF-8 encoded text files. However, for the csv
module in particular, you need to pass in utf-8 data, and that's what you're already getting, so your code can be much simpler:
import csv
def unicode_csv_reader(utf8_data, dialect=csv.excel, **kwargs):
csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
for row in csv_reader:
yield [unicode(cell, 'utf-8') for cell in row]
filename = 'da.csv'
reader = unicode_csv_reader(open(filename))
for field1, field2, field3 in reader:
print field1, field2, field3
PS: if it turns out that your input data is NOT in utf-8, but e.g. in ISO-8859-1, then you do need a "transcoding" (if you're keen on using utf-8 at the csv
module level), of the form line.decode('whateverweirdcodec').encode('utf-8')
-- but probably you can just use the name of your existing encoding in the yield
line in my code above, instead of 'utf-8'
, as csv
is actually going to be just fine with ISO-8859-* encoded bytestrings.
Python 2.X
There is a unicode-csv library which should solve your problems, with added benefit of not naving to write any new csv-related code.
Here is a example from their readme:
>>> import unicodecsv
>>> from cStringIO import StringIO
>>> f = StringIO()
>>> w = unicodecsv.writer(f, encoding='utf-8')
>>> w.writerow((u'é', u'ñ'))
>>> f.seek(0)
>>> r = unicodecsv.reader(f, encoding='utf-8')
>>> row = r.next()
>>> print row[0], row[1]
é ñ
Python 3.X
In python 3 this is supported out of the box by the build-in csv
module. See this example:
import csv
with open('some.csv', newline='', encoding='utf-8') as f:
reader = csv.reader(f)
for row in reader:
print(row)
encoding='utf-8-sig'
helps if your CSV file has a BOM prefix U+FEFF. Opening the file with that encoding will automatically strip the BOM. Otherwise it confuses csv
into thinking the first field name starts with the BOM character and it fails to strip the quotes, and so reader.fieldnames[0] can be '\ufeff"Date"'
instead of 'Date'
.
If you want to read a CSV File with encoding utf-8, a minimalistic approach that I recommend you is to use something like this:
with open(file_name, encoding="utf8") as csv_file:
With that statement, you can use later a CSV reader to work with.
encoding
in open
open('file.csv', 'r', encoding="ISO8859")
open
use "read text" mode
Also checkout the answer in this post: https://stackoverflow.com/a/9347871/1338557
It suggests use of library called ucsv.py. Short and simple replacement for CSV written to address the encoding problem(utf-8) for Python 2.7. Also provides support for csv.DictReader
Edit: Adding sample code that I used:
import ucsv as csv
#Read CSV file containing the right tags to produce
fileObj = open('awol_title_strings.csv', 'rb')
dictReader = csv.DictReader(fileObj, fieldnames = ['titles', 'tags'], delimiter = ',', quotechar = '"')
#Build a dictionary from the CSV file-> {<string>:<tags to produce>}
titleStringsDict = dict()
for row in dictReader:
titleStringsDict.update({unicode(row['titles']):unicode(row['tags'])})
Using codecs.open
as Alex Martelli suggested proved to be useful to me.
import codecs
delimiter = ';'
reader = codecs.open("your_filename.csv", 'r', encoding='utf-8')
for line in reader:
row = line.split(delimiter)
# do something with your row ...
csv
module but do not use it.
The link to the help page is the same for python 2.6 and as far as I know there was no change in the csv module since 2.5 (besides bug fixes). Here is the code that just works without any encoding/decoding (file da.csv contains the same data as the variable data). I assume that your file should be read correctly without any conversions.
test.py:
## -*- coding: utf-8 -*-
#
# NOTE: this first line is important for the version b) read from a string(unicode) variable
#
import csv
data = \
"""0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu
0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris
0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert"""
# a) read from a file
print 'reading from a file:'
for (f1, f2, f3) in csv.reader(open('da.csv'), dialect=csv.excel):
print (f1, f2, f3)
# b) read from a string(unicode) variable
print 'reading from a list of strings:'
reader = csv.reader(data.split('\n'), dialect=csv.excel)
for (f1, f2, f3) in reader:
print (f1, f2, f3)
da.csv:
0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu
0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris
0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert
delimiter=','
instead of dialect=csv.excel
.
Worth noting that if nothing worked for you, you may have forgotten to escape your path. For example, this code:
f = open("C:\Some\Path\To\file.csv")
Would result in an error:
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
To fix, simply do:
f = open("C:\\Some\\Path\\To\\file.csv")
Looking at the Latin-1
unicode table, I see the character code 00E9
"LATIN SMALL LETTER E WITH ACUTE". This is the accented character in your sample data. A simple test in Python
shows that UTF-8
encoding for this character is different from the unicode (almost UTF-16
) encoding.
>>> u'\u00e9'
u'\xe9'
>>> u'\u00e9'.encode('utf-8')
'\xc3\xa9'
>>>
I suggest you try to encode("UTF-8")
the unicode data before calling the special unicode_csv_reader()
. Simply reading the data from a file might hide the encoding, so check the actual character values.
Had the same problem on another server, but realized that locales are messed.
export LC_ALL="en_US.UTF-8"
fixed the problem
I suggest using the following code when you want to open the file.
open('/content/sasan.csv' , 'r', encoding='utf-8', errors='ignore' )
Success story sharing