ChatGPT解决这个技术问题 Extra ChatGPT

Reading a UTF8 CSV file with Python

I am trying to read a CSV file with accented characters with Python (only French and/or Spanish characters). Based on the Python 2.5 documentation for the csvreader (http://docs.python.org/library/csv.html), I came up with the following code to read the CSV file since the csvreader supports only ASCII.

def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs):
    # csv.py doesn't do Unicode; encode temporarily as UTF-8:
    csv_reader = csv.reader(utf_8_encoder(unicode_csv_data),
                            dialect=dialect, **kwargs)
    for row in csv_reader:
        # decode UTF-8 back to Unicode, cell by cell:
        yield [unicode(cell, 'utf-8') for cell in row]

def utf_8_encoder(unicode_csv_data):
    for line in unicode_csv_data:
        yield line.encode('utf-8')

filename = 'output.csv'
reader = unicode_csv_reader(open(filename))
try:
    products = []
    for field1, field2, field3 in reader:
        ...

Below is an extract of the CSV file I am trying to read:

0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu
0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris
0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert
...

Even though I try to encode/decode to UTF-8, I am still getting the following exception:

Traceback (most recent call last):
  File ".\Test.py", line 53, in <module>
    for field1, field2, field3 in reader:
  File ".\Test.py", line 40, in unicode_csv_reader
    for row in csv_reader:
  File ".\Test.py", line 46, in utf_8_encoder
    yield line.encode('utf-8', 'ignore')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 68: ordinal not in range(128)

How do I fix this?

Martin, if you're around, would you consider switching the accepted answer from Martelli's Python 2 only answer.

A
Alex Martelli

The .encode method gets applied to a Unicode string to make a byte-string; but you're calling it on a byte-string instead... the wrong way 'round! Look at the codecs module in the standard library and codecs.open in particular for better general solutions for reading UTF-8 encoded text files. However, for the csv module in particular, you need to pass in utf-8 data, and that's what you're already getting, so your code can be much simpler:

import csv

def unicode_csv_reader(utf8_data, dialect=csv.excel, **kwargs):
    csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
    for row in csv_reader:
        yield [unicode(cell, 'utf-8') for cell in row]

filename = 'da.csv'
reader = unicode_csv_reader(open(filename))
for field1, field2, field3 in reader:
  print field1, field2, field3 

PS: if it turns out that your input data is NOT in utf-8, but e.g. in ISO-8859-1, then you do need a "transcoding" (if you're keen on using utf-8 at the csv module level), of the form line.decode('whateverweirdcodec').encode('utf-8') -- but probably you can just use the name of your existing encoding in the yield line in my code above, instead of 'utf-8', as csv is actually going to be just fine with ISO-8859-* encoded bytestrings.


Does this mean the example in the python docs (where OP copy & pasted from) is wrong? What is the point of the extra encoding step it does if it breaks when you give it a unicode csv?
A
Antti Haapala -- Слава Україні

Python 2.X

There is a unicode-csv library which should solve your problems, with added benefit of not naving to write any new csv-related code.

Here is a example from their readme:

>>> import unicodecsv
>>> from cStringIO import StringIO
>>> f = StringIO()
>>> w = unicodecsv.writer(f, encoding='utf-8')
>>> w.writerow((u'é', u'ñ'))
>>> f.seek(0)
>>> r = unicodecsv.reader(f, encoding='utf-8')
>>> row = r.next()
>>> print row[0], row[1]
é ñ

Python 3.X

In python 3 this is supported out of the box by the build-in csv module. See this example:

import csv
with open('some.csv', newline='', encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)

encoding='utf-8-sig' helps if your CSV file has a BOM prefix U+FEFF. Opening the file with that encoding will automatically strip the BOM. Otherwise it confuses csv into thinking the first field name starts with the BOM character and it fails to strip the quotes, and so reader.fieldnames[0] can be '\ufeff"Date"' instead of 'Date'.
N
Neuron

If you want to read a CSV File with encoding utf-8, a minimalistic approach that I recommend you is to use something like this:

with open(file_name, encoding="utf8") as csv_file:

With that statement, you can use later a CSV reader to work with.


Is it possible that this is Python 3 only? It fails for me, in Python 2. It doesn't accept the encoding in open
@Zvika yes, in python 3 this solution works: open('file.csv', 'r', encoding="ISO8859")
I would also add open(file_name, "rt", encoding='utf-8'), that is, open file in "read text" mode
@JimmyLeeJones 'r' and 'rt' are the same since by default open use "read text" mode
C
Community

Also checkout the answer in this post: https://stackoverflow.com/a/9347871/1338557

It suggests use of library called ucsv.py. Short and simple replacement for CSV written to address the encoding problem(utf-8) for Python 2.7. Also provides support for csv.DictReader

Edit: Adding sample code that I used:

import ucsv as csv

#Read CSV file containing the right tags to produce
fileObj = open('awol_title_strings.csv', 'rb')
dictReader = csv.DictReader(fileObj, fieldnames = ['titles', 'tags'], delimiter = ',', quotechar = '"')
#Build a dictionary from the CSV file-> {<string>:<tags to produce>}
titleStringsDict = dict()
for row in dictReader:
    titleStringsDict.update({unicode(row['titles']):unicode(row['tags'])})

you should put some details of that link in your answer, just in case the link goes broken\
#Downvoter- Not sure why you thought its of no use. The ucsv library worked just fine for me. Helped resolve the unicde error that I had been struggling with since 2 days. If you were looking for some sample code, here it goes in the edit @Yaje- I have given some details; also the sample code. And corrected the link as well, that was earlier pointing to some other post.
Any particular reason you are opening a text file as a binary? 'rb' is for opening binary files.
Í
Ícaro

Using codecs.open as Alex Martelli suggested proved to be useful to me.

import codecs

delimiter = ';'
reader = codecs.open("your_filename.csv", 'r', encoding='utf-8')
for line in reader:
    row = line.split(delimiter)
    # do something with your row ...

It wouldn't work with all CSV, following is a valid csv row: "Foo Bar; Baz"; 231; 313; ";;;"; 1;
You import the csv module but do not use it.
v
van

The link to the help page is the same for python 2.6 and as far as I know there was no change in the csv module since 2.5 (besides bug fixes). Here is the code that just works without any encoding/decoding (file da.csv contains the same data as the variable data). I assume that your file should be read correctly without any conversions.

test.py:

## -*- coding: utf-8 -*-
#
# NOTE: this first line is important for the version b) read from a string(unicode) variable
#

import csv

data = \
"""0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu
0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris
0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert"""

# a) read from a file
print 'reading from a file:'
for (f1, f2, f3) in csv.reader(open('da.csv'), dialect=csv.excel):
    print (f1, f2, f3)

# b) read from a string(unicode) variable
print 'reading from a list of strings:'
reader = csv.reader(data.split('\n'), dialect=csv.excel)
for (f1, f2, f3) in reader:
    print (f1, f2, f3)

da.csv:

0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu
0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris
0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert

I wonder which version of python would this work in? I get errors with both 2.7 and 3.5. "ValueError: not enough values to unpack (expected 3, got 1)"
@eis: I can imagine that on your system comma is not a default delimiter. Try to add delimiter=',' instead of dialect=csv.excel.
O
OfirD

Worth noting that if nothing worked for you, you may have forgotten to escape your path. For example, this code:

f = open("C:\Some\Path\To\file.csv")

Would result in an error:

SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

To fix, simply do:

f = open("C:\\Some\\Path\\To\\file.csv")

g
gimel

Looking at the Latin-1 unicode table, I see the character code 00E9 "LATIN SMALL LETTER E WITH ACUTE". This is the accented character in your sample data. A simple test in Python shows that UTF-8 encoding for this character is different from the unicode (almost UTF-16) encoding.

>>> u'\u00e9'
u'\xe9'
>>> u'\u00e9'.encode('utf-8')
'\xc3\xa9'
>>> 

I suggest you try to encode("UTF-8") the unicode data before calling the special unicode_csv_reader(). Simply reading the data from a file might hide the encoding, so check the actual character values.


P
Piotr Pęczek

Had the same problem on another server, but realized that locales are messed.

export LC_ALL="en_US.UTF-8"

fixed the problem


M
Mohsen Navazani

I suggest using the following code when you want to open the file.

open('/content/sasan.csv' , 'r', encoding='utf-8', errors='ignore' )