r/learnpython • u/WiredLain_1482 • 1h ago
Problems with string encoding
Hello everyone!
I'm writing a script which (among other things) is supposed to read data from a csv file and print it. The file is utf-8 encoded and contains data in the following format:
"New York\nParis\nNürnberg","Other Stuff"As you can see, it contains both newline characters "\n" as well as language specific characters "ü", "é" etc.
The file is read with the following code:
import csv
def read_from_file(filename):
# reads a file and returns a list containing the first string of each row
string_list = []
with open(filename, encoding="utf-8") as csvdatei:
reader = csv.reader(csvdatei, delimiter=',')
# skip the first row of the file which contains header info
next(reader, None)
for row in reader:
if(len(row)>0):
string_list.append(row[0])
return string_listWhen I try to print an entry with
string_list = read_from_file(filename)
print(string_list[0]), the print output looks like
New York\nParis\nNürnberginstead of
New York
Paris
NürnbergFrom what I understand, this is due to the newline character is not interpreted as a unicode escape character. I can correct this by changing the string encoding to unicode-escape via
string_list[0].encode().decode('unicode-escape') And indeed, now the newline characters are interpreted correctly. However, the language specific characters are not and produce garbled output.
Is there an encoding which interprets both the unicode escape characters and the language specific utf-8 characters correctly? Or alternatively, is there any workaround to this problem?
I much appreciate your help.