In addition I documented the marc8_to_unicode function more so that it hopefully makes clear you aren’t supposed to pass in a serialized marc record, but just a chunk of text extracted from the record that you’d like to translate.

Thanks for the info and the patch!

>>> reader = MARCReader(‘bibs105000.out’)
>>> for record in reader:
… print record['245']['a']
…
Traceback (most recent call last):
File “”, line 1, in ?
File “build/bdist.linux-x86_64/egg/pymarc/reader.py”, line 51, in next
ValueError: invalid literal for int(): bibs1
>>>

But I noticed “self.reader = pymarc.MARCReader(file(‘test/test.dat’))” in test/reader.py, so I:

>>> marc8_file = file(‘bibs105000.out’)
>>> reader = MARCReader(marc8_file)
>>> for record in reader:
… print record['245']['a']
…
Microeconomics :
The multilateral development banks :
The Accountants digest.
Achievement.
ALA bulletin.
Acta arithmetica.
Acta crystallographica.
ASLE transactions.
Acta mathematica.
Acta mechanica.
Acta physica Polonica,
Acta physica Austriaca /
Acta physica Polonica.
Acta polytechnica Scandinavica.
[etc. -- big old list of titles, streaming by, very fast]

So MARCReader expects a file object, not a filename. Does that doc string need updating or did I misread it?

I then tried to test marc8_to_unicode:

>>> utf8_file = marc8_to_unicode(marc8_file)
Traceback (most recent call last):
File “”, line 1, in ?
File “build/bdist.linux-x86_64/egg/pymarc/marc8.py”, line 8, in marc8_to_unicode
File “build/bdist.linux-x86_64/egg/pymarc/marc8.py”, line 43, in translate
TypeError: len() of unsized object
>>>

Looking at test/marc8.py I saw marc8_to_unicode expects a string, so I:

>>> marc8_file = file(‘bibs105000.out’)
>>> marc8_file_str = marc8_file.read()
>>> utf8_file_str = marc8_to_unicode(marc8_file_str)
couldn’t find 66 69 221 221
couldn’t find 66 69 220 220
couldn’t find 66 69 221 221
couldn’t find 66 69 220 220
couldn’t find 66 69 221 221
couldn’t find 66 69 221 221
couldn’t find 66 69 220 220
couldn’t find 66 69 220 220
couldn’t find 66 69 221 221
>>>

This took a long time, running at about 98% of my cpu and 20% of my ~4GB of memory. So I timed it (only 3 times because, you know, I’m impatient):

>>> t = Timer(stmt=’utf8_file_str = marc8_to_unicode(marc8_file_str)’, setup=’marc8_file = file(“bibs105000.out”); marc8_file_str = marc8_file.read(); from pymarc import marc8_to_unicode’)
>>> t.timeit(3)
[a bunch of the "couldn't find 66 69 22[01] 22[01]“, three times over]
786.2618350982666

So it took a little while. How big is the file?

>>> len(marc8_file_str)
93747869
>>> len(utf8_file_str)
87474412

Hmm. Those should match, right? Well, let’s see if we can read the new file:

>>> pymarc_utf8_file = file(‘bibs105000_pymarc_utf8.dat’, ‘w’)
>>> pymarc_utf8_file.write(utf8_file_str.encode(‘utf8′))
>>> reader = MARCReader(file(‘bibs105000_pymarc_utf8.dat’))
>>> for record in reader:
… print record['245']['a']
…
None
Traceback (most recent call last):
File “”, line 1, in ?
File “build/bdist.linux-x86_64/egg/pymarc/reader.py”, line 54, in next
File “build/bdist.linux-x86_64/egg/pymarc/record.py”, line 46, in __init__
File “build/bdist.linux-x86_64/egg/pymarc/record.py”, line 123, in decodeMARC
pymarc.exceptions.BaseAddressInvalid: Base address exceeds size of record
>>>

Have I done something screwy, or is it something in the file? I can get the file to you if you want to test on your system. I also have a utf8 file to compare the output against, produced by `yaz-marcdump -f MARC-8 -t UTF-8 -I bibs105000.out > bibs105000_utf8.out`, and of course the diff is a mile long. If I get time later I’ll test line by line, as done in test/marc8.py.

Apologies for the long long comment — should be on some mailing list or something.