Databases
StackOverflow2013
Postsdbo
POIs this a sensible approach for an EBCDIC (CP500) to Latin-1 converter?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs this a sensible approach for an EBCDIC (CP500) to Latin-1 converter?
    primarykey
    data
    text
    <p>I have to convert a number of large files (up to 2GB) of EBCDIC 500 encoded files to Latin-1. Since I could only find EBCDIC to ASCII converters (dd, recode) and the files contain some additional proprietary character codes, I thought I'd write my own converter.</p> <p>I have the <a href="http://www.tachyonsoft.com/cp01047.htm" rel="nofollow noreferrer">character mapping</a> so I'm interested in the technical aspects.</p> <p>This is my approach so far:</p> <pre><code># char mapping lookup table EBCDIC_TO_LATIN1 = { 0xC1:'41', # A 0xC2:'42', # B # and so on... } BUFFER_SIZE = 1024 * 64 ebd_file = file(sys.argv[1], 'rb') latin1_file = file(sys.argv[2], 'wb') buffer = ebd_file.read(BUFFER_SIZE) while buffer: latin1_file.write(ebd2latin1(buffer)) buffer = ebd_file.read(BUFFER_SIZE) ebd_file.close() latin1_file.close() </code></pre> <p>This is the function that does the converting:</p> <pre><code>def ebd2latin1(ebcdic): result = [] for ch in ebcdic: result.append(EBCDIC_TO_LATIN1[ord(ch)]) return ''.join(result).decode('hex') </code></pre> <p>The question is whether or not this is a sensible approach from an engineering standpoint. Does it have some serious design issues? Is the buffer size OK? And so on...</p> <p>As for the "proprietary characters" that some don't believe in: Each file contains a year's worth of patent documents in SGML format. The patent office has been using EBCDIC until they switched to Unicode in 2005. So there are thousands of documents <em>within</em> each file. They are separated by some hex values that are not part of any IBM specification. They were added by the patent office. Also, at the beginning of each file there are a few digits in ASCII that tell you about the length of the file. I don't really need that information but if I want to process the file so I have to deal with them.</p> <p>Also:</p> <pre><code>$ recode IBM500/CR-LF..Latin1 file.ebc recode: file.ebc failed: Ambiguous output in step `CR-LF..data' </code></pre> <p>Thanks for the help so far.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USEisen
    1. USEisen
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIs this a sensible approach for an EBCDIC (CP500) to Latin-1 converter?
      1. This table or related slice is empty.
      1. VTDownMod
    1. CO... Whats the question?
      singulars
      1. POIs this a sensible approach for an EBCDIC (CP500) to Latin-1 converter?
      1. USMatthew Scharley
    2. COYou should ask a question or you risk having your question closed. "Are there ways to make this code faster?" / "Am I taking a sensible approach, or is there a better way?"
      singulars
      1. POIs this a sensible approach for an EBCDIC (CP500) to Latin-1 converter?
      1. USRichieHindle
    3. CO@Eisen: You say that your files are encoded in "EBCDIC 500" plus "additional proprietary character codes" but then say you have the mapping and give a link to IBM code page 1047. Any extra codes in IBM cp1047 are no more "proprietary" than those in IBM cp500 :-) Please clarify what character set you need: cp500? cp500 + extras? cp1047? cp1047 + extras? Something else? Note: one gets the impression from googling "code page 1047" that requesting that 1047 be added to Python's codec repertoire might not be an unreasonable idea.
      singulars
      1. POIs this a sensible approach for an EBCDIC (CP500) to Latin-1 converter?
      1. USJohn Machin
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload