Comments on: python, re-encoding incorrected encoded string

By: Oleg

Oleg — Tue, 07 Jul 2009 04:32:49 +0000

I've noticed that after all the code changes it looks like:

def reenc_next(cin, cout, i, s):
  try:
    ch  = chr(i)
    ch1 = cin(ch)[0]
    ch2 = cout(ch)[0]
    s = s.replace(ch1, ch2)
  except:
    pass
  return s

def reenc_string(from_enc, to_enc, s):
  from_dec = codecs.getdecoder(from_enc)
  to_dec   = codecs.getdecoder(to_enc)
  for i in xrange(256):
    s = reenc_next(from_dec, to_dec, i, s)
  return s

By: olpa

olpa — Fri, 17 Aug 2007 11:34:42 +0000

Can’t believe! Actually, the problematic symbol was the Russian small letter “r’. Old-time programmers can remember a lot of problems with this letter in 1990th years.

Thanks Adobe for reminding it!

By: olpa

olpa — Fri, 17 Aug 2007 11:31:18 +0000

The very final solution is to use the first variant, but wrap chr(ord(ch)) to a try-except block, and replace bad characters with “?”. It’s ok for me for now,

By: olpa

olpa — Fri, 17 Aug 2007 11:24:39 +0000

Don’t worry, the code is correct. In my case, my source data is actually in MAC ROMAN encoding, and it contains a character (less or equal), which is not in latin-1 encoding.

Probably it’s also the reason why I failed with a simple solution: I can’t convert non-latin-1 data to latin-1.

By: olpa

olpa — Fri, 17 Aug 2007 11:16:37 +0000

Oh, god. The code is just invalid. For my goals, it works:

import codecs
w1251dec = codecs.getdecoder('windows-1251')

def reenc(s):
  s2 = ''
  for ch in s:
    try:
      ch2 = w1251dec(ch)[0]
      s2 = s2 + ch2
    except UnicodeEncodeError:
      s2 = s2 + ch
  return s2

s = unicode("\\xc3\\xeb\\xe0\\xe2\\xe0", 'latin-1')
s = u'\\u2264' + s
print s, repr(s)
s = reenc(s)
print s, repr(s)

Still TODO: how to convert an unicode string to an usual string in a given encoding.