Comments on: python, re-encoding incorrected encoded string

By: olpa

olpa — Fri, 17 Aug 2007 11:34:42 +0000

Can’t believe! Actually, the problematic symbol was the Russian small letter “r’. Old-time programmers can remember a lot of problems with this letter in 1990th years.

Thanks Adobe for reminding it!

By: olpa

olpa — Fri, 17 Aug 2007 11:31:18 +0000

The very final solution is to use the first variant, but wrap chr(ord(ch)) to a try-except block, and replace bad characters with “?”. It’s ok for me for now,

By: olpa

olpa — Fri, 17 Aug 2007 11:24:39 +0000

Don’t worry, the code is correct. In my case, my source data is actually in MAC ROMAN encoding, and it contains a character (less or equal), which is not in latin-1 encoding.

Probably it’s also the reason why I failed with a simple solution: I can’t convert non-latin-1 data to latin-1.

By: olpa

olpa — Fri, 17 Aug 2007 11:16:37 +0000

Oh, god. The code is just invalid. For my goals, it works:

import codecs
w1251dec = codecs.getdecoder('windows-1251')

def reenc(s):
  s2 = ''
  for ch in s:
    try:
      ch2 = w1251dec(ch)[0]
      s2 = s2 + ch2
    except UnicodeEncodeError:
      s2 = s2 + ch
  return s2

s = unicode("\\xc3\\xeb\\xe0\\xe2\\xe0", 'latin-1')
s = u'\\u2264' + s
print s, repr(s)
s = reenc(s)
print s, repr(s)

Still TODO: how to convert an unicode string to an usual string in a given encoding.