Compiling Python with MinGW gcc under Windows

Oleg Paraschenko

Document is provided under the terms of GNU Free Documentation License.

Abstract

You need a C compiler if you wish to compile Python for Windows. The sad thing is that this compiler is called “Microsoft Visual C++”. In order to use other compiler, you have to spent a time and effrots. This article contains a log of succesfull compilation of Python 2.2. You can use this log as starting point for your own compilation.

25 Feb 2005. Project pyMinGW obsoletes this page. Anyway, the log contains a set of MinGW tricks, so the page will never be deleted.


Table of Contents

Introduction
First attempt. Using right way.
Second attempt. Using right win32 way.
Attempt to compile a module

Не было у бабы забот — купила баба порося.

Introduction

Compilation of Python for Windows using gcc compiler from MinGW is one of the frequently asked questions in Python conferences. I succesfully did it. In short:

  • it is possible;

  • result works;

  • you must not do it unless you know for sure that you must do it;

  • you must have a good knowledge of C and of debugging coredumps;

  • in most cases all you need is to complile an extension module. This task is described by Sebastien Sauvage in article “Writing C/C++ Python extensions without Microsoft Visual C++”.

Finally, working result was thrown away. Reason is simple: number of fixes is too big and process should be performed by experienced person. It is good for fun, but it is not good for business.

Anyway, I think that description of process of compilation is interesting for others. Indeed, it is not description, it is formatted but unedited log. But if you are ready to compile Python, then you have enough skills to get useful tips.

I compiled Python 2.2.2 on win2k professional using:

MSYS 1.0.9;
MinGW 3.1.0-1;
gcc-core 3.3.1-20030804;
w32api-2.4;
good code editor (vim).

First attempt. Using “right” way.

Right way to compile a program is to use configure, make, make install. With Python on Windows, it is a wrong way.

This way is wrong because system-dependant code for Windows is not in main sources, but in separate folder. It is possible to compile executable file, it will look like working, but I afraid to use such result.

This section is not a log, but a set of patches. Some of them are wrong, but it is possible to produce python.exe which you can run. If you are curious, look at it and estimate how it is hard to introduce win32-specific code into the main sources tree. I stopped to work when start to understand that next issues are over my patience:

  • handling of separators (/ vs \);

  • loading of dynamic libraries.

Example 1. Configure script forconf.w32

./configure --with-threads=no --prefix=.

Example 2. Include/osdefs.h

#ifndef DELIM
#define DELIM ':'
#endif

<new>
#undef SEP
#undef ALTSEP
#undef MAXPATHLEN
#undef DELIM
#define SEP '\\'      or /
#define ALTSEP '/'    or \
#define MAXPATHLEN 256
#define DELIM ';'
</new>

#ifdef __cplusplus

Example 3. Modules/getpath.c

reduce(char *dir)
{
    size_t i = strlen(dir);
<old>		while (i > 0 && dir[i] != SEP)</old>
<new>    while (i > 0 && (dir[i] != SEP) && (dir[i] != ALTSEP))</new>
        --i;
    dir[i] = '\0';
}

Example 4. Modules/getpath.c

<new>//</new>#ifndef PYTHONPATH
<new>//</new>#define PYTHONPATH PREFIX "/lib/python" VERSION ":" \
<new>//</new>              EXEC_PREFIX "/lib/python" VERSION "/lib-dynload"
<new>#define PYTHONPATH PREFIX "\\lib"</new>
<new>//</new>#endif

OR:

<new>
#ifndef PYTHONPATH
#define PYTHONPATH PREFIX "/lib/python" VERSION ";" \
              EXEC_PREFIX "/lib/python" VERSION "/lib-dynload"
#endif
</new>

Example 5. Modules/getpath.c

<old>
static void
copy_absolute(char *path, char *p)
{
    if (p[0] == SEP)
        strcpy(path, p);
		else {
</old>
<new>
static void
copy_absolute(char *path, char *p)
{
    if (p[0] == SEP) {
        strcpy(path, p);
<v1>    } else if ((p[1] == ':') && (p[2] == '\\')) {</v1>
    } else if (p[1] == ':') {
        strcpy(path, p);	
		} else {
</new>
<new>
copy_absolute(char *path, char *p)
{
    if ((p[0] == SEP) || p[1] == ':') 
        strcpy(path, p);
    else {
</new>

Example 6. Modules/getpath.c

            else {
                <new>//</new>strcat(buf, defpath);
                break;
            }

Example 7. PC/getpathp.c

<new>
#define MS_WIN32 1
#define MS_WINDOWS 1
</new>

static char prefix[MAXPATHLEN+1];
static char progpath[MAXPATHLEN+1];
static char *module_search_path = NULL;

Example 8. Modules/posixmodule.c

extern char **environ;
#endif /* !_MSC_VER */

<new>
#undef HAVE_TTYNAME
#undef HAVE_FORK
#undef HAVE_GETEGID
#undef HAVE_GETEUID
#undef HAVE_GETGID
#undef HAVE_GETPPID
#undef HAVE_GETUID
#undef HAVE_KILL
#undef HAVE_WAIT
#undef HAVE_WAITPID
#undef HAVE_PIPE
#undef HAVE_POPEN
</new>

...

#ifndef HAVE_UNISTD_H
<new>//</new>#if defined(PYCC_VACPP)
<new>//</new>extern int mkdir(char *);
<new>//</new>#else
<new>//</new>#if ( defined(__WATCOMC__) || defined(_MSC_VER) ) \
  && !defined(__QNX__)
<new>extern int mkdir(const char *);</new>
<new>//</new>#else
<new>//</new>extern int mkdir(const char *, mode_t);
<new>//</new>#endif
<new>//</new>#endif
...
<new>//</new>#if ( defined(__WATCOMC__) || defined(_MSC_VER) || \
  defined(PYCC_VACPP) ) && !defined(__QNX__)
<new>	res = mkdir(path);</new>
<new>//</new>#else
<new>//</new>	res = mkdir(path, mode);
<new>//</new>#endif

Example 9. Makefile

<new># </new>LIBOBJS=	 Python/fileblocks.o
<new>LIBOBJS=</new>

Example 10. Python/pythonrun

<old>int Py_NoSiteFlag; /* Suppress 'import site' */</old>
<new>int Py_NoSiteFlag = 1; /* Suppress 'import site' */</new>

Example 11. Lib/os.py

in posix:
<old>
    linesep = '\n'
    curdir = '.'; pardir = '..'; sep = '/'; pathsep = ':'; altsep = '/'
</old>	
to

Second attempt. Using right win32 way.

This attempt was succesfull. Most of self-tests were passed.

Please note that this log is presented as-is, without any attempt to make a text with logic.

Started over.

Where to read about win32?

$ find . -type f | xargs grep -l -i win32 | tee log

Interesting results:

./Misc/cheatsheet 
./PC/example_nt/readme.txt 
./PC/readme.txt 
./PCbuild/readme.txt 
no: ./Misc/cheatsheet
no: ./PC/example_nt/readme.txt, it's for building modules
good: ./PC/readme.txt, general description of bulding for PC
yes: ./PCbuild/readme.txt, build Python for Win32 platforms

./PCbuild/readme.txt says “For other Windows ... compilers, see ../PC/readme.txt

All is too hard...

Small idea: Configure by cygwin, check win32 settings, compile by mingw

(I know, I should check “nt” module instead of “posix”)

Let start configure with cygwin:

$ ./configure --with-threads=no --prefix=.

After it: check that no name of folder inside!

I decided that idea is bad. Cygwin is posix emulator, not just a compiler for win32. But MinGW may compile for win32. So delete folder and unpack again.

With MinGW:

$ ./configure --with-threads=no --prefix=.
...
creating Modules/Setup.config
creating pyconfig.h
creating Setup
creating Setup.local
creating Makefile

So, ideally python depends only from pyconfig.h. But I know about folder ./PCbuild.

Carefully reading ./PCbuild/readme.txt. Need to check option in pcbuild.dsw. Then pythoncore.dsp and python.dsp. From pythoncode.dsp, set of preprocessor options:

/D "WIN32"
/D "NDEBUG"
/D "_WINDOWS"
/D "USE_DL_EXPORT"

And from pythonw.dsp

/D "_MBCS"

Looking at .dsp I see that another set of source files is used.

Copy Makefile to Makefile.orig

Add to OPT:

-DWIN32 -DNDEBUG -D_WINDOWS -DUSE_DL_EXPORT

Now it is time to redefile what files to compile: comment-out all obj-variables

MODOBJS
LIBOBJS
MACHDEP_OBJS
UNICODE_OBJS
MODULE_OBJS
SIGNAL_OBJS
POBJS
PGOBJS
...

Too lazy... It is better to find “dsp2make” or “dsp mingw

First attempt is to use MinGW dsw2make

Folder PCbuild is copied as build. In this folder:

$ gawk -f /mingw/bin/dsw2mak pcbuild.dsw

With some errors, something is done

Let start with code.

$ make -f pythoncore.mak 2>&1 | tee pythoncore.log

A lot, a lot of warnings and finally an error:

../Python/dynload_win.c:29: conflicting types for `strcasecmp'
e:/MinGW/include/string.h:166: previous declaration of `strcasecmp'

In dynload_win.c, comment-out function strcasecmp and try again:

../PC/getpathp.c:359:39: missing terminating ' character

Just drop this strange “'” and try again:

$ make -f pythoncore.mak 2>&1 | tee -a pythoncore.log
...
../PC/import_nt.c:12:22: importdl.h: No such file or directory

Looks like with import_nt we have a problem... Does something strange, accesses registry. Comment-out inclusion of importdl.h and modify function PyWin_FindRegisteredModule. Let it return NULL. In case of problems we will try to do something better.

$ make -f pythoncore.mak 2>&1 | tee -a pythoncore.log
...
../Modules/posixmodule.c: a lot of errors

Looks like windows.h is not included Yes, it is so. Inclusion of windows.h is wrapped by #ifdef _MSC_VER. After this block, add includes:

#include <direct.h>
#include <io.h>
#include <process.h>
#include <windows.h>

Next errors are:

../Modules/posixmodule.c:136: conflicting types for `mkdir'
e:/MinGW/include/io.h:145: previous declaration of `mkdir'
../Modules/posixmodule.c:149: conflicting types for `chmod'
e:/MinGW/include/sys/stat.h:175: previous declaration of `chmod'
../Modules/posixmodule.c:151: parse error before "uid_t"

Before this block, started with #ifndef HAVE_UNISTD_H, add #define HAVE_UNISTD_H.

Now:

../Modules/posixmodule.c:1159: `DIR' undeclared (first use \
  in this function)

It comes from dirent.h. Add #define HAVE_DIRENT_H before #ifdef HAVE_DIRENT_H.

../Modules/posixmodule.c:1236: too many arguments to function `mkdir'

Disliking it. Start over. Use original posixmodule.c.

Insert as first line:

#define _MSC_VER 1
$ make -f pythoncore.mak 2>&1 | tee -a pythoncore.log
...
../PC/python_nt.rc:4:17: ver.h: No such file or directory

Resources are not important. so update Makefile: was:

OBJS=$(patsubst %.rc,%.res,$(patsubst %.cxx,%.o,$(patsu\
  bst %.cpp,%.o,$(patsubst %.cc,%.o,$(patsubst %.c,%.o,$(fil\
  ter %.c %.cc %.cpp %.cxx %.rc,$(SRCS)))))))

now:

OBJS=$(patsubst %.cxx,%.o,$(patsubst %.cpp,%.o,$(patsubst \
  %.cc,%.o,$(patsubst %.c,%.o,$(filter %.c %.cc %.cpp %.cxx,$(SRCS))))))
$ make -f pythoncore.mak 2>&1 | tee -a pythoncore.log
...
../Modules/timemodule.c:169: cast to union type from type not present in union

Replace code of function time_clock by return NULL.

$ make -f pythoncore.mak 2>&1 | tee -a pythoncore.log
...
e:\MinGW\bin\dllwrap.exe: no export definition file provided.
Creating one, but that may not be what you want

Will understand it later

Fantasic! I have python22_d.dll. Time to build python.

$ make -f python.mak 2>&1 | tee -a python.log
e:\MinGW\bin\windres.exe: can't open icon file `pycon.ico': \
  No such file or directory

Updating python.mak: was:

%.res: %.rc
	$(RC) $(CPPFLAGS) -o $@ -i $<

now:

%.res: %.rc
	$(RC) $(CPPFLAGS) --include-dir ../PC -o $@ -i $<
$ make -f python.mak 2>&1 | tee -a python.log
...
f:/p/singletext/software/Python-2.2.2/build/../Modules/python.c:10: \
  undefined reference to `Py_Main'

Let try pythonw.mak:

$ make -f pythonw.mak 2>&1 | tee -a pythonw.log
...
g++  -W -fexceptions -g -O0 -I../Include -I../PC -DWIN32 -D_DEBUG\
  -D_WINDOWS  -Wl,--subsystem,windows --image-base "0x1e190000"\
  -o pythonw_d.exe ../PC/python_exe.res ../PC/WinMain.o\
  -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32\
  -lshell32 -lole32 -loleaut32 -luuid -lwsock32
g++.exe: 0x1e190000: No such file or directory

Change:

#LDFLAGS+=-Wl,--subsystem,windows --image-base "0x1d000000"
LDFLAGS+=-Wl,--subsystem,windows -Wl,--image-base,"0x1d000000"

Error is the same...

Time to investigate pythonw.dsp

No...

After reading http://sebsauvage.net/python/mingw.html, new ideas are appeared.

Run:

$ dlltool --dllname python22_d.dll --def \
  ../PC/os2vacpp/python.def --output-lib libpython22_d.a

Copied to the same dir. Deleted PROTMODE.

dlltool --dllname python22_d.dll --def python.def \
  --output-lib libpython22_d.a

I have .a!

Now it is time to try to build Python again. Add to python.mak (after conditional sections)

LIBS += libpython22_d.a

and try again

make -f python.mak 2>&1 | tee -a python.log

It is compiled!!!!

It works!!!

Now it is time to check that it works correctly.

Copy PC/testpy.py to build. Run testpy.py.

Example 12. test.bat

python_d.exe testpy.py
pause
102 tests OK.
43 tests failed:
...
48 tests skipped:
...
21 skips unexpected on win32:
...

It's ok. I don't need, for example, sgmllib.

But running real program with regexpes I get:

ImportError: No module named _sre

Going back to pythoncore.mak In pythoncore.mak, add ../Modules/_sre.c to variable SRCS.

$ make -f pythoncore.mak 2>&1 | tee pythoncore.log

To python.def, added _init_sre and others.

$ dlltool --dllname python22_d.dll --def python.def --output-lib \
  libpython22_d.a
$ make -f python.mak 2>&1 | tee -a python.log

Well, have to examine soures. Looks like find_module in import.c.

$ gdb.exe ./python_d.exe
(gdb) break main
(gdb) run
(gdb) break find_module
(gdb) cont

And really. There are no _sre.dll in sys.path. We should add _src into python.

In Modules/config.c.

What a surprise! _PyImport_Inittab already contains _sre! But also there are posix. Probably another file.

Found. Another location is PC/config.c. Added.

$ make -f pythoncore.mak 2>&1 | tee pythoncore.log

All works!

Attempt to compile a module

Problems here are not Python problems, but I recorded these problems for full collection.

Let now compile libxml binding

I will compile without iconv. Bad, but let it be so.

Found win32/Makefile.mingw. There are no config.mingw, so I should manually edit Makefile.mingw.

$ make -f Makefile.mingw 2>&1 | tee -a log

A lot of missed includes. Searching for files *h.in.

Just copied config.h.in to config.h.

Anyway problems.

Removed config.h.in. Started over.

Standard configure run:

$ sh forconf
$ make 2>&1 | tee -a log

Compiled without errors. Hm.

$ make test
make: Nothing to be done for `test'.

Ok.

But nothing in folder python. Enter folder and start building:

$ make 2>&1 | tee -a log

Hm... Nothing to do

I see setup.py

So:

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build
failed to find headers for libxml2: update includes_dir

Update setup.py:

ROOT = r'..'

We should first install libxml. Go back and:

$ make install

Istalls to /usr/local. It is bad. I forget about prefix.

Rerunning configure with prefix.

$ sh forconf
$ make 2>&1 | tee -a log
$ make install

Now try build python module again:

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build
failed to find headers for libiconv: update includes_dir

Debugging code... why it is so?

Tried

ROOT = r'f:\p\singletext\software\libxml2-2.5.10'

But now it dislikes absence of iconv. Comment-out this check.

Now “build” starts but:

NameError: name 'iconv_includes' is not defined

so empty iconv_includes is defined Not helps. Defining iconv_includes to be /usr/include

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build

Now it attempts to use Visual Studio instead of MinGW compiler. Tuning of build-tool is required.

In Lib/distutils, a set of compiler-descriptrion files is found. Best is cygwinccompiler.py which says:

...also contains the Mingw32CCompiler class which handles the mingw32 port of GCC (same as cygwin in no-cygwin mode).

How to use it?

A table compiler_class is defined in ccompiler.py. We should specify mingw32 somehow.

setup.py --help gives me nothing. I lazy to read manuals. Also, I consider it is not correct to insert compiler specification to module definition or command line.

So hacking ccompiler.py. So method get_default_compiler always return mingw32.

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build
...
warning: Python's pyconfig.h doesn't seem to support your compiler.\
Reason: couldn't read 'f:\p\singletext\software\Python-2.2.2\include\
\pyconfig.h': No such file or directory.Compiling may fail because of\
undefined preprocessor macros.

Yes, pyconfig.h is not in include, but level higher. But is it correct pyconfig? No! We should use one in the PC folder!

Easiest way is to copy file, but I dislike such solution. So let first try to fix cygwinccompiler.py.

Looks no so simple. So making a copy.

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build

A lot of errors. Starts with:

In file included from libxml2-py.c:7:
libxml_wrap.h:78: parse error before "xmlCatalogPtr"

Putting

e:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -If:\p\singletext\
  \software\libxml2-2.5.10\include/libxml2 -I/usr/include -If:\p\si\
  ngletext\software\Python-2.2.2\include -c libxml2-py.c -o build\te\
  mp.win32-2.2\Release\libxml2-py.o"

to shell-file compile.sh. Quoting “\” by additional “\” and running:

$ sh compile.sh 2>&1 | head
libxml_wrap.h:78: parse error before "xmlCatalogPtr"

I understand... I compiled libxml without catalog support. Looking at other messages, I think that the easiest solution is to return back to libxml and compile it with all options, including catalog, http, ftp (but try without iconv...).

Returning to libxml

$ make distclean
$ ... edit forconf...
$ sh forconf
$ make
$ make istall

Again, go to python folder

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build

We already seen it:

failed to find headers for libiconv: update includes_dir

Fixing and again:

$ f:/p/singletext/software/Python-2.2.2/build/python_d.exe setup.py build

libxml still not comlete, looks like html and relaxng should be added. But I don't want to do it. So I just modify libxml2-py.c: add return NULL for bad functions.

Maybe it is not good, because first comment says “Generated”. Now error is:

e:\MinGW\bin\..\lib\gcc-lib\mingw32\3.2.3\..\..\..\..\mingw32\bin\
  \ld.exe: cannot find -llibxml2

Bad prefix lib. In setup.py, change

if sys.platform.startswith('win'):

to

if sys.platform.startswith('xwin'):

Now:

cannot find -lz

Rewrite condition as:

libraryPrefix = ''
platformLibs = []

Now:

cannot find -lpython22

Ok, it is time to return to 'distutils'.

At this time I decided that all this is enough. Let use python compiled by MS VC. Let users get 2 Mb of software, but users will get software. If we will compile Python, we will never release it.

Our nice tracelog is finished.


http://uucode.com/texts/python-mingw/python-mingw.html
Oleg A. Paraschenko <olpa uucode com>