Re: Python binaries for mozilla

This is a multi-part message in MIME format.
Mark Hammond wrote:
> I've been thinking about and playing with ways to create a Python XPI for
> Mozilla.

Okay, you motivated me to start writing up what Komodo does for its 
embedded/siloed Python. I'll include my notes and then respond to some 
of your points below that:

[Trent's Komodo-Python notes]
> Embedding Notes
> ---------------
>
> - [Windows] pythonXY.dll, pythoncomXY.dll and pywintypesXY.dll are placed in
>   ``$mozBin``.
>
> - [Linux/Unix] ``libpythonX.Y.so.1.0`` is placed in ``$mozBin`` and
>   ``libpythonX.Y.so`` is a symlink to the versions shared obj in that dir.
>
> - [Mac OS X] The Python build is places in
>   ``Contents/Frameworks/Python.framework`` under the Mozilla-based app dir
>   (e.g. ``/Applications/Komodo.app``). XXX Describe other tweaks made in there
>   (symlinks removed and added).
>
> - [All Plats] A patch to properly namespace the expat symbols used by Python's
>   PyExpat to avoid conflict with an expat loaded by other parts of Mozilla is
>   used in the Python build. If I get off my a$@ this will be in Python 2.5
>   (I have patches mostly ready). See these Python tracker items:
>
>   - `[ 1462338 ] upgrade pyexpat to expat 2.0.0
>     
> <http://sourceforge.net/tracker/index.php?func=detail&aid=1462338&group_id=5470&atid=305470>`_
>   - `[ 1295808 ] expat symbols should be namespaced in pyexpat
>     
> <http://sourceforge.net/tracker/index.php?func=detail&aid=1295808&group_id=5470&atid=105470>`_
>
> - [Windows] A patch to stop Python looking in the Windows registry for
>   ``sys.path`` entries is applied to the build.
>
> - [Linux/Unix] Relocation of install-dir in built bits is necessary. For
>   example, this changes the explicit reference in ``libpythonX.Y.so`` to the
>   build-time specified install dir (e.g. ``/usr/local``) to the actual install
>   dir. This relocation must be done when post-install when the install
>   directory is known. See ``Komodo-devel/src/install/_install.py.template``
>   for details. There are some gotchas in this process.
>
> - [Mac OS X] Relocation of Python library reference in main Python binary.
>   First, this is only necessary to include a ``python`` binary to run an
>   independent Python process -- i.e. not strictly necessary for linking to
>   the Python library for PyXPCOM.
>
>   On Mac OS X this relocation is simpler than on Linux/Unix because (1)
>   ``otool`` and ``install_name_tool`` can be used to do the work and (2)
>   relative directory references are possible so it can be done
>   pre-install-time. Basically you need to (1) get the current Python library
>   path reference in the main Python binary via::
>   
>         otool -L path/to/bin/python
>
>   and then make it a relative reference via::
>   
>         install_name_tool -change $oldRefPath @executable_path/../Python \
>             path/to/bin/python
>
>   We currently do this as part of the embedded ActivePython build in
>   ``ActivePython-devel/build.py#target_image_embedding``.
>
> - [Linux/Unix] the ``bin/python`` shell script thang.
>
> - [Windows and Linux/Unix] Setting PYTHONHOME on startup. The PYTHONHOME
>   environment variable is set to the appropriate directory on Komodo startup.
>   This is not necessary on Mac OS X: the Frameworks system handles this
>   for us.
>
> - [All Plats] Unset PYTHONPATH on setup. For Komodo, an embedded Python is
>   an implementation detail, therefore a user's PYTHONPATH should not influence
>   Komodo behaviour. It is unset on Komodo startup on all platforms.
>
> - [All Plats] Trimming the install. A lot of parts of a normal ActivePython
>   build are trimmed out. Docs. Tcl/Tk. Tools. Scripts. Parts of the stdlib.
>   Test suite bits. "Resources" bits from the Mac OS X install image.


[Back to Mark's notes]
> The XPI installs:
>
> * python24.dll into the moz 'bin' directory.  

Yup. Similarly for Linux/Unix. Not necessary for Mac OS X. On Windows we 
also place the PyWin32 sysdir DLLs there.

> On Windows, we patch the
> resource string that is used for 'winver' - hence the normal Python24
> registry keys will be totally ignored.

I have a patch to the Python source that I apply pre-build to instead 
disable Windows registry lookup for sys.path. I think that is better 
solution for the more generic "building Python for embedding" problem. 
I've attached the patch (which includes some notes at the top).

Thoughts? Yours is obviously a little simpler because it can be done 
post-Python-build.

> * python24.zip into the 'bin' directory.  This is just the standard Python
> library compiled as .pyo files.  The source files to the stdlib are not
> shipped.  A 'manifest' in the build script defines exactly what is included
> (eg, test suite and all 'plat-*' dirs are removed).  This is created with no
> compression (so should be *faster* than including the lib on disk)

Currently Komodo is just including the Python stdlib '.py' files over in 
the equivalent of the 'bin/python24' directory that you mention in your 
next point. Zipping up the stdlib sounds like a good idea. See my 
comments on your next point.

> * A 'bin/python24' directory is created.  This has a single subdir 'DLLs'
> with the standard Python .pyd files.  The intent is that this directory be
> laid out like a standard Python install (but minus the 'lib' dir if the zip
> file is included - which it currently is).  The intent is that code like
> distutils, which sniffs out various important Python directories should
> still work if executed in this environment.

Komodo's embedded Python isn't as tightly integrated into the $mozBin 
directory as you are describing here. Generally the layout is like this:

        $komodoInstallDir/
                lib/
                        mozilla/...   # the mozilla build is here
                        python/...    # **

** All the Python bits are here except the main DLLs in $mozBin. This 
dir basically looks like a std, but trimmed down, Python install.

On Mac OS X the "Frameworks" system allows for the pretty much typical 
Python install image to be placed in:

        $komodoInstallDir/Contents/Frameworks/Python.framework/...

As you mention, one of Komodo's goals for the embedded Python is that it 
can be run as a normal Python install (for building extensions and for 
development-debugging).

Does your moving the stdlib to 'python24.zip' in the $mozBin directory 
break that? I.e. will distutils work the way you've described it.

Also, will just including the .pyo files in the stdlib zip cause 
problems/confusion trying to run distutils there (people typically won't 
run with "python -O"/PYTHONOPTIMIZED).


> * A 'bin/python' directory - this is the Mozilla related Python library as
> currently created by the Mozilla build process (assuming Python extensions
> are included ) - ie, the 'xpcom' and 'nsdom' packages.  Although much of
> this could also go into a .zip file, I figured that initially we just ship
> with these .py files on disk to help people who want to read/debug the
> source (which they really have no need to do with the stdlib itself)

Right. This is what Komodo is currently doing (no 'nsdom' in Komodo yet, 
of course):

        $mozBin/
                python/
                        xpcom/
                

Note that we also place Python modules and packages used by 
Komodo-specific PyXPCOM components under there:

        $mozBin/python/
                  komodo.pth   # add 'komodo' dir to sys.path
                  komodo/...   # our Python libs here

A general best-practices along these lines (or something more 
appropriate) for xulrunner-based apps might be in order at some point.


> * 3 components are shipped - the pyxpcom loader, the nsdom component, and
> pyabout.py.  This means that once installed, about the only thing you can do
> is type "about:python" into the browser bar.  No examples or samples are
> shipped - they can come in their own .xpi file.

Sounds good.


> Issues:
>
> * On non-windows platforms, will there be a need to "patch" our .so files to
> remove the reference to /usr/local, or whatever was used when pyxpcom was
> actually build?  I seem to recall Shane mentioning something about that.
> Any other non-windows issues I remain oblivious to?

This is mentioned in my notes at the top.

Windows: nothing required.

Mac OS X: easy steps that can be done before packaging the XPI.

Linux/Unix: PITA steps that have to be done post-install. Can this stuff 
be done with the XPI system? Some gotchas:

1. Relocating from Python's build PREFIX to whereever the Mozilla-based 
app is installed requires that the PREFIX has more characters than the 
actual install dir. Hence you need a Python build with a intentionally 
long and bogus PREFIX.

(My suspicion is that there may be some complicated source/built work 
that could be done to Python to get around this requirement. I don't 
know that though.)

2. I still haven't really dealt with the libstdc++ dependencies in my 
embedded Python builds that has issues with Linux compatibility. I'm not 
sure if other Python builds have these issues. My understanding is that 
Firefox builds are dealing with this by statically building in 
libstdc++. The embedded Python build should do the same.


Cheers,
Trent

--
Trent Mick
trentm@xxxxxxxxxxxxxxx
diff

Disable the use of the main Python DLL embedded version string in (1) finding
module names (via .../PythonCore/$winver/PythonPath/Modules/...) and (2)
determining the sys.path. Generally, the latter means that apps embedding this
Python should set PYTHONHOME in their environment before Python initializes
itself.

If this patch is to be proposed for the Python core then (a) it should be made
a configurable #define and (b) consider changing the meaning of sys.winver to
be None if the winver is NOT used for registry operations. Would have to update
Modules/sysmodule.c and the docs for that.

--- python/Python/import.c.original     2006-02-14 10:19:57.000000000 -0800
+++ python/Python/import.c      2006-02-14 10:20:08.000000000 -0800
@@ -1049,10 +1049,6 @@
    corresponding filedescr struct, and (via return arguments) the
    pathname and an open file.  Return NULL if the module is not found. */
 
-#ifdef MS_COREDLL
-extern FILE *PyWin_FindRegisteredModule(const char *, struct filedescr **,
-                                       char *, int);
-#endif
 
 static int case_ok(char *, int, int, char *);
 static int find_init_module(char *); /* Forward */
@@ -1155,13 +1151,6 @@
                        return &fd_frozen;
                }
 
-#ifdef MS_COREDLL
-               fp = PyWin_FindRegisteredModule(name, &fdp, buf, buflen);
-               if (fp != NULL) {
-                       *p_fp = fp;
-                       return fdp;
-               }
-#endif
                path = PySys_GetObject("path");
        }
        if (path == NULL || !PyList_Check(path)) {
--- python/PC/getpathp.c.original       2006-02-14 10:21:02.000000000 -0800
+++ python/PC/getpathp.c        2006-02-14 10:21:07.000000000 -0800
@@ -489,8 +489,6 @@
        }
  
        skiphome = pythonhome==NULL ? 0 : 1;
-       machinepath = getpythonregpath(HKEY_LOCAL_MACHINE, skiphome);
-       userpath = getpythonregpath(HKEY_CURRENT_USER, skiphome);
        /* We only use the default relative PYTHONPATH if we havent
           anything better to use! */
        skipdefault = envpath!=NULL || pythonhome!=NULL || \