Replace ctypes.DllGetClassObject and remove DllCanUnloadNow

@junkmd

As far as I can tell, these functions are hooks: third-party code is meant to replace them.

Their implementation in ctypes (i.e. their default behaviour) is to import and call the same-named functions from a third-party library, comtypes.server.inprocserver. This is not good. comtypes should instead register their hook on import.

Here's a possible plan to make the API boundary better without breaking users.

`DllCanUnloadNow`

While the Python interpreter is running, it is not safe to unload the shared library that contains _ctypes. Therefore:

The C function DllCanUnloadNow exported from _ctypes should be changed to always return S_FALSE. We should change that now, without a deprecation period. (Note that the comtypes hook already does this.)
We should stop importing and calling comtypes.server.inprocserver. I'm not sure about the necessary deprecation period, but I think that it should be a non-breaking change and can also be done immediately. Or is someone relying on it for side effects? O_o
Setting and getting the hook should be deprecated. In about Python 3.18 we should stop calling it, and remove it.

`DllGetClassObject`

This one, on the other hand, sounds like a useful hook. It also looks like an inprocess COM server need a special build so it's not useful to allow multiple hooks -- replacing a global one is enough. Is that so?
If yes:

ctypes.DllGetClassObject (the default implementation) should raise a DeprecationWarning. In about Python 3.18, it should be changed to do nothing, just, return CLASS_E_CLASSNOTAVAILABLE.
comtypes should be changed: on import, it should replace ctypes.DllGetClassObject with its own hook.

This should ensure that old versions of comtypes still work as before (until after the deprecation period).

Does that sound reasonable?
cc @junkmd

Linked PRs

gh-127369: Deprecate ctypes.DllGetClassObject and ctypes.DllCanUnloadNow #127766

@junkmd

I suspect these may be here because comtypes calls back through ctypes, and so to COM it looks like all the calls are coming from _ctypes.pyd? I've never noticed them in our code before (all my COM work with Python has been in my own extension modules).

No doubt it's a useful and important hack at some point, but without a more concrete use case they're probably not so necessary.

Changing default behaviour to not import comtypes is a good move, but it may not be possible for comtypes to hook them itself if it doesn't get imported before these are called. DllGetClassObject in particular is meant as an entry point, but I'm not sure how much gets executed before it may be called. It probably depends on how Python is registered as a local server, which is something that's definitely outside of our upstream support, and so I think it's reasonable for comtypes (or whoever) to figure out a preferred way to set that up.

If comtypes doesn't need these, then I'd prefer to deprecate and remove. I guess that's what @junkmd can tell us.

For those interested in this discussion, I would like to introduce some technical references related to DllCanUnloadNow and DllGetClassObject.

A notable reference book on OLE and related COM topics is "Inside OLE, 2nd Edition, Kraig Brockschmidt, Microsoft Press, 1995, ISBN: 1-55615-843-2".
The author, kraigb, has made the book publicly available under the MIT License.

Regarding DllGetClassObject, page 183 of the PDF states the following:

In-Process Server

Every DLL server must implement and export a function named DllGetClassObject with the following form:
STDAPI DllGetClassObject(REFCLSID rclsid, REFIID riid, void **ppv);
When a client asks COM to create an object and COM finds that an in-process server is available, COM will pull the DLL into memory with the COM API function CoLoadLibrary. COM then calls GetProcAddress looking for DllGetClassObject; if successful, it calls DllGetClassObject, passing the same CLSID and IID that the client passed to COM. This function creates a class factory for the CLSID and returns the appropriate interface pointer for the requested IID, usually IClassFactory or IClassFactory2, although the design of this function allows new interfaces to be used in the future. No straitjackets here.

DllGetClassObject is structurally similar to IClassFactory::CreateInstance, and as we'll see later in the sample code for this chapter, the two functions are almost identical: the difference is that CreateInstance creates the component's root object, whereas DllGetClassObject creates the class factory. Both query whatever object they create for the appropriate interface pointer to return, which conveniently calls AddRef as well.

Because DllGetClassObject is passed a CLSID, a single DLL server can provide different class factories for any number of different classes that is, a single module can be the server for any number of component types. The OLE DLL is itself an example of such a server; it provides most of the internally used object classes of OLE from one DLL.

Be Sure to Export DllGetClassObject

When creating an in-process server or handle, be sure to export DllGetClassObject as well as DllCanUnloadNow. (See "Unloading Mechanisms" later in this chapter.) Failure to do so will cause a myriad of really strange and utterly confusing bugs. I guarantee that you'll hate tracking these bugs down. Save yourself the trouble and write yourself a really big, hot-pink fluorescent Post-it note and stick it in the middle of your monitor so you'll remember.

Regarding DllCanUnloadNow, page 186 of the PDF states the following:

In-Process Server

Being rather passive, the DLL unloading mechanism is fairly trivial. Every now and then primarily when a client calls the function CoFreeUnusedLibraries COM attempts to call an exported function named DllCanUnloadNow in the same manner that it calls DllGetClassObject. This function takes no arguments and returns an HRESULT:
STDAPI DllCanUnloadNow(void)
When COM calls this function, it is essentially asking "Can I unload you now?" DllCanUnloadNow returns S_FALSE if any objects or any locks exist, in which case COM doesn't do anything more. If there are no locks or objects, the function returns NOERROR (or S_OK), and COM follows by calling CoFreeLibrary to reverse the CoLoadLibrary function call that COM used to load the DLL in the first place.

Note: Early 16-bit versions of OLE did not implement the CoFreeUnusedLibraries, so DllCanUnloadNow was never called. This led to the belief that it was not necessary to implement this function or IClassFactory::LockServer because the DLL would stay in memory for the duration of the client process. This is no longer true because the COM function is fully functional and will call all DLLs in response. To support proper unloading, you must always implement IClassFactory::LockServer as well as DllCanUnloadNow.

A DLL's response to a DllCanUnloadNow call does not take into consideration the existence or reference counts on any class factory object. Such references are not sufficient to keep a server in memory, which is why LockServer exists. To be perfectly honest, an in-process server could include a count of class factories in its global object count if it wanted to, but this doesn't work with a local server, as the following section illustrates.

Below is documentation for using comtypes (as of version 0.5) ¹⁾ as a COM server:
https://pythonhosted.org/comtypes/server.html

¹⁾ The comtypes documentation has not been updated for many years. This is due to a combination of factors: pythonhosted can no longer be updated; I am not yet well-versed in Sphinx; I am unfamiliar with how to migrate documentation to hosting platforms like ReadTheDocs; and I lack case studies on how external collaborators without ownership, like myself, can work with other maintainers to maintain documentation. One reason I started contributing to the cpython documentation was to familiarize myself with modern Sphinx practices.

@encukou

Here are my current thoughts on this:

Change the implementation of these ctypes APIs so they no longer depend on comtypes.
Keep these APIs but modify them to make the hooks in ctypes easier for third-party libraries and applications to use.
In any case, implement a COM server using ctypes to verify its behavior during the process.

I am not against changing ctypes to remove its dependency on comtypes. Considering the proper dependency relationship, the current implementation is suboptimal.

However, just as COM interfaces can be implemented by projects other than comtypes, changes to ctypes should leave room for implementing COM servers without relying on comtypes.

I think the hooks defined in ctypes and the callbacks defined in _ctypes (CanUnloadNow and DllGetClassObject) should remain.
I suspect these are essential when implementing a COM server using ctypes.

Regarding hook handling, I propose creating a relationship similar to sys.excepthook and sys.__excepthook__.
An implementation like below would allow application developers to specify the behavior invoked by hooks easily when combining ctypes and COM packages such as comtypes.
It would also enable calling the pre-overloaded function.

def DllCanUnloadNow():
    return __dll_can_unload_now__()


def __dll_can_unload_now__():
    ...


def DllGetClassObject(rclsid, riid, ppv):
    return __dll_get_class_object__(rclsid, riid, ppv)


def __dll_get_class_object__(rclsid, riid, ppv):
    ...

The main challenge is that we have yet to find modern use cases for implementing COM servers using ctypes or comtypes.

To make production changes, I believe comprehensive testing that covers actual use cases is essential.

The comtypes community likely includes developers who have implemented COM servers using comtypes or even developers familiar with implementing COM servers in languages other than Python.

If agreeable, I plan to reach out to them and invite them to participate in this discussion to provide their insights and cooperation.
I am also reaching out to developers interested in this topic in enthought/comtypes#671, but directly contacting developers who could be stakeholders in this change would be more effective.

cc: @encukou, @zooba

The main challenge is that we have yet to find modern use cases for implementing COM servers using ctypes or comtypes.

Agreed. There are no shortage of use cases for using COM servers, but very few that require implementing them (especially when they're then going to be activated through COM's global interfaces, as opposed to being directly passed in).

I assume this only applies to in-process activation as well? If you are registering a COM server for out-of-proc activation then I'm pretty sure ctypes isn't going to work anyway. So it's really only able to handle in-process activation, which is also not widely used.

If anyone has any such cases for implementing/registering a COM server in Python, modern or legacy, it would be helpful to list them here.

If we can't find sufficient modern cases to motivate this (and I emphasise modern here because we need to justify users who can update their CPython to a later version but somehow can't modify their own code or update other parts of their system - legacy cases where nobody has touched it in 20 years don't count, because you couldn't update CPython in that case), then I think deprecation and complete removal is on the table.

I suspect these may be here because comtypes calls back through ctypes, and so to COM it looks like all the calls are coming from _ctypes.pyd? I've never noticed them in our code before (all my COM work with Python has been in my own extension modules).

But, it's not safe to unload _ctypes.pyd (or any other extension's DLL) while a Python interpreter is running. So, we should make the DllCanUnloadNow hook a no-op.

Regarding hook handling, I propose creating a relationship similar to sys.excepthook and sys.__excepthook__.

IMO, that would make sense if __dll_get_class_object__ was complicated. But here the base implementation would be two lines:

def __dll_get_class_object__(rclsid, riid, ppv):
    return -2147221231  # CLASS_E_CLASSNOTAVAILABLE

(There's a magic number because ctypes doesn't have a good way to work with HRESULT error constant. But a library dealing with COM should make them easy to work with.)
It could even be no lines -- the C function can do this in case the Python hook does not exist at all.

If we need something more complex than a replaceable hook, i was thinking about a list, named for example ctypes.DllGetClassObject_hooks, to which users could add their hooks. The C function DllGetClassObject would call Python functions in this list in order, until one returns something else than CLASS_E_CLASSNOTAVAILABLE. It would return the result of the last call.

(We could instead make the list internal, and add a “register” function, but then we'd also need “unregister” and “clear” and so on -- a list sounds more convenient.)

Unlike with sys.excepthook, it makes sense to me that there could be multiple extensions that want to export COM objects. But, I have no idea if it's even something to keep in mind as a future possibility.

@junkmd

I am one of the developers using comtypes to make servers rather than clients that @junkmd asked to weigh in on this discussion. I was going to say that I know absolutely nothing about this subject because I have only used local servers rather than inproc ones, but that felt a bit underwhelming, so I have done a bit of extra digging to try and understand at least a little bit that I might be able to contribute, apologies if none of this is new or interesting.

Local servers made intuitive sense to me because registering them just sets the Python executable and a script to run when the server is created, but it didn't make sense to me how an inproc server could work - how would the interpreter get set up, etc? So I looked at the comtypes registration code for inproc servers here which requires a frozendllhandle attribute to exist to register as inproc.

From somewhere in my distant memories that reminded me that comtypes had been associated with py2exe, so I checked that out and sure enough turned up this page on how to generate a COM DLL server. Some further digging through the code turned up the file source/run_ctypes_dll.c and particularly these lines. Basically what Py2exe is doing is compiling this C file into a DLL and embedding a Python interpreter into it to run the comtypes server code. As the DLL being used to provide the inproc server it has to implement DllGetClassObject() and DllCanUnloadNow, but it does so by loading the implementations out of ctypes (which of course then goes on to get them from comtypes).

I know that @theller was heavily involved in the early days of ctypes, comtypes, and py2exe and could probably add more insight into this situation, but I assume that the issue is that because comtypes is a pure Python package it doesn't have a DLL which could be imported by run_ctypes_dll.c, so it was convenient to make use of the _ctypes.dll to provide exported C functions which call the Python functions in turn.

From my perspective, there doesn't seem to be any reason to keep these functions in ctypes at all. Although they are nominally standard COM functions as explained above by @junkmd, there is no process to invoke them except via Py2exe and run_ctypes_dll.c. It would therefore seem much more logical to move the logic contained in the callbacks directly to Py2exe, but rather than invoking ctypes.DllGetClassObject it could directly invoke comtypes.DllGetClassObject (comtypes is guaranteed to be installed via Py2exe's freezing process anyway). Arguably, the inproc server code in comtypes should even be moved into Py2exe anyway given that it can only be used via Py2exe in the first place.

I will confess that I personally have little appetite for taking on the responsibility of updating Py2exe to no longer depend on these functions from types, I have never used Py2exe and it is sufficiently complex (and insufficiently documented) that I don't fancy setting up the necessary development environment to get it building. But it is actively being developed and I think that the actual fix is probably fairly simple for a maintainer to implement (mainly just copying code from ctypes/callbacks.c). I think that with a fairly long deprecation cycle before removing them from ctypes, combined with a bug report to Py2exe, anybody that is interested in using inproc servers in this way will have plenty of opportunity to implement the fix.

It seems that py2exe has special support for comtypes.server.inprocserver that it runs in a special ctypes_comdll target. It's not clear to me what an “inprocess COM server” is; is this it?

If it is, looks like it will call comtypes.server.register.register before Windows will make any calls to DllGetClassObject, so that's a function where comtypes can ensure that its hook is installed.

@encukou

@encukou an inproc COM server is one which is contained in a DLL and so can be instantiated in the calling process, and COM objects so produced can be used directly via function pointers. In the more common case (at least in my experience) the COM server is a separate process and methods are called via RPC with Windows messages.

There are different hooks for different purposes in the DLL. DllRegisterServer() and DllUnregisterServer() are called by regsvr32 in order to create the relevant registry entries to allow the server to then be discoverable by potential clients (or to remove the entries, respectively). These functions would not normally be called when a registered server is being instantiated - that is when DllGetClassObject() is called.

I think the most important point to emphasise is that the DllGetClassObject hook which Windows calls is in the DLL which is registered for the inproc server. This is not the _ctypes.pyd but rather the custom DLL created by Py2exe. The DllGetClassObject in ctypes is therefore not actually a hook. AFAICT there is no reason for any of this code to be in ctypes at all - all of these functions in ctypes will only ever be called from Py2exe DLLs, there is no other mechanism to use them, so they really belong in Py2exe.

You can see here that the Py2exe DLL implementations of DllRegisterServer() and DllUnregisterServer() call into the Python interpreter that is provisioned in DllMain(), which has already imported boot_ctypes_com_server.py and so can directly call the Python versions of the functions here. For reasons that are not clear to me, the Py2exe DLL does not do the same thing for DllGetClassObject(), but instead imports the C functions from _ctypes.pyd that then do use the same paradigm of calling in to the Python code in ctypes. I think it should be very easy to move the Python DllGetClassObject() and DllCanUnloadNow() code from ctypes into boot_ctypes_com_server.py and then just change the way that the Py2exe DLL forwards the calls to match Dll(Un)RegisterServer()

In fact, I think there is definitely a case to be made for moving all of comtypes.server.inprocserver into Py2exe. The implementation in comtypes is custom designed for Py2exe to use (and cannot be used in any other way), and there is plenty of messy stuff like this where Py2exe is overwriting a private comtypes member because really all that logic belongs in Py2exe.

Based on previous discussions, I think the ideal approach is to implement the hooks directly in freezing tools like py2exe, with ctypes and comtypes providing supporting utility functions as needed.

Considering cases where implementing COM servers with win32com in pywin32, it is neither reasonable nor feasible for ctypes or comtypes to take responsibility for properly handling inproc COM server hooks for every possible COM implementation.

Moreover, as seen in py2exe/py2exe#24 (and mhammond/pywin32#868), it seems that registering COM servers with py2exe does not work well for either win32com or comtypes.

It might also be worthwhile to invite the maintainers of py2exe to participate in this discussion to help investigate the technical challenges and determine the required deprecation timeline (i.e., the duration of the deprecation period to be set).

I opened py2exe/py2exe#217.

@bennyrowland

@bennyrowland

Thank you for your investigation.

Sorry for causing confusion with my previous claim that InprocServers would only work with Py2Exe.

Don't worry.
We are discussing features that are among the most mature functionalities in Windows, some of the oldest APIs in ctypes, long-untested features in comtypes, and ones that we are no longer even sure work with py2exe.
Given this state, it's understandable that anyone could become confused or misinterpret things.
I appreciate your participation in this discussion.

Once implemented, this would probably be the best solution, but it would add complexity to the comtypes build, as the package is currently pure Python, so having to compile even a very simple DLL would be a big change.

As you mentioned, including a DLL in the build of comtypes would be a significant change.
One of the key features and differentiators of comtypes is that it is Pure Python, so I would like to maintain that concept.

As an alternative, based on my earlier suggestion and encukou's advice, I currently think it would be better to implement a system where each project can optionally set hooks.

if _os.name == "nt": # COM stuff
    __dll_get_class_object_hooks__ = []  # or set()?

    def DllGetClassObject(rclsid, riid, ppv):
        if not __dll_get_class_object_hooks__:
            return -2147221231  # CLASS_E_CLASSNOTAVAILABLE
        results = {f(rclsid, riid, ppv) in f for __dll_get_class_object_hooks__}
        if 0 in results:
            return 0  # S_OK
        if len(results) == 1:
            return tuple(results)[0]
        return -2147467259  # E_FAIL

For example, a package like comtypes would register its hooks in the following manner.

import ctypes

def dll_get_class_object_hook(rclsid: int, riid: int, ppv: int) -> int:
    """Returns S_OK, E_FAIL, or CLASS_E_CLASSNOTAVAILABLE.
    See also https://learn.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-dllgetclassobject
    """
    ...

if sys.version_info >= (3, x):
    ctypes.__dll_get_class_object_hooks__.append(dll_get_class_object_hook)

Of course, this implementation is merely an example, and I have no objections if a better implementation or operation is found that removes unnecessary dependencies from the standard library and preserving the concepts of third-party projects.

Also, I do NOT strongly desire to change the implementation of ctypes as described above right away.
I would like to wait for developers actively using this feature to notice the DeprecationWarning (Is FutureWarning more appropriate?), find this issue, and share their actual use cases by participating in this discussion.

I appreciate everyone participating in this discussion.

I'd be fine with ctypes providing a “trampoline” C functions that call user-defined callbacks, so that provided that:

it's possible to create a pure-Python COM server that works with current ctypes
there's a finite number of the necessary functions, so we don't get requests for new ones in every release

But what's not clear to me is if, with current ctypes, comtypes (or any other library) needs to build its own DLL. Are DllRegisterServer and DllUnregisterServer also needed (as in AvmcIfc.cpp), or are they optional?

If building a DLL is necessary, I think it's better to remove this functionality from ctypes.
I'd be fine with delaying the deprecation, though.

If building a DLL is not necessary for the third-party library to provide a useful COM server, let's

add optional functions like DllRegisterServer to ctypes as well, if there's a small well-defined set of them. (Are there others? Is this it?)
add a registration mechanism
provide a supported API to call the DLL functions
add tests to CPython (and I'd ask you to write these, like with the recent COM-related tests)

If APIs are needed to implement, the COM specification has remained unchanged for decades since its inception, so once the *Dll* APIs are defined, there should be no additions beyond that.

I plan to spend my free time this weekend writing tests to implement, register, and unregister a COM server using ctypes.

@encukou

@encukou DllRegisterServer and DllUnregisterServer are standard Windows entry points when using the regsvr32.exe application to register a DLL in the registry, they are not used during the creation and use of the COM objects. comtypes provides its own mechanism for doing registrations for Python COM servers which I think is much more practical and convenient than passing _ctypes.pyd to regsvr32.exe, and it would also be impossible in that case to specify which server(s) you wished to register, there would have to be some kind of blanket registration of all discoverable servers, which would be complicated to define and set-up.

The code in comtypes/source (Avmclfc.cpp etc.) is a C++ COM server that is (or was) being used as a test object for comtypes COM client code to be tested against. I believe that DllGetClassObject and DllCanUnloadNow are the only two functions that are required for a COM server to be created and used (I think they are the only ones implemented in current _ctypes.pyd and Py2Exe's DLLs, and things do work like that).

@junkmd the current comtypes tests in test_comserver.py are exactly already registering, using, and unregistering the COM server implemented in TestComServer.py. They test both InprocServer and LocalServer options, and your tests posted above show that they work just fine with current ctypes.

I do not see how the hook mechanism can work though, because the current mechanism for creating an InprocServer looks like:

The COM client creating the object calls ole32.CoCreateInstance() or similar to invoke the COM machinery
The COM machinery looks up the clsid in the registry and finds out that it is an InprocServer and gets the path to the DLL (which in this case is _ctypes.pyd)
It loads the DLL and calls DllGetClassObject()
_ctypes.pyd calls Py_Initialize()
_ctypes.pyd imports the ctypes Python version of the function (func = _PyImport_GetModuleAttrString("ctypes", "DllGetClassObject");) and calls it.
The Python ctypes DllGetClassObject() imports the comtypes version and runs that.

So there is no point at which comtypes or any other module gets imported where it can insert itself into that train of events. It only works because importing comtypes is hard-coded into ctypes. I suppose that we could do something with entry points, but that would involve a lot of design work to create something that would almost certainly never be used except by comtypes (it is very hard to see anyone creating a competing library, given the enormous complexity of working with COM, and how well, by and large, comtypes already works).

@junkmd if you want to keep comtypes as a pure Python package (which I do understand) then perhaps a solution would be to create a separate package (e.g. comtypes-inproc-support which would just contain the DLL (and install it somewhere in the comtypes namespace in site-packages) and which could be listed as an "extra" dependency of comtypes so people could do pip install comtypes[inproc] (or similar) to get the extra DLL. Then when registering an InprocServer comtypes could check for the availability of the DLL and if it is not available then give an error message explaining the extra package is needed to register InprocServers. This package could surely use the limited API and wouldn't have any Python code of its own so should require very little maintenance. I would be happy to help out with creating such a package.

As I mentioned earlier, I attempted to write tests this weekend to implement, register, and unregister an in-process COM server using only ctypes during my free time.

First, I investigated the winreg function calls used by comtypes to register an in-process server. For details about what gets registered in the registry, please refer to the test I added to comtypes.

In my PoC repository, I defined a simple TestCtypesComServer.idl file containing only a single custom interface. I then built it into a .tlb file using the midl command and wrote a test to register and instantiate it.

results of executing the unit tests

EF
======================================================================
ERROR: test_1 (__main__.TestInprocServer.test_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\comtypes-python-dev-compatibility\comtypes-python-dev-compatibility\src\test.py", line 192, in test_1
    create_instance(CLSCTX_INPROC_SERVER)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\comtypes-python-dev-compatibility\comtypes-python-dev-compatibility\src\test.py", line 134, in create_instance
    ole32.CoCreateInstance(
    ~~~~~~~~~~~~~~~~~~~~~~^
        byref(CLSID_TestCtypesComServer),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        byref(psvr),
        ^^^^^^^^^^^^
    )
    ^
  File "_ctypes/callproc.c", line 1041, in GetResult
OSError: [WinError -2147221231] ClassFactory cannot supply requested class

======================================================================
FAIL: test_2 (__main__.TestInprocServer.test_2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\comtypes-python-dev-compatibility\comtypes-python-dev-compatibility\src\test.py", line 211, in test_2
    create_instance(CLSCTX_INPROC_SERVER)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\comtypes-python-dev-compatibility\comtypes-python-dev-compatibility\src\test.py", line 134, in create_instance
    ole32.CoCreateInstance(
    ~~~~~~~~~~~~~~~~~~~~~~^
        byref(CLSID_TestCtypesComServer),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        byref(psvr),
        ^^^^^^^^^^^^
    )
    ^
  File "_ctypes/callproc.c", line 1041, in GetResult
OSError: [WinError -2147418113] Catastrophic failure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\comtypes-python-dev-compatibility\comtypes-python-dev-compatibility\src\test.py", line 213, in test_2
    self.fail(f"FAIL:\n{e}\n{pprint.pformat(call_args_list)}")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: FAIL:
[WinError -2147418113] Catastrophic failure
[('{C0A45AA7-4423-4263-9492-4BD6E446823F}',
  '{00000001-0000-0000-C000-000000000046}',
  926066[58](https://github.com/junkmd/comtypes-python-dev-compatibility/actions/runs/12339073919/job/34434835640#step:7:59)8144)]

----------------------------------------------------------------------
Ran 2 tests in 0.017s

When comtypes is not installed, DllGetClassObject always returns CLASS_E_CLASSNOTAVAILABLE (-2147221231, 0x80040111), causing an OSError with the same error code to be raised (test_1).

Here, if I patch DllGetClassObject to always return S_OK, even when rclsid matches CLSID_TestCtypesComServer, E_UNEXPECTED (-2147418113, 0x8000FFFF) is raised. At this point, the riid that is passed in matches IID_IClassFactory (test_2).

I believe it is necessary to perform actions similar to what comtypes does, such as obtaining the Vtbl structure mapped to the iid for a COMObject and then calling IUnknown_QueryInterface as shown here.

If you have any ideas for an easier way to implement an in-process COM server, please feel free to let me know.
I will continue investigating the minimal requirements for registering _ctypes.__file__ as a DLL, but please give me some more time.

@bennyrowland

@bennyrowland

Thank you for your detailed explanation.
I understand the current mechanism and how hard it is to introduce any hook system cleanly.
I actually tried writing a minimal test and realized how challenging it is.

However, I want the functionality in comtypes that currently relies on ctypes to continue working in the future.
While it is true that the proposed hook mechanism might primarily be used by comtypes for now, I believe it is valuable to keep the door open for other packages to leverage it in the future.
There are already multiple COM client packages out there, and increasing the flexibility could encourage more entrants or use cases.

That said, I also understand the challenges of continuing to maintain this functionality.
Therefore, I think it’s worthwhile to simultaneously consider launching a project to provide such DLLs or helpers to implement them, as well as modifying comtypes so that server.register.RegistryEntries can optionally specify a custom DLL.

@bennyrowland

As far as I can see, one way for CPython to avoid hardcoding comtypes would be to look in the registry for the requested rclsid, and require comtypes (and others) to register Python classes in two places -- Windows itself (for _ctypes.pyd) and a Python-specific location (for the Python module).

However, that's a lot of new functionality. If we put that functionality in CPython, it would be hard to iterate on it -- the release schedule is slow, and if we find an issue, users wouldn't be able to easily update only the COM support part.
@bennyrowland's idea of a helper binary extension sounds much better.

@bennyrowland

Breaking the dependency on comtypes and providing a way for ctypes to implement an inproc-server in pure Python certainly requires a significant and complex effort.
I am inclined toward @bennyrowland's idea of a helper binary extension.
However, I believe that deprecating an API without first providing a recommended alternative would confuse users.

I have no objections to launching the comtypes-outproc-server-support project, which will provide the necessary DLL for implementing inproc servers, with the help of @bennyrowland, @encukou, and others participating in this issue.
I am also willing to allocate resources to modify comtypes to make it easier to use this DLL.

Once comtypes-outproc-server-support has been tested and confirmed to work in conjunction with comtypes, I will not oppose deprecating the hooks currently present in ctypes.
By ensuring that users encountering the DeprecationWarning receive clear messaging on how to migrate, we can avoid confusion.

It is worth noting that even if these hooks are removed from ctypes, there remains the option to implement a local server in pure Python, as demonstrated by @forderud.

I also think that if users have the permissions to install Python, comtypes, and register _ctypes.pyd in the registry to implement and register an inproc COM server, replacing _ctypes.pyd with a third-party DLL might not pose any issues.

Progress Update:

Regardless of the direction we choose to take, understanding what exactly is happening in comtypes' DllGetClassObject is essential.
To achieve this, I have been refactoring to improve readability and adding unit tests to better understand its behavior and prevent regressions.

The test for COMObject.IUnknown_QueryInterface, which is called internally by inprocserver.DllGetClassObject, is particularly noteworthy.

When passing an instance created by the high-level comtypes.client.CreateObject instead of an instance returned by the low-level ole32.CoCreateInstance function—without using byref—the test passes despite access violations during release hooked by the metaclass.

    def test_sample(self):
        dic = comtypes.client.CreateObject("Scripting.Dictionary")
        scrrun.Dictionary().IUnknown_QueryInterface(
            None, pointer(scrrun.IDictionary._iid_), dic
        )

...\comtypes> python -m unittest comtypes.test.test_comobject -v
test (comtypes.test.test_comobject.Test_IPersist_GetClassID.test) ... ok
test (comtypes.test.test_comobject.Test_IProvideClassInfo2_GetGUID.test) ... ok
test (comtypes.test.test_comobject.Test_IProvideClassInfo_GetClassInfo.test) ... ok
test_s_false (comtypes.test.test_comobject.Test_ISupportErrorInfo_InterfaceSupportsErrorInfo.test_s_false) ... ok
test_s_ok (comtypes.test.test_comobject.Test_ISupportErrorInfo_InterfaceSupportsErrorInfo.test_s_ok) ... ok
test (comtypes.test.test_comobject.Test_IUnknown_AddRef_IUnknown_Release.test) ... ok
test_e_no_interface (comtypes.test.test_comobject.Test_IUnknown_QueryInterface.test_e_no_interface) ... ok
test_e_pointer (comtypes.test.test_comobject.Test_IUnknown_QueryInterface.test_e_pointer) ... ok
test_sample (comtypes.test.test_comobject.Test_IUnknown_QueryInterface.test_sample) ... Exception ignored in: <function _compointer_base.__del__ at 0x00000208EBFBADE0>Traceback (most recent call last):
  File "...\comtypes\_post_coinit\unknwn.py", line 280, in __del__
    self.Release()  # type: ignore
    ^^^^^^^^^^^^^^
  File "...\comtypes\_post_coinit\unknwn.py", line 413, in Release
    return self.__com_Release()  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^
OSError: exception: access violation writing 0x0000000000000000
ok
test_valid_interface (comtypes.test.test_comobject.Test_IUnknown_QueryInterface.test_valid_interface) ... ok
test_valid_pointer (comtypes.test.test_comobject.Test_IUnknown_QueryInterface.test_valid_pointer) ... ok
test_custom_interface (comtypes.test.test_comobject.Test_QueryInterface.test_custom_interface) ... ok
test_dispatch_interface (comtypes.test.test_comobject.Test_QueryInterface.test_dispatch_interface) ... ok

----------------------------------------------------------------------
Ran 13 tests in 0.061s

In comtypes, the reason why the dst argument of CopyComPointer is specified as c_void_p in this particular case, despite being almost always a subclass of POINTER(IUnknown), is believed to be due to this behavior.

Please allow me more time to investigate what is minimally required to implement a COM server.

Progress Update:

I have been analyzing the registry entries required for comtypes to register non-frozen inproc servers.
Up to now, the creation of these entries has been handled through the same method for both non-frozen and frozen servers, with complex internal conditional branches determining their content.
I have separated the methods for generating entries into distinct ones for non-frozen and frozen servers, clearly defining the necessary and sufficient entries for each.

Additionally, I made it possible to test third-party packages by rewriting _ctypes.__file__ to point to any inproc server DLL file path (this line).
I also discovered extremely simple COM server examples in the comtypes documentation, which should allow for minimal PoC based on these examples.

Please allow me more time.

@moi15moi

Progress Update:

I released comtypes==1.4.10, which includes the refactoring of the previously mentioned registry entries.
Since no bug reports have been filed after about a week, projects and ecosystems that depend on it are likely functioning without problems.

COMObject is used when implementing a COM library or COM server with comtypes.
I hypothesize that understanding its behavior will help unravel the processing flow of a COM server.

Aside from defining the behavior of COM objects, COMObject is also used when defining event sinks. By investigating how methods defined in an event sink are triggered as callbacks, we might gain insight into how COM objects are defined in Python and how a server's behavior is implemented.

Thanks to @moi15moi’s contribution (enthought/comtypes#769), I noticed that functions for registering, retrieving, and revoking active objects using COMObject instances are implemented in comtypes/server/....
This implementation includes a process where a Vtbl pointer is copied to a COM interface pointer.
This might serve as another clue to understand the underlying mechanism.

Please give me more time.

Update README.md

If it can be useful to you, I have eventually made a DLL that can be used as a COM server (through the InProcServer32 subkey of the CLSID entry in the registry) and act as a bridge to a python module, calling its implementations of DllGetClassObject and DllCanUnloadNow. It currently loads a hard coded module (WICPy), which must be located in the same folder, but it would be possible to get the name/path of the module from a .cfg file or from a dedicated value in the InProcServer32 key. I have tested it with the IFileSystemBindData interface and its proxy/stub class factories (included in WICPy), in Python and PowerShell, and everything seems to work.

It can be found at https://github.com/PCigales/WICPy/blob/main/comserver.c and https://github.com/PCigales/WICPy/blob/main/comserver.dll. The source code has been compiled with 'gcc -shared comserver.c -o comserver.dll -I "C:\Program Files\Python\include" -L "C:\Program Files\Python\libs" -l python313'.

@PCigales

@PCigales

Thanks for your insight.
Knowing the GitHub Actions workflow for compiling and building the code you provided might offer some valuable hints.

My situation has changed, and I'm finding it difficult to allocate resources to this issue.

I think we might not delve into more detail than our current hypothesis-based discussion until we find a user who is actually using this feature.

Navigation Menu

Replace ctypes.DllGetClassObject and remove DllCanUnloadNow #127369

`DllCanUnloadNow`

`DllGetClassObject`

Linked PRs

In-Process Server

Be Sure to Export DllGetClassObject

In-Process Server

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Replace ctypes.DllGetClassObject and remove DllCanUnloadNow #127369

Description

DllCanUnloadNow

DllGetClassObject

Linked PRs

Activity

zooba commented on Nov 28, 2024

junkmd commented on Nov 30, 2024

In-Process Server

Be Sure to Export DllGetClassObject

In-Process Server

junkmd commented on Nov 30, 2024

junkmd commented on Nov 30, 2024

junkmd commented on Nov 30, 2024

zooba commented on Dec 2, 2024

encukou commented on Dec 2, 2024

bennyrowland commented on Dec 2, 2024

encukou commented on Dec 3, 2024

bennyrowland commented on Dec 3, 2024

junkmd commented on Dec 4, 2024

encukou commented on Dec 4, 2024

4 remaining items

junkmd commented on Dec 11, 2024

encukou commented on Dec 11, 2024

junkmd commented on Dec 11, 2024

bennyrowland commented on Dec 11, 2024

junkmd commented on Dec 15, 2024

junkmd commented on Dec 15, 2024

encukou commented on Dec 16, 2024

junkmd commented on Dec 16, 2024

junkmd commented on Dec 22, 2024

junkmd commented on Jan 19, 2025

junkmd commented on Feb 18, 2025

PCigales commented on Aug 9, 2025

junkmd commented on Aug 24, 2025

junkmd commented on Aug 24, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`DllCanUnloadNow`

`DllGetClassObject`