holy_city 5 days ago

This is similar to the work done in winapi [1], com-imp[2] and my own tangential work on porting VST3 [3] to Rust [4]

I'm really glad MS is doing this. What needs to be a bit clearer to me is how they maintain ABI compatibility under the hood of MSVC for COM interfaces (which uses the vtable layout of an inherited class) and how that's compatible with MinGW/GCC stacks on Windows, mostly what can break it. I got stuck porting VST3 with multiple inheritance, and it was a headache trying to reverse engineer the appropriate struct layouts for COM implementations.

[1] https://github.com/retep998/winapi-rs/blob/0.3/src/macros.rs

[2] https://github.com/Connicpu/com-impl

[3] https://github.com/steinbergmedia/vst3sdk

[4] https://github.com/m-hilgendorf/cli-host (sorry for the messy code, it was a weekend of hacking away trying to host a VST3 in pure Rust)

  • barrkel 5 days ago

    COM is an ABI standard. The structs are defined in terms of C with Winapi (stdcall) calling convention. Very little needs reverse engineering - it's all pIntf->vTable->func(pIntf, ...). You can explain it on a whiteboard in a couple of minutes.

    The way MSVC does it may need reverse engineering (it may be patented btw). I could explain how Delphi implements COM interfaces, but any specific implementation is actually more complicated than the ABI, because they're trying to add implementation ergonomics on top of the basic calling convention.

    • ChrisSD 4 days ago

      > The structs are defined in terms of C with Winapi (stdcall) calling convention.

      This isn't quite right. The calling convention is a Windows C++ variant of stdcall. See this issue: https://github.com/rust-lang/rfcs/issues/1342

      However, 99% of the time the difference won't be an issue.

    • holy_city 5 days ago

      Is the ABI documented anywhere? Every time I google around for it, I just get information like "COM is ABI stable and language agnostic" but not what the ABI is. I've successfully implemented implementations of single COM interfaces and get the basics, my trouble was in implementing many interfaces for the same implementation and running into copious segfaults when testing the Rust implementation through a reference app written in C++.

      • pjmlp 5 days ago

        Here are some good COM books.

        Essential COM by Don Box

        https://www.amazon.com/Essential-COM-Don-Box/dp/0201634465

        Inside COM by Dale Rogerson

        https://www.amazon.com/Inside-Microsoft-Programming-Dale-Rog...

        Then if you have access to public libraries, maybe one of them has one of the several Microsoft Systems Journals issues, later MSDN Magazine, with plenty of low level COM articles.

        COM is from the days where good documentation was to be found in books, not on the Interwebs.

        • teh_klev 5 days ago

          +1 for Essential COM by Don Box, it still survives my bookshelf purges..."just-in-case". IUnknown and IDispatch are burned permanently in my memory from a period of my life building a COM/CORBA bridge.

          • alxlaz 4 days ago

            I'm gonna piggyback on your comment to give another shout out to Essential COM. I haven't touched COM in ages but that book is so good I still pick it up every once in a while -- and it was a lifesaver back when I did touch COM on a daily basis.

        • taude 5 days ago

          Wow, that's a throw back to the late 90s, early 2000s. I'm still slightly scarred from working with ATL and COM. A lot of people around here would probably be surprised to hear that there were even Python -> COM bindings back in the day that were even used to ship server software once upon a time. Anyway, I remember that Don Box book well.

          • ptx 5 days ago

            Not only "back in the day" – those COM bindings for Python were most recently updated last month: https://pypi.org/project/pywin32/#history

            I highly recommend using them if you're ever forced to work with VBA and need to implement some non-trivial functionality.

            • 7thaccount 5 days ago

              Yea...Python and Powershell are the only languages that I know of outside C#/VBA/VB.NET with decent COM support. I hate COM, but it works.

              • pjmlp 4 days ago

                Delphi, Eiffel as well.

                • 7thaccount 4 days ago

                  True... I've looked at Delphi, but at this point I doubt the license fee is worth it compared to say C#. I've never known anyone who actually used Eiffel though.

            • crtlaltdel 5 days ago

              oh how i wish i knew this back when...or just looked harder i guess ;)

      • barrkel 5 days ago

        What's to document? The stdcall / WINAPI calling convention, which is of course OS and architecture dependent, but can be summarized as on the stack, right-to-left, callee pops.

        The rest is just that interfaces are doubly indirected to get to a vtable (an array or struct of function pointers), the method is chosen by ordinal in the vtable (first three are always the IUnknown methods), and the interface is passed as the first argument.

        How you construct those vtables and how you select one to return in QueryInterface, and how you implement interfaces (i.e. traits) so you can convert them into a vtable is where all the work is. You can do anything you like that works as long as it's called according to the COM conventions.

        Delphi works by implementing each method in the vtable with a stub which subtracts the offset of the vtable pointer in the instance data from the passed in interface, and then jumps to the implementation method on the instance after the instance has now been adjusted. Instances look like this:

            [class pointer, the native vtable] <- normal instance pointers
            [vtable interface 1] <- COM interface pointers
            ...
            [vtable interface n] <- COM interface pointers
            instance field 1
            ...
            instance field n
        
        So you can see that in order to convert a COM interface pointer into an instance pointer that the methods expect, the COM interface pointer needs to be adjusted depending on the offset of the vtable in the instance.

        In a language with multiple inheritance like C++, the compiler vendor targeting Windows may choose a layout which is suitable for COM's calling convention (there's more than one way to do MI, e.g. fat pointers is another way to go that wouldn't be COM-compatible). If the vendor does that, then they make implementation of COM interfaces much easier. And if they don't, well, life isn't going to be easy. Technically you could do a bunch of stuff with reflection and code generation, but that's harder and harder these days with security restrictions around code writing code. You could write macros which create statically initialized structures of the right shape, and fields of the right type, to emulate the same effect as Delphi's scheme as I sketched above, or some other method which would work with COM's calling convention.

      • snagglegaggle 5 days ago

        Realize that COM is supposed to be used with code generators. You can write it in raw C even, but it quickly becomes intractable. Perhaps I did not read completely but this is something I see missing from the post -- the annotations fit what the pre-pass C++ code looks like, but don't mention that layer?

      • sayrer 4 days ago

        You may want to look at XPCOM. At one point it interoperated with MS COM, and it might still.

  • pjmlp 5 days ago

    COM layout is followed upon most mainstream Windows compiled languages, namely major C++ compilers, .NET, Delphi, Eiffel, Ada, so it is not MSVC++ keeping ABI compatibility under the hood on their own.

    • holy_city 5 days ago

      My issue isn't ABI stability but the ABI itself w.r.t vtable layout. Best I can tell it should be similar to Itanium's spec? [1]. It's been months since I did this, but iirc my problems stemmed from having multiple interfaces on top of the same implementation and the ordering/layout of those interfaces, though the IUnknown interface which is supposed to handle that.

      [1] https://itanium-cxx-abi.github.io/cxx-abi/abi.html#vtable

      • barrkel 5 days ago

        COM is agnostic as to how you do multiple inheritance - it doesn't have the concept. It specifies the QueryInterface protocol, but you don't need to return the same instance for the result of the QI call, just one that uses the same lifetime refcount.

        Tear-off interfaces and delegated implementations are things in this world.

        • ptx 5 days ago

          Meaning that the AddRef/Release counter has to be shared between that group of objects?

          • pjc50 5 days ago

            Same object, different interfaces.

            • magicalhippo 5 days ago

              You can implement the other interfaces using other objects, but they should delegate the ref counting to the root/main object.

      • jcranmer 5 days ago

        MSVC uses a completely different ABI from Itanium, and you shouldn't rely on the Itanium ABI to inform you what it might look like.

        vtable layout in the most basic situations is going to be accidentally portable because those situations boil down to "it's a struct of function pointers," and there's only so many ways you can order the fields of a struct. But even here, MSVC uses a quite different ABI: the order of the vtable entries can change if you overload a virtual method with a non-virtual method.

        • spacechild1 4 days ago

          AFAIK, non-virtual methods never affect the vtable layout. But when you overload a virtual method with another virtual method, the ordering in the vtable is unspecified!

          Also, a public COM interface mustn't have a virtual destructor, because some compilers (e.g. recent GCC) put more than one method in the vtable. Implementation classes might define a virtual destructor, though.

      • Const-me 5 days ago

        If you can read C#, maybe this will help: https://github.com/Const-me/ComLightInterop

        Specifically, this class implementing vtable callable from C++: https://github.com/Const-me/ComLightInterop/blob/master/ComL...

        About multiple interfaces, all of them need 3 first vtable entries pointing to the 3 IUnknown methods. Also, don't forget that when client calls QueryInterface on any interface of the same object with IID_IUnknown argument, you must return same IUnknown pointer. Some parts of COM use that pointer as object's identity.

thrower123 5 days ago

COM interop used to be a big buzzword say, a decade ago, but I haven't really heard or seen much usage of it since.

Most of my experience in using it has been horribly abusing it to drive Office applications and Notes to do some awful things because at the time there weren't any libraries to manipulate certain file types. I have done some truly heinous things with the Lync/Skype COM-based client SDK, back in the times when the Office 365 version of that product was viable, and there was no other API options available.

  • emddudley 5 days ago

    > I haven't really heard or seen much usage of it since.

    I think COM is incomprehensible to most developers today--it certainly is to me, and I maintained a .NET application with COM components for several years! COM is so arcane and verbose.

    Why invest hundreds of hours learning something with no clear benefit or future? The documentation introduces COM as:

    https://docs.microsoft.com/en-us/windows/win32/com/component...

    > COM is a platform-independent, distributed, object-oriented system for creating binary software components that can interact. COM is the foundation technology for Microsoft's OLE (compound documents) and ActiveX (Internet-enabled components) technologies.

    What does that even mean? It sounds obsolete.

    • ralphc 5 days ago

      When I read the headline it was a mix of WHYYYYYYYY and NOOOOOOO

      Really, can't we just leave COM in the 90's?

      • pjmlp 4 days ago

        Win32 is the one left in the 90's.

        COM gets regularly new APIs.

    • pjmlp 4 days ago

      Except the one without a bright future is Win32.

      Since Vista all major APIs are based on COM.

      • emddudley 4 days ago

        My point is that for the vast majority of developers this does not matter one bit. The Windows API is not relevant to the work that most developers do today. In the cases where a developer actually needs to interact with Windows there are easier technologies available.

        • pjmlp 3 days ago

          Yeah, if we are talking about web developers I guess.

          How do you think those easier technologies are implemented?

          Just to give an example, React Native for Windows uses UWP.

          As for not being relevant, the software for a platform with 80% market share across desktop and laptops, does not write itself.

  • siggen 5 days ago

    Marshaling COM objects for use among different threads was fun. Remember the function 'CoMarshalInterThreadInterfaceInStream'? :-)

    I feel Windows' C API (i.e., Win32 API) was adopted at a larger scale than COM. It's much easier to use the Win32 C API than COM objects. You'd have to use COM if you wanted to program the shell or DirectX but most of everything else you could use the C API.

    • pjmlp 5 days ago

      Only if you intend to use pre-Vista Win32 APIs only, afterwards most of the new stuff comes packaged as COM or .NET.

  • pjmlp 5 days ago

    Since Windows Vista the large majority of new native APIs on Windows are based on COM.

    You probably haven't heard much about it, because now it has the fancy name of UWP, or it comes with .NET wrappers on top.

    • withinrafael 5 days ago

      UWP is ambiguously defined and the acronym's usage is on the way out. An evolution of UAP and Windows Store apps, it boiled down to a whitelist of APIs allowed on multiple devices (the U in Universal) and encompassed Win32, WinRT and COM APIs. UWP itself isn't an entity with APIs, is not a technology, and the original concept is dead thanks to new platform leadership at Microsoft. (Still in play on Hololens, Surface Hub, and other sandboxed devices for now...)

      That said, "UWP" is confusingly also used to refer to Windows apps that tap into WinRT APIs, participate in the modern app lifecycle, are modeled and have identity, or generally tap into newer technologies such as AppContainer. You'll see this, for example, perpetuated in Visual Studio in its project templating. It's a BIG OL' MESS right now; I'm working with/pushing Microsoft to get this communicated clearly and mopped up but old habits die hard.

      • pjmlp 5 days ago
        • withinrafael 5 days ago

          These docs are outdated and incorrect, sadly. For example, nearly everything under the UWP heading can be done via traditional Windows apps. Current leadership seems to get it, "write an app, use any APIs you need" (rough paraphrasing). Just hasn't fully set internally and externally so these warts exists.

          • contextfree 5 days ago

            On a technical level, there are still some apps that can get CoreWindows, respond to PLM events, receive callbacks from contracts involving CoreWindow UI like Share target, as well as go into certain UI modes involving shell participation like modern fullscreen and CompactOverlay, and some that can't (without stapling them together with one that can via Desktop Bridge) ... aren't there? Maybe that will change in the future but it hasn't yet.

            And then there are some OS editions where some apps can't run or can only run in a container environment.

          • pjmlp 4 days ago

            Waiting to see that message at BUILD 2020 or the upcoming Ignite.

            Until then, sorry to say but I follow up on what is available to us.

  • hnick 5 days ago

    Yeah we're on a bit of a purge right now to remove a lot of MS Office automation. We're just reading data from files - nothing fancy - but it's all over the place and apparently technically outside of licensed usage so compliance is cracking down on us.

    COM is an interesting concept. If it were never invented I have a feeling native apps would be spinning up a simple REST API server instead, since that's what we all do now apparently.

  • secraetomani 5 days ago

    DirectX is COM based. Which means that pretty much every Windows game, and many applications (like Chrome) use COM, at least in part.

    • thrower123 5 days ago

      I've done a fair amount of work with DirectX, and I've never particularly had to be cognizant that it used COM.

      That's probably more a reflection of DirectX being reasonably well designed.

      • pcwalton 5 days ago

        It's mostly a reflection of the fact that you don't have to implement COM interfaces to use DirectX—you only have to call them. Also, the constructors are wrapped in functions like D3D11CreateDevice() so that you don't have to use raw functions like CoCreateInstance().

        If you're using an API like DirectWrite (at least older versions) that requires you to implement interfaces to do basic things like load fonts from memory, then the COM underpinnings of the APIs become very apparent.

      • secraetomani 5 days ago

        That's because there are two COMs:

        1. The good COM, which is just a class-like ABI.

        2. The horrible OLE COM, which allowed you to embed Word docs in Excel and the like which is what most people associate with COM.

Crinus 5 days ago

Weirdly, the github page says:

> COM has been superseded by WinRT which builds on COM to provide even more guarantees about the binary interface.

Yet the WinRT page says:

> The Windows Runtime is based on Component Object Model (COM) APIs, and it's designed to be accessed through language projections. A projection hides the COM details, and provides a more natural programming experience for a given language.

Which implies that WinRT doesn't supersed (ie. replace) COM but it uses COM itself (though the "language projection" part makes me think that the way it is used is so convoluted that you need language extensions to make it tolerable :-P).

  • pjmlp 5 days ago

    WinRT is an evolution of COM.

    Basically there is a new base interface, IInspectable instead of IUnknown, and many more datatypes are allowed to cross the wire between processes, making it easier to also use generics, lists and arrays across COM libraries.

    Additonally, .NET metadata libraries are using instead of the old COM type libraries.

    If you prefer, it is the original design of .NET before the CLR came into play, and related to the Longhorn architecture, just using COM instead of .NET. A path started in Vista.

    Now for this Rust approach to win the hearts of Windows devs, it needs to be as productive as C++/CX or C++/WinRT VS tooling, in all COM/UWP usage scenarios.

    • mwcampbell 5 days ago

      > Now for this Rust approach to win the hearts of Windows devs, it needs to be as productive as C++/CX or C++/WinRT VS tooling, in all COM/UWP usage scenarios.

      Possibly related, I just spotted this [1] on the personal blog of Kenny Kerr, creator of C++/WinRT:

      > On a personal note, I’m spending a lot of my time working with the Rust language and can’t wait to share more about that eventually.

      Disclosure: I work on Windows at Microsoft, but not in this area. Posting on my own behalf though.

      [1]: https://kennykerr.ca/2019/10/09/xlang-cppwinrt-repos/

      • pjmlp 4 days ago

        Very interesting, given how he made C++/WinRT possible.

        Thanks for bringing it up.

    • Crinus 5 days ago

      One thing i'm not sure about WinRT and cannot figure out from the docs, AFAIK it is possible to "bit bang" (or "byte bang" if you will) a COM object by arranging everything in memory using plain old C structs (or equivalent for other languages) from compilers that weren't COM aware. Is it possible to do the same with WinRT or it requires the language to be WinRT aware? F.e. could i get something like TCC and write some code that uses a WinRT object even though TCC has no idea about WinRT?

ajxs 5 days ago

I'm interested in more information regarding the Safe Systems Programming Languages team's work. Did they assess other languages other than Rust? I'm scanning through their blog entries, without having read every single one in depth so please pardon me if I miss something, and I'm not seeing anything mentioning formal methods or other systems programming languages. It's a very interesting development for sure!

jchw 5 days ago

Very cool.

I have played with COM in Go to a lesser extent. It’s not too bad actually. My intent was to write Go wrappers around all of the TSF COM interfaces, but I burnt out after getting basic IME working - TSF is enormous!

  • MehdiHK 4 days ago

    Hi,

    I believe you are talking about this repo https://github.com/jchv/go-tsf

    Do you know any other project that's working on creating basic IME on top of TSF in Go?

    • jchw 4 days ago

      You are correct, and no I am not. Is your goal to build an application with IME support, or an entire IME?

      • MehdiHK 3 days ago

        An entire IME.

        • jchw 3 days ago

          Ah I see.. sadly, I never really figured out a great way to support full COM interop in Go. It would be ideal if you could implement an interface and register it as a COM object, but I hadn’t hashed out exactly how.