This is a long awaited update on my misadventures with the Emacs Widget system. This update contains some good news. From some points of view, it contains all good news. But the tone is coloured by a few real world happenings, so it is not all positive. I will allow myself some profanity, more so than usual, and this may not be charitable.
Emacs, the Program
Emacs is a beast that you can love or hate. Having worked with it and using its packages for a very long time, almost a decade by now, I can say that it is my favourite programming environment. It is rather a good program, and it has a rich and fruitful history. It is a program that existed longer than I have. It is a program that can keep up with the modern day best-of-the-best programming environments.
And yet, the quality of the code base and the code repository is not what I would consider good. The fact that this program can run for as long as it does without causing any major issue with respect to memory, is a miracle. I can appreciate its design and capabilities, the same way I would appreciate the design and capabilities of the Great Wall of China. It is functional, purposeful, and the ugliest code base that I have worked on.
When I started out, I had assumed that the causes of the issues with Emacs were deeply rooted, complex, and intricate. What I found is banal neglect. A text editor like this should cope with longer lines. A text editor like this ought to be able to interact with language servers and send a great deal more data. A software platform, such as Emacs, based around Emacs lisp can well scale to the level of an operating system, and cope with a great deal more performance-critical work. Do it well. The fact that it does not, has more to do with the fact that the blunt-force approach of accretion of features, without any design, any foresight or for that matter, any restraint, than it does with complex fundamental limitations.
Elisp is not the enemy. The programming model is.
What I Managed
I had been continuously experimenting with the code base, and I found that I cannot do many of the things that I wanted. Not because they are technically impossible, but because it requires a grand redesign of the program.
I managed to get very basic and unstable rendering of one buffer on screen using SDL. The performance was comparable to that of lucid. In all fairness, lucid is as fast as I can make the SDL backend, and is just as independent of the dynamic libraries on the system. Rewriting the backend to SDL accomplishes some things, but it would take years to replicate all of the existing functionality at the rate at which I managed to do it so far.
The build system is the most consistent source of frustration. The code base has a plethora of platforms that it must support and in very specific ways, and just getting the program to compile on Linux and on Mac OS, required a considerable amount of duplicated effort. While the low overall quality of the code meant that most code generated by LLMs was an improvement, the tight coupling and contexts and layers upon layers of ad-hoc solutions resulted in what I consider untenable issues. If I myself struggle with the build system, and all I can offer is a poor proxy for what already exists in lucid what reason do other people have to engage with the code?
The expansion of what the editor toolkit should be able to do requires some concepts to be restructured. Windows don't work the way I need them to. They have way too many bits of data carried from multiple layers of abstraction to one mess of functions. It is not at all capable of what I want it to be. The limitations of posframe now become readily apparent, and explain why such functionality is often avoided. It's not because such functionality is inherently difficult to implement… just that it would be inherently difficult to implement, without simultaneously rewriting much of the window.h and window.c.
Quite frankly what I need is something that is too close to window, to be accepted. People will say that because 90% of the functionality is shared, that I should use Windows for the purpose, and I should not implement viewports. In fact, not only would that extend to the people reviewing the eventual code merge into upstream Emacs, but also people in the audience today have suggested that. Extending windows with that 10% of extra functionality is intractable, and will open the door to many bugs. For this to be easy to do, one would have to do decades worth of refactoring.
The reason why refactoring the code base, good though it would be an idea in principle, and necessary in practice, is not going to work has a very simple explanation. People are stupid.
The code is there in the project in the shape that it is. The code hasn't changed, because most of the people working on it either have a pathologically poor taste and complete lack of awareness of why things are not usually done this way, or because they recognise it as problematic, but have learned to work around those limitations. In the former case the merge would be rejected, because of the implication that bad code is bad. People get defensive about their work. In the latter case, the merge would be rejected, on the basis upon which every initial change had been rejected: the activation energy needed to fix the original problem is too great.
Getting this project into a merge-able state is a problem not just for me, because shall we say… I can foot the bill. It is a problem for the project maintainer. There's volunteers that review the merge requests. There's coriphei that produce horrible code in one place, but are kept around because their poor code solves a problem that a naive rewrite cannot (c.f. redisplay).
And I would not be able to do what I want. This is a list which includes:
- Floating toolbars and panels that can be docked as in professional applications.
- Nuanced input processing that allows for input of more than the keyboard, and is not confined to technical limitations that are at least two decades obsolete.
- Viewport system that would allow you to run Doom as an elisp sub-program inside of Emacs.
- High performance canvas, allowing one to use Emacs as more than just a text editor with overlays.
- A flexible, portable and unencumbered programming platform for Elisp, that allows you to run Emacs as an operating system for distributing graphical programs, but with a unified system of configuration and packaging.
- A high flexibility graphical toolkit that would allow you at least the same level of programmability as the web does.
- All of this in the same Emacs package. It isn't a separate plugin that you have to fiddle around with. It's stock emacs.
- Something which respects the ethical commitments of the project. It has a hard-line stance on freedom-restricting software.
This is not another EAF. That already exists, and is far too unpalatable for me. This is not another posframe, because it is meant to be built-in.
And unfortunately this cannot exist. I would be butting heads with people that I should be friends with; I would be working around code limitations that I neither understand nor support; I would be committing to supporting a great many workflows, for which the old method of doing things works just as well.
I am in the precarious position of giving up on point 7.
I must fork, and start from basically a minimal Elisp environment. I must build to Emacs, what neovim was to vim.
I can only hope that this code will eventually be the way by which mainline Emacs operates, but for that to happen it must do some things right.
Why Fork
The simplest answer, is because it allows me to keep the things that I like about Emacs, and get rid of things that I dislike. And there are forks of Emacs, there was the Guile port. There's the rather misguided emacs-ng and before that there was remacs. There are a dozen of half-finished Emacs-based half-projects that have become abandoned, because the code was too radical to be merged upstream. Some of these included great ideas, such as more efficient communication with Elisp.
I shall therefore explain why I want to do something that is a bit more radical.
Decoupled Code
The main issue with the way in which Emacs operates is that it is a monolith. It being a monolithic repository is bad enough, but many of its components cannot be modularised. I cannot extract the graphical toolkit into its own library and expose an interface that can choose the toolkit live. It must be a monolith.
remacs and emacs-ng have done precious little to fix this. If anything, they made the situation worse, because instead of having reusable and re-used components, they instead provided a single executable that bundled everything.
What I would like to have instead is a clear separation of concerns.
ELI
I want there to be eli which stands for Emacs Lisp Interpreter. It is a stripped down version of the core of Emacs. This is equivalent to python3 on the command line, and it has the option of loading select packages and scripts. This allows Emacs Lisp to be used as a scripting language.
Gel, Gef, Gdired
I want there to be gel and gef. The former is a library of reusable components that can be called from Elisp, and that allow you to create User Interfaces. This is the toolkit. The Graphical Emacs Library should be something on feature parity with TkInter, and used as the basis for most of the editor. The graphical Editor Frame is the entrypoint into the text editor component. It leverages gel to draw the editor window. It is the text editor component, but it is not all of the system. It leverages Elisp, and it can run some of the plugins that regular Emacs can, but it is not limited to that.
I would then like to add more components to the system. I would prefer that there be more than just a few. Dired is a file manager that works around the limitations of the text editing paradigm. Textual interfaces allow great many things. But they are also limited and limiting. I believe that a file manager that leverages some of the graphical components, should exist.
Magit
Ideally the way in which the system is designed would permit packages such as magit to work with minimal porting. I want there to be interoperability between Emacs, and the components that I will design.
Use cases
Imagine that you are dealing with an exotic deployment system. It's kinda like Docker, but weird. You can write a few Elisp files to wrap the program in a UI. We are doing this today with Emacs already, I'm just thinking of providing those components a different way.
Those components show up in your graphical window manager's task-bar with different icons, can be tiled with your system's built-in windowing program, and are effectively indistinguishable from what you could create with PyQt. Except it's not Python.
Then you have a collection of CLI programs that create those windows. One of which is a shortcut to a macro-based editor. You use it they way you would use sed, except instead of trying to do the manipulation in your head, you open your gef, record a macro, as you would, close the frame, and it automatically inserts the directives. They can be in-line, and they can be a file. Sort of unix-y, except convenient and easy to reason about.
Then there's going to be eplot, which is a scientific plotting program that has a native Elisp-based drawing system that gives you access to calc. But it draws to PDF, to ePS, to the screen and runs fast.
Finally, your gef can multitask and multiplex. The best way to put it, is to say that the current Emacs doesn't need multithreading as much as it needs the equivalent of Posix processes that don't necessarily load all of your customisations.
Multiprocessing
The current greatest headache is the amount of time, a task could have been done easily in the background, but is instead done in the foreground thread, and locking up the UI.
This is not an inherent property of programmable editors. It is not necessarily a property of Elisp. It is, however, a property of how the programming model for Emacs looks like.
Right now, all of your packages interact with potentially all of your packages. Every single line of Elisp can potentially affect every other line of Elisp. There is the concept of a buffer local variable, and lexical scoping is often encouraged, but not enforced, but because you cannot trust the script to do the right thing, you have to load one big interpreter instance, load everything into it, and block every time the GC runs.
This cannot be fixed without a major rewrite. A major rewrite that can happen as a consequence of writing everything from scratch and imposing some further limitations onto Elisp.
This means that the language can be updated such that packages can be loaded in two modes. One which is suitable for the new process-isolated model, and one which has the same behaviour in the old system. It may look like a collection of functions, setn that resolves to sending an inter-process message on gef, and setq in regular Emacs.
This can and will eventually allow for more performance-oriented programming. Elisp is not a great medium for performance critical work, but neither was JavaScript back in the early days of the internet. Frankly, removing blocking calls and a universal stop-the-world style of GC will most likely fix most of the performance issues. Web-browser-based editors such as VSCode and Cursor don't have particularly good performance characteristics either.
Comprehensive Extensible Input
The Emacs input system is rather archaic. The limitations that it imposes are both annoying and limiting. But imagine a different world. Imagine that you could have a macro pad that is entirely managed by the editor. Depending on the buffer, the major mode and some minor mode variables, the input of each key becomes context-sensitive. You can draw cartoons, such that even if you have no labels on the macropad, you could still figure out which key does what in which context.
Another example would be using the editor with voice commands. I quite happily use numen for most of my text input needs, but I'd rather be able to give context-sensitive commands through a programming model in a language that is designed to map input to action.
Elisp can do that very well with keymaps, but it struggles with other kinds of input. Mice and context menus are a great example of how some of this can be mapped back into keymaps as a concept, but I believe that a refactoring of the input system is in order.
This will break compatibility with the input-sensitive packages in Emacs. But I quite frankly think that chorded input can only be done at the level of input event processing. The major modes provide syntax-table level context, but not other kinds of context. I believe that can be changed, but not in mainline emacs. Changing how major modes work, would break far too many packages, and in a fork, I can simply say "they are not compatible, use this instead". Upstream, the collective voices of all the authors of the packages that I have broken will prevent the merge.
Quite honestly, I'd rather not engage with that.
Comprehensive Accessibility
Emacspeak is an afterthought. I don't have the best eyesight in the world. I may need to use it one day.
I'd rather have something that is a much more robust system.
Elisp Programs
I sometimes need to use Magit just to manage the Git repository. At the moment, running that is a matter of:
- Open editor window.
- Navigate to the location
- Open Magit.
This can be shortened considerably. And it can also be extended into oblivion. Personally, I just like the idea of being able to type magit . into the terminal window, or better yet, have a graphical file manager that does not take ages to load.
I can also create domain-specific programs that don't require and don't assume text editing. I want to be able to draw SVG inside of a program that respects my ability to program complex interactions. There isn't an Emacs of graphic design, and I wish that niche were not vacant. I'd be able to find quite a few applications for macros and sequences and composing functions.
Better Languages
The main problem with trying to upstream into the Emacs Savannah repository is that this would impose a great many limitations on what I could do.
I would have to use C. And C by itself is not a bad language. It is rather simple for what it can do, and conversely it can do quite a bit given its simplicity. But it is not the only choice, and has some great drawbacks.
It is much easier to do data-oriented design in Zig. Optimising code such that I could e.g. create a DAW that lives inside Emacs would be much easier in Zig, than it would in C.
The fact that the incomprehensible jumble of spaghetti that is the Emacs code base does not leak memory and is stable enough that it does not crash if operated days at a time, is not a consequence of the elegance of its design. The new systems in place would have a myriad of bugs, tracking which down would either rest on my shoulders and stretch my time budget, or would be delegated onto the existing maintainers. Neither outcome is particularly good.
Rust would take care of those issues, and more.
Removing the restriction on having to upstream the code removes the confines of C. I would be able to modularise the work to my heart's content, write what is sensible in the language sensible for that task and that task alone.
Further, the process-isolated interaction scheme would allow modules to be written in other languages as well. Emacs lisp is going to be the first and mainly supported language, but it would not be inconceivable to have the C-code of Doom, merely have small adjustments made to render to a window canvas, and use the aforementioned input system instead of its own. At the moment, this is quite hard to do, because one has to think about the multi-modal interactions between Elisp and other objects.
Conclusion
I am essentially writing a brand new text editor from scratch, with the requirement that it used Emacs Lisp and maintained some compatibility with some Emacs packages.
Eventually that code can be subsumed into Emacs, as happend with XEmacs, and it might die a painful slow death as emacs-ng did. This is a me problem now.
I'll borrow liberally. If you want to play around with it, I shall post a link to the repository when it is public.
If you want to play around with the intermediate versions of the code… I'm afraid there's not much to play with. It was unstable, it barely built and I cannot spend any more time trying to unfuck this jumbled mess. The new code shall not borrow any from the old experiments.
This may sound like bad news, but this can have some positive outcomes. I'm rather unhappy that doing the smaller incremental changes is not going to work… but to be quite frank… this would not have worked, because either Eli Zaretski is operating under constraints that I cannot operate under, or he is blind. In either case, all I'm doing is avoiding the ranks of a plethora of projects that never made it.