Learn from history.
Anyone who is working on improving terminals now, in the second decade of the 21st century, needs to learn from all of the work that went on in the 1970s and 1980s addressing much the same needs and wants, let alone what has happened since then.
Terminals got graphics capabilities. Operating systems vendors enhanced their operating systems with abstractions with far better programmatic interfaces like the KBD/VIO/MOU subsystems in OS/2 and the console subsystem in Windows NT, providing mouse and keyboard input messages that did not have to be parsed and directly addressable video buffers that could be written to and read from.
Real terminals and terminal emulators converged on the ECMA-48 and DEC VT control sequences. For terminal emulators people invented remote terminal protocols more geared towards common IBM PC compatible hardware such as AVATAR (see FSC-0025 from 1988), and even protocols for encapsulating higher-level things like TUI widgets.
* https://jdebp.eu/FGA/tui-console-and-terminal-paradigms.html
* http://ftsc.org/docs/fsc-0025.001
* http://ftsc.org/docs/fsc-0037.001
As for moving terminal handling out of the various operating system kernels, people were doing that years ago too. I wrote a white paper on it for Linux in 2006, and the ideas were not new then, they having already existed in systems like GNU Hurd and Windows NT. David Herrman, one of the systemd people, wrote KMSCON; which, ironically given the comments about systemd on this page, was actually included in systemd (as systemd-consoled) for about nine months until he (with much less fanfare than accompanied the inclusion and no comment from Lennart Poettering when asked about it) removed it again. Many people wrote framebuffer terminal emulators, including me.
* http://jdebp.eu./Proposals/linux-console-daemon.html
* https://cgit.freedesktop.org/systemd/systemd/commit/?id=ce7b...
* https://plus.google.com/+LennartPoetteringTheOneAndOnly/post...
* https://github.com/systemd/systemd/pull/747
Your stack exchange post is epic. Thank you!
I have some reading to do...
I probably should not point to any more, then. (-:
* https://unix.stackexchange.com/a/289871/5132