Developer Framework - Ubiquitous Bash - InterProcess Communicaton, UNIX, MSW/Cygwin

“Ubiquitous Bash” (CC0 public domain license) is a huge ~800KiB ~17.5kSLOC script (‘lean’ variation ~280KiB) written by me over about 2-3 intensive years . Mostly, its purpose is to provide a ‘framework’ for ‘_getScriptAbsoluteFolder’ (portability), managing dependencies, launching applications within virtual machines (with translated fileparameters), abstracting filesystem locations (tricking Arduino), setting environment variables to open ports (eg. for OpenOCD/ARM-GDB ), reading scripts from project directories (Arduino ‘sketch ops’), navigating remote shell multipath, installing/configuring a network of OS image/LiveCD/VM using ChRoot/VirtualBox/Qemu/Docker (any backend supported for same image)…

Since I don’t have much time to document or even demonstrate best practices, or even if I did so fully, developers might not search/read it anyway…

Just ask me for support. As long as I don’t have to bury my head in another complicated project in the middle of some other complicated infrastructure projects (which I hope not to be doing anymore) it is *really easy for me to just answer questions.

Please share this with other developers who may be interested as a sort of ‘very early release’.

Anyway, I hope this is not too much trivial self-promotion…

A few new features have been added that are starting to make this directly relevant to VR.

UNIX/MSW/Cygwin

A ‘Cygwin Portable’ installer is included, with functions to rewrite the symlinks (replacing or making them relative as desired), as well as to pack/unpack ‘tar.gz’ format.

‘Anchor’ scripts run the neighboring ‘ubiquitous_bash.sh’ script with the first parameter being their own name. These are ‘shortcuts’ like the LNK files many MSW users are familiar with. However, the ‘.bat’ anchor scripts also search a number of standard locations if not neighboring the expected script and are simultaneously interpretable as Batch/bash scripts due to comment characters in their header. Thus, on both MSW and UNIX these scripts may call up exactly the same program.

Override functions allow native binaries (eg. TigerVNC , VirtualBox) to be called on the MSW/Cygwin ‘platform’.

Thus, an application ‘contained’ within “ubiquitous bash” may be made to work portably under both MSW and UNIX.

Bonus - run ‘_bash.bat’ to get a nice bash shell on UNIX/MSW/Cygwin . Or ‘_test.bat’ to do some really extreme tests of semi-real-time InterProcess Communication timing, dependencies, shell behavior, etc.

MetaEngine

Need to connect multiple output files/pipes/buffers to another program with multiple input files/pipes/buffers? MetaEngine is an efficient framework to create such complex ‘pipelines’ .

Sort of like LabView or GNURadio Companion I guess, but a bit more accessible to the wider variety of programs available from a typical UNIX shell (which is quite a bit more powerful within “ubiquitous bash” context). In particular, IIR biquad filter preprocessor macros should be considered, for having enough efficiency to implement extremely narrow/brickwall filters of very high ‘Q’ and thousands of orders on microcontrollers.

MetaEngine also specifies and illustrates a variable resolution coordinate system for arranging files. This is intended to share data between such things as graphics/physics/netcode processors at ‘instances’ of different scale (think: Elite Dangerous), both in memory (eg. C implementation) and more persistently on ‘disk’.

Example MetaEngine code . Example commands included in ‘README.md’ .

InterProcess Communication

Need to share data from a program or serial device (eg. VR controllers, 3D printer motors, etc) with several other programs or devices, and maybe the existing ‘HID’ drivers are not flexible enough? Emulate a ‘shared pair of wires’ with ‘queue’, and send a packet. Both ‘tripleBuffer’ and automatic connection of multiple ‘pipes’ are available as backends. For less real-time purposes, ‘interactive’ and ‘database’ backends exist as well.

Such a ‘shared pair of wires’ intentionally resembles the sort of ‘anything to anything’ bus found in some hardware systems (ie. CAN Bus) . Aside from precompiled binaries for VXWorks using some pipe feature I am not familiar with, it seems software implementations of this technique were not common, let alone such multiplatform software implementations.

Having this implemented in shell script, the same script that can launch applications in entire Virtual Machines ad-hoc on UNIX/MSW/Cygwin should make it a lot easier to share devices or files between different applications, independent of any specific precompiled binary program specific to a single project on a single platform.

Admittedly, for performance reasons (either due to MSW/Cygwin script init times or single-application high-speed memory-only graphics/physics data), precompiled binary code may be needed in some cases. However, even then, these shell functions may serve as (compatible) reference implementations.

https://github.com/mirage335/ubiquitous_bash/blob/master/queue/tripleBuffer/scrap.txt
https://github.com/mirage335/ubiquitous_bash/blob/master/queue/tripleBuffer/test_broadcastPipe_page.sh

https://github.com/mirage335/ubiquitous_bash/blob/master/queue/aggregator/static/scrap.txt
https://github.com/mirage335/ubiquitous_bash/blob/master/queue/aggregator/static/test_broadcastPipe_aggregatorStatic.sh

https://github.com/mirage335/ubiquitous_bash/blob/master/queue/zInteractive/scrap.txt

https://github.com/mirage335/ubiquitous_bash/blob/master/queue/zDatabase/scrap.txt

https://github.com/mirage335/ubiquitous_bash/blob/master/_doc/broadcastPipe/README.md

Future Work (where some of this may be going)

  • Installer for extendedInterface . VoiceAttack macros , MSW workarounds , SteamVR workarounds , PimaxExperience batch files , sequence of background programs to launch for the likes of DCS World , etc. As of now, there are a lot of features (probably way too many) for users to figure out what to do with.

  • KWrite / Kate / Virtual Machine / OVRDrop Integration

  • OpenHMD / OpenXR / Pimax Vision 8kX - Self-contained multiplatform ‘driver’/‘compositor’ build/development environment .

  • Unity / Unreal / FOSS Game Engine / etc - Self contained multiplatform build/development environment. May drastically improve flexibility and reduce dependence on any specific game engine.

  • Framebuffer interface to work independently of OpenHMD/OpenXR/SteamVR with adaptive resolution input/output to ‘force’ sensible values respective of TotalSR .

  • Simulated 3D printers, using Klipper firmware (LinuxVM), announcement of emulated serial ports, and ‘devices’ implemented as game engine objects (3D printing in VR).

  • Better MacOS support. I do not have a Mac, nor much time. Then again, MacOS is not exactly known for being likely to have the best possible GPU (yet) .

  • Possibly much more, and much more important, things.

_

Some minor problems still need to be overcome. All of them are expected to be trivial.

  • VirtualBox / QEMU native MSW binary overrides. Such overrides are already done for TigerVNC, and should be absolutely trivial, as the binaries basically do the same things with the same command line arguments. Most of the difficulty will probably fall on the due diligence and testing required to make sure every command behaves as intended in every plausibly relevant way.

  • VirtualBox / QEMU serial port declaration. Connecting things like 3D printers requires knowing where the virtual serial ports may be. Trivial because this just means adding a few parameters when the VMs are configured, and writing out the environment variables some place.

  • Remote shell multipath navigation does work through MSW/Cygwin already, and this is helpful with sites like GitHub. However, there may be some annoying prompt windows, and the latest versions of the relevant scripts have not been tested in some time.

  • Years of use without problems builds confidence. Construction of Virtual Machines, networks of Virtual Machines, and the Inter-Process Communication functions, are relatively new features. The latest versions are believed to be robust and feature complete, however, half a year or so of everyday use is desired.

_

A few things probably will not happen

  • MSW/Cygwin ChRoot/Docker support. MSW probably cannot even be built automatically into a LiveCD/VM to begin with, and there is probably no real need to build UNIX systems directly from it either. At least use a LinuxVM to get these capabilities with “ubiquitious bash” .

_

Before any of these more recent developments, the ‘PanelVM’ was created. This turns a typical Linux workstation into something more like Atom text editor (or other typical IDE), opening a project across conventionally assigned virtual desktops by project script from the host filesystem, with keyboard shortcuts to allow voice command, and single-click tree browser navigation (bringing already open documents to the top of the window manager rather than reopening). This is very useful during flights in VR.

Further improvements to the MSW/Cygwin VM backends will allow the PanelVM system as well to be copied and used portability.

1 Like