Our Team

Len Du - Senior Machine Learning Engineer

Company Name (if applicable)

Project details

A lot can be said about how (hyper)graphs can be considered closer to the “essence” of the semantics of a program (or “meaning” in the general sense), such as naturally circumventing α-equivalence (that we can change variable names inside a program without changing its behaviour) and being “point-free”, which might have been one of the many motivations behind the original OpenCog. But to reap these benefits, directly editing the graphs really needs to be natural and approachable.

It is sort of a Sci-fi trope (e.g. Iron Man) to feature a 3-dimensional (or “spatial computing”, as marketed by Apple) interface where the user interacts with, manipulate, and in particular make connections between, objects or “nodes” floating around in a 3-dimensional space, as opposed to draging-and-dropping on a 2-dimensional plane in all real world visual programming environments, the most significant and established example being LabView.

The artistic intuition might actually happen to have some scientific merit in it. The representation of, and the interaction with graphs in 3-D, as opposed to that in 2-D, is far from superficial. It is a known result that arbitrary graphs (or, in the most demanding case, complete graphs) can be embedded in a 3-D Euclidean space (R^3) rather than a 2-D one (R^2). A common complaint about real world 2-D visual programming languages, is that they seem to be very bad at scaling up with complexities. When the expressed logic is simple – corresponding to a short script in textual programming languages – the visual language seems very attractive. When the project gets larger, the scalibility problem starts to surface, often eventually remedied by implementing part of the functionality in textual languages. This is not so much of a problem in some niche applications such as composable shaders, where the complexity is limited by, among other things, that loops are not typically involved.

Embedding a graph in 2D, often with limited edge crossing allowed, introduces all sorts of intellectually interesting problems, having invited a long history of research. Users of a 2D visual programming environment are often effectively solving such problems with their brain, rather than leaving these problems to the computer. Hopefully, as the complete graph is embeddable in 3D, directly editing the graph in 3D can take this burden off. If we were to visualize the process of graph rewriting, it may also be necessary for successive layouts to be done without needing user input.

Note that hypergraphs and metagraphs (in the sense that edges can point to edges, in addition to nodes – a “metagraph” can otherwise mean different things) are not necessarily embeddable even in the 3D space (or in any Euclidean space), so for the purpose of visualization and interactive editing, they have to be converted to proper graphs. Hypergraphs can readily be converted to bipartite graphs. We can then relax the bipartition by allowing edges between the “hyperedge” nodes.

In this project we will use MORK/MeTTa as the underlying machine representation, so that future growth in 3D graph editing could gravitate towards the application and extension of MORK/MeTTa. It could otherwise also be an attractive proposal for knowledge graph researchers or practitioners outside the OpenCog/SingularityNET ecosystem to come up with ideas of similar 3D visualization or interactive editing methods. A major point of this implementation is to potentially attract such efforts to be based on MORK/MeTTa and grow the influence of this ecosystem.

There are a few design choices that had been deliberated on. First is how the human user would give input to the graph. Second is whether we use VR – while we don’t have holographics like those in Sci-fi movies, VR headsets are the closest thing we get. And there is certainly a push towards adapting VR as the general productive user interface (“Spatial computing” as Apple touts), although the major barrier seems to be the cost vs display quality of the actual hardware, which we can expect to gradually improve over the following years.

Taken the visualization (computer → human) alone, at this stage VR headsets might not necessarily be a superior choice to a GUI window on a traditional flat display. While we could say that VR is likely the future (which depends on hardware becoming cheaper, a quite predictable trend) and particularly suitable for 3D interaction (and further that we would likely want to “dive in” to see closely a small part of a large graph), it would still be a barrier of entry to require a VR headset at all. 

However, VR headsets today also provides immediately usable hand tracking, or just 3D input, which otherwise requires niche equipment on a personal computer. So if we consider both input (human → computer) and visualization (computer → human) together, a VR headset is actually a lower barrier of entry than doing handtracking or other 3D input on a computer without VR. Hence, we would envision our program to be a PCVR application. For the scope of this project, the application targets a Linux desktop environment, simply due to that’s what the team is familiar with. Although, compatibility other operating systems will be taken into consideration in other design choices.

The ability for the user to quickly switch between giving 3D input and typing on a keyboard is also crucial, as we can imagine the user will need to very frequently type in attributes associated with nodes and edges. Hence hand tracking is preferred over substantial controllers, even though the latter still provide more responsive and more accurate input. A potential solution are low profile hand-tracking gloves that have been designed to accommodate handling objects with the gloves on. How compatible they are to keyboarding remains to be tested. The application should be usable with just a modern VR headset (such as Meta Quest 3) and a personal computer, although use of hand-tracking gloves will be incorporated.

Another important choice is the graphics pipeline, or whether to use a video game engine, or to directly program against graphical APIs. Compared with video games, we may need to draw many similar objects on screen very efficiently, while having less variaty over these objects. After some initial research, we think that it may actually be a better idea to directly program against the graphical API (the modern choice being Vulkan), and particularly Vulkan with SDL2, a combination proven to run straight away on Linux. Incidently, we will bridge the MORK represetation and flat arrays (that GPU likes) with toolchain for Graph Neural Networks (that is, pytorch-geometric in this instance), which has the added potential to attract the research community on graph neural networks towards MORK/MeTTa as well.

Open Source Licensing

To be fully open sourced in the same way as other OpenCog libraries.

Background & Experience

Machine learning background related to AGI and mainstream deep learning.

Publications 
https://dblp.org/pid/258/7336.html

Previous work experience in video game graphics. Had done some work in GPGPU computation with shading language when CUDA was not as popular.

Not published but relevant to this project, some work had been done using PyTorch, PyTorch-geometric as well as second order optimization to solve for forces in a cable-stayed truss structure. It was actually quite similar to solving graph embedding/layout.

Links and references

https://link.springer.com/chapter/10.1007/3-540-52698-6_1

https://math.mit.edu/~urschel/publications/p2021a.pdf

https://math.stackexchange.com/questions/3651675/can-all-hypergraphs-be-embedded-in-3d-space

https://docs.unity3d.com/6000.1/Documentation/Manual/shader-graph.html