Difflogic should lead to very efficient hardware implementations of inference for a wide range of problems. But is it going to allow to generate the logical chains that elude current LLMs?
--- edit ---
After reading the paper more thoroughly, I find the way they implement differentiable logic clever. They use continuous relaxations of 16 logic operators, run them in parallel and apply a softmax to select the most useful operator. At inference time, everything is binarized.
The original cited paper https://arxiv.org/abs/2210.08277 "Deep Differentiable Logic Gate Networks" struck me then as very clever, ultra-efficient/small, hardly any inductive bias or prior... But you have to wonder if it's able to scale reasonably well. Differentiating continuous versions of 16 discrete operations in parallel sounds expensive, especially since you presumably need a bunch of them chained in order to approximate a single neural primitive. Even distilling an existing LLM down might be too hard.
It's definitely slept on. I do think it ought to be very powerful given enough compute to throw at it, hopefully. I think short description length algorithms such as simple compression algorithms or instant-ngp could be interesting to play with through that paradigm.
This is fascinating—I've recently been experimenting with something spookily similar after coming across the same paper as OP, only I'm trying to make random DAGs of logic gates reconstruct images rather than audio signals. Great minds like a think, I guess :P
Got anything written up and/or code on github?
Cool concept and it seems weirdly effective. It would certainly help us lazy people to have the web version hosted somewhere, I'm assuming it could be a static site, since the meat of it is in WASM, right?
Would this be amenable to "morphing between presets", or even manually combining a selection from one network into another network? Lots of things to try out here!
Most interesting! :D
Logic gates implemented with non-ideal transistors have non-zero rise times. Therefore, they are smooth and differentiable.