• mountainriver 2 days ago

    Deep Entropix vibes

    • andrewmonostate a day ago

      Thanks for bringing this up! Good catch on the similarities! Yes, both use entropy/uncertainty to allocate compute intelligently.

      From what I understand, Entropix is an entropy-aware decoder - it monitors token entropy during generation and dynamically adjusts sampling or spawns parallel CoT branches at high-uncertainty points. It's a decoding-time intervention.

      My approach doesn't touch decoding at all. I:

      1. Generate normally (standard sampling)

      2. Capture logprobs + top-k alternatives

      3. Check if perplexity/entropy/confidence triggers exceed thresholds

      4. If yes, do ONE refinement pass with an "uncertainty report" showing the model exactly which tokens were uncertain + their alternatives + context

      The key difference: Entropix steers the ship while sailing; my loop reviews the voyage log and decides whether to make one correction pass. No branching, no custom samplers, deterministic cost (0 or 1 extra pass).

      They're actually complementary - you could use Entropix entropy-aware sampling for initial generation and still apply a refinement loop afterward. Same underlying signal (entropy), different control points! The result of combining both should be outstanding! I will test it soon.