Proposed 9.4B Parameter Model Uses Engram Tables and AttnRes
September 13, 2026
A developer is training a 9.4b parameter dense model utilizing Moonshot's AttnRes modeling, Engram tables, and 3:1 RoPE/NoPE layering. The training utilizes logit-level extraction from a Llama 3 teaching model to improve capability over existing 9b models.
HOW THIS AFFECTS YOU
●
researcherYou should monitor the performance of these architectural modifications in small-scale dense models.