What exciting architectural moves are we looking to see next in AI models?
We know from all the major chinese labs the current cutting edge architectures such as: conditional engram memory, hyper connections, attention residuals, sparse attention, etc...