New sparse attention method for ultra-long context
I am on the team behind the attached open-source research on novel approaches to attention - specifically we show: • You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy • Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is…