Synapse Open Synapse
AI

Saw this benchmark where small changes to coding tasks made a big difference

Reddit · r/LocalLLaMA ·

Saw this tweet about a coding benchmark and thought it was kinda interesting. Apparently they took 89 DeepSWE tasks and changed little things in the instructions or starting code. The tweet says some models that were all around 74% on DeepSWE ended up 23 percentage points apart on the changed tasks.

Latest from Reddit · r/LocalLLaMA