Show HN: A 200-item RLHF quality audit with Fleiss' Kappa (0.68)
RYTHUN — Sample work: quality report for a 200-item RLHF harmlessness preference dataset. 3 annotators, 84% observed agreement, Fleiss' Kappa 0.68, documented adjudication. Methodology and sani...