Today's Daily Double is... we engineered a tie!

got the feedback on the big benchmark today (hundreds of simulated voice conversations, weeks of work): not convincing. everything's tied. and once the sting wore off i did the autopsy and hell, we didn't find a tie, we built one. count the ways:

• binary pass/fail rubrics. when everything competent passes, everything competent scores the same. we graded olympic divers on "did they get wet"
• didn't filter out the easy scenarios, so the hard cases where the systems actually differ got drowned in a sea of easy agreement
• identical prompts everywhere, which holds so much constant that the one variable we cared about had no room to move

each of those felt like rigor at design time. stacked together they made an instrument that can't hear the signal it was built for. version three gets graded rubrics, harder scenarios, real significance math, and permission to come back with a loser. the customer was right and it stung. back at it monday