policy: P(action=1)=0.5; reward=action; derivative is wrt logit
exact_gradient=0.250000
N=1 batches=2 plain=0.250000 batch_mean=0.000000 leave_one_out=undefined
N=2 batches=4 plain=0.250000 batch_mean=0.125000 leave_one_out=0.250000
N=4 batches=16 plain=0.250000 batch_mean=0.187500 leave_one_out=0.250000
N=2 variance: plain=0.031250 leave_one_out=0.062500 constant_0.5=0.000000
copied_samples_N=2: leave_one_out=0.000000
PASS 34 exact checks; independent one-step model, no RLHF training
