Contrastive reinforcement learning (CRL) is a popular self-supervised RL algorithm that learns to reach goals without using any external rewards. However, existing results on BuilderBench (Ghugare et al., 2025) tasks show that CRL struggles when scaled to open-ended environments with increasing degrees of freedom. We first propose modified BuilderBench environments that run entirely on the GPU, speeding up training by up to $16 \times$. We then perform a large-scale study of CRL failure modes and design choices on BuilderBench. We find that horizon reduction methods are effective at improving performance for both PPO and CRL, and that horizon reduction implicitly regularizes CRL's contrastive objective. Using these insights, we introduce StableCRL, a tuned implementation of CRL that incorporates regularization techniques such as in-trajectory negative sampling. On both short- and long-horizon tasks, StableCRL can rival or exceed on-policy RL methods such as PPO.