This paper argues that Japan’s excessive labor market duality can reduce Total Factor Productivity (TFP) due to a negative impact on non-regular workers’ effort and on firms’ incentives to train them. On the basis of cross-country empirical evidence, the paper proposes some reform options. In particular, our analysis suggests that reducing the difference in employment protection between regular and non-regular workers would substantially reduce labor market duality in Japan. One reform consistent with these findings is the introduction of a Single Open Ended Contract for all newly hired workers. This reform could be complemented by a shift towards a model that combines labor market flexibility and security (“flexicurity”) and by policies aimed at encouraging wage growth.