The purpose of this paper is to examine the effectiveness of elementary school English education programs (ESEEP) first implemented in the 2000s. I examined this issue from an evidence-based policy approach, rather than a second language acquisition perspective. In other words, I employed a research design that ensured internal validity and external validity, both of which are a prerequisite for obtaining reliable evidence although they have generally been disregarded by the previous work on ESEEP. Based on this design, I examined whether the implementation of ESEEP has achieved its stated goals, by analyzing questionnaire data from nationally representative 2,967 junior high school students (provided by an open data archive). Based on structural equation modeling, I estimated causal effects of respondents' ESEEP experience on their English language skills, intercultural awareness, and positive attitudes toward English learning. The result suggests that the ESEEP experience did not have any substantial impact. This result, I argue, can be considered as counter-evidence against the current policy reform on ESEEP, and thus suggest that the government, and pro-ESEEP researchers as well, should provide further evidence that indicates the effectiveness of ESEEP if they attempt to maintain the policy.