Sweet corn was originally due to the recessive allele sugary1 (su1). Sweet corn breeders frequently use field corn genotypes for broadening the narrow genetic base of sweet corn but they have to deal with the reduced viability of su1 plants within some field corn genetic backgrounds. Emergence and seedling vigor are the most critical traits affecting the viability of su1 plants. In two populations of RILs involving sweet corn inbred lines developed from B73 × P39 and B73 × IL14 h, a net natural selection was revealed acting against the su1 allele. In the field, 27 QTLs were detected for the RILs released from B73 × P39 and 24 QTLs for those from B73 × IL14 h, while different numbers of QTLs were detected in growth chamber trials, depending on the seed origin and evaluation conditions that were not consistent across genotypes or environments. The viability of su1 is under genetic and environmental controls with significant additive effects that are probably due to multiple genes with minor contribution. There are specific genes involved in the mutant viability and these genes depend not only on the mutant and the environment but also on the genetic background into which the mutant is introduced. Some of the QTLs identified in this study explained large proportions of variance and could be used by sweet corn breeders in breeding new genotypes from field × sweet corn crosses.