Most membrane and secreted proteins are glycosylated on certain asparagine (N) residues in the endoplasmic reticulum (ER), which is crucial for their correct folding and function. Protein folding is a fundamentally inefficient and error-prone process that can be easily interfered by genetic mutations, stochastic cellular events, and environmental stresses. Because misfolded proteins not only lead to functional deficiency but also produce gain-of-function cellular toxicity, eukaryotic organisms have evolved highly conserved ER-mediated protein quality control (ERQC) mechanisms to monitor protein folding, retain and repair incompletely folded or misfolded proteins, or remove terminally misfolded proteins via a unique ER-associated degradation (ERAD) mechanism. A crucial event that terminates futile refolding attempts of a misfolded glycoprotein and diverts it into the ERAD pathway is executed by removal of certain terminal α1,2-mannose (Man) residues of their N-glycans. Earlier studies were centered around an ER-type α1,2-mannosidase that specifically cleaves the terminal α1,2Man residue from the B-branch of the three-branched N-linked Man₉GlcNAc₂ (GlcNAc for N-acetylglucosamine) glycan, but recent investigations revealed that the signal that marks a terminally misfolded glycoprotein for ERAD is an N-glycan with an exposed α1,6Man residue generated by members of a unique folding-sensitive α1,2-mannosidase family known as ER-degradation enhancing α-mannosidase-like proteins (EDEMs). This review provides a historical recount of major discoveries that led to our current understanding on the role of demannosylating N-glycans in sentencing irreparable misfolded glycoproteins into ERAD. It also discusses conserved and distinct features of the demannosylation processes of the ERAD systems of yeast, mammals, and plants.