Understanding and controlling materials’ resistance to fracture is critical for various applications. However, the structural origin of toughness, brittleness, and ductility remains poorly understood. Here, based on the experimental testing and atomistic simulations of a series of aluminosilicate glasses with varying thermal and pressure histories, we investigate the role of structure in controlling fracture toughness at fixed composition. We show that fracture toughness decreases with density, but strongly depends on the details of the temperature and pressure histories of the glass. This behavior is found to arise from a loss of nano-ductility rather than a loss of cohesion. Finally, we demonstrate that the propensity for nano-ductility is primarily controlled by the extent of angular flexibility between the rigid polytopes of the network. Tuning the extent of nano-ductility in silicate glasses would permit the design of ultra-tough glasses.