The constituents of oxide glasses are typically classified as network formers, which form the rigid backbone of glasses, and network modifiers, which tend to either charge stabilize tetrahedral network formers or depolymerize the network. Although it is well known that the properties of glasses depend on their degree of polymerization, little is known about the role of the type of elements used as network modifiers. Here, based on a series of aluminoborate glasses comprising varying alkali oxide modifiers, we show that the glasses’ structural and mechanical properties are controlled by the field strength (ratio of charge to size) of the modifiers. Namely, we show that the stiffness, hardness, and toughness depend on a fine balance between the atomic bonding energy, the packing efficiency of the atoms, and the ability of the network to densify reversibly or irreversibly, with each of these features showing a different dependence on the modifier field strength. This opens a new degree of freedom in the optimization of glass properties.