Elastoplastic lattice models for the response of solids to large-scale deformation typically incorporate structure only implicitly via a local yield strain that is assigned to each site. However, the local yield strain can change in response to a nearby or even distant plastic event in the system. This interplay is key to understanding phenomena such as avalanches in which one plastic event can trigger another, leading to a cascade of events, but is typically neglected in elastoplastic models. To include the interplay one could calculate the local yield strain for a given particulate system and follow its evolution, but this is expensive and requires knowledge of particle interactions that aren’t necessarily pairwise additive or possible to extract from experiments. Instead, we use a structural quantity, “softness,” obtained using machine learning to correlate with imminent plastic rearrangements. We show that softness correlates with local yield strain and use it to construct a “structuro-elasto-plasticity” model that reproduces particle simulation results reasonably well for several observable quantities, confirming that we capture the influence of the interplay of local structure, plasticity, and elasticity on material response.