Cavitation within soft, elastic solids is not well understood despite the role it plays in the failure of materials important to many commercial and biological applications. We examine the elastic properties and cavitation behavior of polymeric materials as governed by the network structure through coarse-grained molecular dynamics simulations, and the model captures the cavitation process as described by continuum models. Inspecting local displacements reveals an anharmonic distribution of nonaffine displacements, which takes the form of a Gaussian core and an exponential tail. This form is present even for highly idealized defect-free networks, but the non-Gaussian character is more pronounced in the realistic networks. We find that the deviations of the network junctions from affine behavior have a strong radial dependence on the distance to the cavity site. We quantify the set of nonaffine displacements to reveal substantial correlations in the network response across independent cavitation simulations of the same network, suggesting that permanent inhomogeneities in the network play a role in determining the response during cavitation.