Polymer field theory has emerged as a powerful tool for describing the equilibrium phase behavior of complex polymer formulations, particularly when one is interested in the thermodynamics of dense polymer melts and solutions where the polymer chains can be accurately described using Gaussian models. However, there are many systems of interest where polymer field theory cannot be applied in such a straightforward manner, such as polymer nanocomposites. Current approaches for incorporating nanoparticles have been restricted to the mean-field level and often require approximations where it is unclear how to improve their accuracy. In this paper, we present a unified framework that enables the description of polymer nanocomposites using a field theoretic approach. This method enables straightforward simulations of the fully fluctuating field theory for polymer formulations containing spherical or anisotropic nanoparticles. We demonstrate our approach captures the correlations between particle positions, present results for spherical and cylindrical nanoparticles, and we explore the effect of the numerical parameters on the performance of our approach.