Data from (non-) attenuation of gamma rays from active galactic nuclei (AGN) and gamma ray bursts (GRBs) give upper limits on the extragalactic background light (EBL) from the UV to the mid-IR that are only a little above the lower limits from observed galaxies. These upper limits now rule out some EBL models and purported observations, with improved data likely to provide even stronger constraints. We present EBL calculations both based on multiwavelength observations of thousands of galaxies and also based on semi-analytic models, and show that they are consistent with these lower limits from observed galaxies and with the gamma-ray upper limit constraints. Such comparisons "close the loop" on cosmological galaxy formation models, since they account for all the light, including that from galaxies too faint to see. We compare our results with those of other recent works, and discuss the implications of these new EBL calculations for gamma ray attenuation. Catching a few GRBs with ground-based atmospheric Cherenkov Telescope (ACT) arrays or water Cherenkov detectors could provide important new constraints on the high-redshift star formation history of the universe.