Control over the nanoscale distribution of light fields in the optical spectral range is of great importance in many technological fields, including photovoltaics, photocatalysis, and molecular sensing. A unique way to study optical near fields at very high spatial resolution is by cathodoluminescence (CL) spectroscopy in which a high-energy (1-30 keV) electron polarizes a material and the emitted optical radiation is analyzed. The spatial distribution of the CL signal measured in a scanning electron microscope (SEM) provides a two-dimensional map of the nanophotonic density of states at nanoscale spatial resolution. We present a SEM-CL instrument that analyses the spectrum, polarization, time-dependence and photon statistics of the CL emission. We use CL to excite plasmonic metamaterials and control their emission characteristics, and use CL holography to reveal the phase distribution of plasmonic scattering wavefronts. We show how time-resolved CL reveals the emission statistics of optical emitters and demonstrate nanoscale measurements of temperature and thermal conductivity. Using pump-probe CL spectroscopy we study the emission dynamics of NV centers in diamond with electron pulses as pump and laser pulses as probes, to prepare and read out the NV states. The experimental data are explained with a model considering carrier dynamics (0.8 ns), NV0 spontaneous emission (20 ns), and NV0 → NV− back transfer (500 ms). The results provide new insights into the NV− → NV0 conversion dynamics and show how electrons control the transfer from NV- to NV0.