Like Ben said, you just need to tap into the O2 sensor voltage output with the positive lead and ground the negative lead. But That being said, in your case, it probably won't be useful to you because Im assuming that you have a standard O2 sensor, 1-4 wire. A standard sensor has a range of 1 volt and is really just binary cycling between rich and lean, 0.1 and 0.9V. I'm not sure what you will be able to see with a DVM. Auto techs use an osciloscope.
In my case I have a LAF Sensor aka wideband which has a 0-5V LINEAR output. The output would be much more informative, especially in a vehicle that will purposefully burn lean. I theorize that I should be able to detect lean burn with the DVM by seeing a constant lean condition rather than the normal oscillation between rich and lean.