Read a page of text, then read it again to see if it changed. This test never seems to fail, at least in the sort of dreams I can remember having after waking up. Usually I can't read at all in dreams, and when I can, the text is different every time I read it.
I thought this was just me! There have been times when I've "read" in dreams (signs, usually), but I don't actually visualize the words. I just "know" what they say.
We're assuming in this scenario that your memory is still basically functional, so the math on a sheet of paper won't be replaced suddenly without you noticing.
If you can't even remember things in the medium term that's a level beyond "trapped in a dream" that's much more hopeless.
So, with that assumption, you make a computer solve a problem that's impossible for a human to work through in a practical amount of time. And then you verify it got the right answer. This proves the math wasn't done by your dreaming brain. (At least it proves it to a pretty good certainty, and you can repeat the test.)
Many NP-complete problems are good candidates here for slow solutions and fast verifications.
Yeah I kept thinking about simulation stuff too but GP's method is a good one for being trapped in one's own mind situation.
Also for situations that have the standard dream level of fidelity, you can try basic reality checks like putting your finger through your hand or more conspicuously spinning along your own axis.
