Great question! Not really my area of expertise, but probably there are at least a couple of possible avenues. One is decompilation and/or disassembly and static analysis. (Basically use automated tools to reconstruct the original source code as best it can and then read that imperfect reconstruction of the source code to figure out what it does.) Another is isolating it (“air gap” – no network or connectivity to anything you care about) so you’re sure it can’t do any damage and running it with tools that record/report everything it does. (On Linux, one could use strace and/or GDB. On Mac, dtrace. Not sure what the equivalent is for Windows programs running on Windows.)

Actually, I guess another option could be to set up an isolated system, record a whole bunch of information about it before running the .exe then after running the .exe, examine it to see what you can find on the filesystem or in the registry or in RAM or whatever that might have changed. It wouldn’t catch everything, though. Like if it made a network connection or something but didn’t actually change anything on the filesystem, it might not leave any traces.

Whatever the case, it’d probably require some specialized tools and expertise. But it’d be an interesting project.