Yep! In each game's directory there is a file called "winsetup.exe". Run that file. You'll see a checkbox labelled "run in windowed mode." Tick that box and select "save" (or "save and run game"). Voila! Windowed mode.
Be sure that the game's resolution is set to at least 640 x 480, or the window will be very very small.