Is there a way to determine how much space a package will take up in an image before compiling the image? I have a router with 4MB flash and I'm right on the borderline each time I compile it with it being too large. Is there a way in menuconfig or somewhere else to determine how much space a package will consume upon compilation?

No, you can't calculate the size requirements before compiling and mksquashing - especially not exact enough to predict hit or miss on a tiny margin (sizes may even vary slightly between builds of the same code, due to the order the binaries get inside the squashfs, timestamps, etc.). All you can do is making a rough guess and just trying, but the closer you get to the limit, the harder it gets to predict the result (keep in mind that you also need a few free blocks to store your configuration).

I'm not sure, but i think you could install packages to the RAM with "-dest ram" switch and look the /tmp folder for determine the size.

Of course you can do that, but mksquashfs for a 4-8 MB filesystem images takes a couple of seconds, which means it makes virtually no difference to just create the whole firmware image properly. What takes time, is compiling toolchain and cross compiling the packages - and before you've done that, any size estimates are just (more or less informed) guesswork.