There are two main approaches to characterising an antenna and its effectiveness in-system, free space
and in-situ
. The first will give you a best-case baseline, is less susceptible to measurement noise (especially if you have access to a large anechoic chamber), and probably works best for selecting between candidate antenna variations. In-situ
measurements are much less definitive, but allow you to be specific about a possible working scenario, and rule out silly mistakes (like forgetting that the final device has a conductive casework).
Both approaches require a transmit-receive pair. Either one or both can be real devices, propagation is commutative. Take care to use representative power supplies and autonomous operation on real test device. If you connect power or serial cables, these become part of the antenna system.
A good alternative to a spectrum analyser is a Software Defined Radio. Certain USB-DVBTV dongles can be used at very low cost. You shouldn't be as concerned about signal fidelity in this scenario, simple signal strength should be enough.
Ideally you will be close enough in that uncorrected error rate is not a useful metric. Independently, you should understand how signal strength, uncorrected error rate and system performance are related to work out what operating point you are aiming for.
For free space measurements, ideally you will be in an open field, free of power lines and reflective structures for many 10s of metres. Place the DUT on a wooden stand, and observe it from a distance of 5-10 metres (several wavelengths at least). Rotate it in all directions, and measure. Repeat with a comparative device, repeat with any other variations you care about. A directional antenna at the frequency of interest is normally used at one end of the test range, this removes some variation due to the test location (and your presence near this end)
For in-situ measurements, it is important to trial as many realistic deployments as you can. Ideally you would run device-to-device, with signal quality being logged on the device. Look for the worst-case scenarios that you can find - end users are likely to find even worse ones.
If in doubt, consult an EMC test facility. They are often able to consult on, for example, validating your approach with some spot measurements. With a custom antenna, you'll probably need their services later anyway.