Hello Aaron,
For the moment you cannot select just some people from the watchers list when you initiate the audio conference. You can create a temporary Topic specific for the subject you would be discussing in the call, and invite only the interested people. Alternatively, you can nominate in the original Topic who is expected to join the call, so that the people who are not required to attend can continue with their work as usual.
Hope this helps!