So here's what I have now.
On the far left, is a wide bandpass filter between the first variable gain amp (out of frame to the left, but exactly the same as VGA2 in the middle) and the second VGA.
The variable gain amplifier, LMH6505, is configured for single-supply operation and thus has some biasing resistors in place. Maximum stage voltage gain is currently set to be 17.6dB. (I think. Max gain is 0.940 * (R28 / R27), which comes out to be 17.6dB by my math, however the datasheet calls out this specific combination at 26dB...) The maximum gain can be adjusted from 0 to 100, so I can (and probably should) increase R27 to 100 ohms, and R28 to 2.7K, to bring gain up to 25dB or more. The datasheet seems unclear on what is the "max gain", but mentions that the gain control input should be low-pass filtered.
C1 may not actually be needed.
R29 is fitted as per the LMH6505 datasheet to present an output impedance of 50R.
The RMS-2+ mixer's RF port on pin 4 receives the signal from the variable gain amplifier. Since this mixer will be asked to output down to audio, the IF port on pin 5 will be the audio output in receive mode, and the RF output in transmit mode. It's specified for DC-1000Mhz operation so should be no issue.
The BFO/Transmit Carrier Oscillator is applied to the mixer's LO port through a series resistor and a pi-network pad. 1K is used as a placeholder value, R8 may well be a 0 ohm link and the value of R5, R6, and R7 will be adjusted to match the applied LO signal level. Even though this is a Level 7 mixer, very good performance can be had at a LO level of +4dBm, while reducing LO feedthrough.
Now we get to the sticky bits. Port 5 of the mixer, the IF port, needs that broadband 50R termination for best performance. The AD8307 Log Detector's input impedance is 1.1k, and needs a 52.3 ohm resistor to match to 50R. I happen to have a strip of that value so I've used it here.
But now how to pick off both an audio signal, AND the Transmit RF source, which will range from 3Mhz to 30Mhz? My first thought is a 1Mhz diplexer, sending the below 1Mhz to the audio system, and above 1Mhz to the transmit chain. This also eliminates the need to relay-switch the IF chain output.
The log amp is configured as a power detector with a 25mV/dB output, with the 0 intercept at -84dB. I've used this exact circuit before as a power meter driver and it works very well, and is fairly accurate (good enough for the girls I go out with, anyway!) Power cleanliness is important though, as the circuit is just as sensitive at 60Hz as it is at 600Mhz, and still has some decent response at 1Ghz...and any of these signals will cause the AGC as drawn to respond and reduce the gain of the VGA's. I figure this will give Audio AGC on receive and RF AGC on transmit.
Now, coming out of the log detector is a simple rail-to-rail opamp, which can operate from a single supply voltage no problem. I haven't gotten too far into this yet, as you can see. The log detector output is 25mV/dB, and the VGA's gain control input is...25mV/dB. Perfect. Just needs to do a level shift, and a low-pass filter to have this thing working at a basic level.
If the level shift is configured at unity gain, inverting, then the default state of the VGA should be maximum amplification. (0V output from the log detector should mean +2V output from the opamp, which should give maximum gain from the VGA) I need to level shift it up to a +6V to +8V range, as the 0-2V variable gain input has to be biased up when running from a single supply (or I need to redesign the thing for a +5/-5V supply, which honestly wouldn't hurt things.)
Questions:
What do do about the output match for the RMS-2 mixer?
What to do about trying to make sure that the VGA closest to the mixer will lower it's gain first, then we start lowering gain at the VGA before the noise filter?
How much IF gain do I really need? As drawn, and if I believe the datasheet instead of my math, I'll have 56dB of gain before the mixer, maximum. This seems a bit low. Like half as much as I should need low.