OK, It's the O/P from the LTC, so if you set the carrier to less than 50% the bottom of the sine will clip.
Sadly, you didn't DC couple your 'scope on the output waveform. If you do it will be obvious (? lol), or rather, its obvious to me after many years of faffing around with this crap that the modulator cannot ever output NEGATIVE DC (which is what you would require for *symmetrical* mod in the 60/40 case. Even if it could, your PA transistors won't thank you for supplying negative drain volts!)
It looks to me as if your circuit has the expected behaviour -- you say its not 100% positive modulated @ 60:40 because you are not increasing the input once you see the negative peaks clipping -- or are you?
If peak supply is 40V, and you set carrier @20V, now you will see "symmetrical" modulation -- and you do, right?
So now, for argument's sake (silly example) set the resting carrier voltage to 10V. The modulator will clip the neg peaks very soon (coz 0v is only 10V away) but the positive peaks should not clip until they hit 40V. It will "overmodulate" like mad in the negative direction (just to prove the point), but increasing drive should increase the pos peaks til they get to the supply volts, minus a little tiny bit.
Is this what you are seeing? If it is, then its expected behaviour, and its your outboard audio processing (or make a NEGATIVE ONLY peak clipper) to smooth off the negs just before the modulator hits 0v.
J, (is it John?) am I on the same page as you. Perhaps I missed something but I did take the time to look a couple of times at your posted images.
Summary: if you offset the carrier for more than 100% peak positive modulation, your audio chain has to take care to avoid the carrier getting shut off on the negative peaks. I do this with a peak limiter that only works on the neg peaks. Some people connect a diode to the drains from a very low voltage constant source. It still clips in the negatve direction on a sine, but the carrier can't shut off due to the diode conducting 2 or 3 volts to the drains when the modulator pinches carrier.
Anyhow, I hope you get some more responses because I have what I think should be a fairly common and logical statement on here for three weeks, and not one bugger has replied!
BTW: if you do have a modulator that takes care of this asymmetrical modulation automatically, I want to know about it :-)
Steve