"None of my cables have the shell of the neutrix attached to pin 1 ground though. Shouldn't matter though??"
Yes, it may matter very much in your case. Test the same setup you are using, except for a hard ground on both connector ends: Connect the cable's shield AND cable ground wire* to pin 1 on both connectors, as well as making a wire connection from pin 1 to the connector housing. Then please report back!
*I assume you are using Gotham GAC3 or genuine Neumann IC3 cables which both have a dedicated ground return wire, in addition to the cable's shield. No need to even try this with a cable other than the aforementioned- I found it hard to tame RF with cheap cables that lack the double Reussen layer of shielding of Gotham and Neumann cables.