You probably have an ejector designed for the 9mm, which is angled inward more than the ejector for the 40S&W. If you want to go back and forth, either use the ejector for the 40S&W or buy another trigger housing with ejector and switch them when you switch calibers.
9mm - ejector only is part #336
40S&W - ejector only is part #1882
if you look at the right side of the ejector, you should see the part number stamped on it.
9mm - ejector only is part #336
40S&W - ejector only is part #1882
if you look at the right side of the ejector, you should see the part number stamped on it.