As I've been discovering the Classic Traveller rules have a little to say on everything. So with a little tweaking they can simulate anything. One of my favorite game mechanics was Chaosium's Sanity Check, prevalent in Call of Cthulhu. Even in a hard SF setting there may be aliens that are just to darn alien for us to abide or maybe H.P. Lovecraft was right and we'll run into Mi-Gos or their friends out there.
I think the Morale rules from LBB 1 stand in quite well for Sanity checks. Deep Ones, Werewolves and humanoid monsters might invite a standard throw for Morale and maybe not till they revealed their true nature. Most Vargr and Irklan don't eat people they have just met. The more exotic races like the Mi-Go or Flying Polyps would warrant a -1 or -2 when first encountered. The more mind blasting cosmic horrors would start at -3.
Failing a Morale throw not only means the character runs but would temporarily reduce their Int, Edu or Soc by one or more dice. The first throw applies the entire reduction to one characteristic (sound familiar?) The dice of damage would equal the minus to moral throws give or take a die.
You could impose some positive modifiers to give the poor characters more of a chance than Call of Cthulhu character (you thought dying in character generation was rough, hah!) Combat drug (or liquor) could give a +1 or +2 to Morale throws. Being in a large group could add +1. It's the ref's call on what a large group is but I'd put it at five times the negative modifier the creature imposes. Being under cover or concealment would give +1 at Short range, +2 at Medium etc. Scouts should get a +1 because they get paid to deal with weird stuff.
If your character blows the morale throw and their characteristics are reduced you might want to give them the following penalties:
Int 3 or less: you are unaware of anything beyond your immediate surroundings and will run into a wall or obstacle or attack a friend on +9.
Edu 3 or less: Your responses are instinctual. You will use a gun as a club or try to batter down a door with your head.
Soc 3 or less: You disgrace yourself with your show of fear. Your Social Standing is reduced by 3 until you find a way to redeem yourself (usually by facing a similar threat and not needing to dry clean your reflec afterwards.)
Note that if a character starts with a three or less in any of these stats they do not suffer any negative effects until it is reduced by one or two.
A character who has one characteristic reduced to zero has a breakdown and will be good for nothing for at least a day. A character with two characteristics is not only unstable but dangerous to themselves and others. A character with all three characteristics reduced to zero is either catatonic or in need of restraints.
The characteristics return to normal at a rate of one point a day per characteristic if one characteristic is reduced to zero, at a rate of one point a week per characteristic if two characteristics are reduced to zero and at one point a month per characteristic if all three are reduced to zero. In addition characters with all three characteristics reduced to zero need 1-6 months of therapy at Cr. 5000 per month.