So the ability and intent to blow-up an apartment building full of people does NOT meet YOUR 'standard' for being terrorism, eh? How is that? What would it have taken for you to "take em serious"? And how can you say there are so few of them?
This item puts the number of active members of the various armed militia groups in this country at 40,000, and this was before Trump started his run for president:
http://www.quora.com/How-many-militia-members-are-there-in-the-US
As for the KKK, the Southern Poverty Law Center puts that number at something like 5,000 to 8,000:
http://www.attn.com/stories/6255/how-big-kkk-is-2016
So how many so-called Black Panthers do you think there are (please keep in mind that the 'Black Panther Party' was dissolved in 1982)? Note that I couldn't find anything that gave any sort of estimate of their membership that was more recent than from the late 60's.

OCU