Thats what happened to me. Ate a few ants while grinding skills. Bought Hunting to get Leatherworking. Went to raid Hill. Get knocked out after they finally attack...
They only attack you after you can initiate combat with ants (with hunting you finally can).
Tested this on an alt.
loftar: The inner chaos of the Jorbian mind is hard to conceal. jorb: It's called creative license. You know, that thing you seem to want to apply to logic, grammar and coherence?