I find an amazing proof for this speculation in the kingdom that ruled Tulu region in the past, Alupas. Though Alupa or Aluva is a native Dravidian word (literally, ruler)* they claimed to belong to "Pandayvamsa"(Pandya Dynasty). And according to Wikipedia:
Their coins carried the dynastic title "Sri Pandya Dhananjaya" which means "Arjuna among the Pandyas".
It appears one branch of matrilineal dynasty of Alupas(The dynasty also ruled Kodava region too...and was speculated to be matrilineal there too...or Kodavas were also believed to be matrilineal initially, M N Srinivas) became patrilineal Pandyas in Tamil region.
In my opinion, by 350-300BCE, based on Megasthenes description, a matrilineal dynasty of Tulu women was created in Tulu region.
*Plausibly wrong. It is pronounced as alUpa and not ALupa, where "ALu" means 'rule' in Tulu/Kannada. No idea about etymolgoy of "alUpa".