You see, Tadaksahak speakers are currently mostly bilingual in Tuareg, and well integrated into Tuareg culture. Most of the Berber loanwords in Tadaksahak are from one or another Tuareg variety. But quite a few – including some of those irregular causatives and most numerals up to 20 - are demonstrably not from Tuareg, but from some other Berber language, closely related to Tetserrét (Niger). Today, Tetserrét is nearly extinct, and nobody speaks it as a second language; obviously things must have been different in the past. It looks like most Tadaksahak speakers are visibly of Berber descent, so probably they shifted from Tetserrét to Northern Songhay and then came under Tuareg influence. But why would anyone want to adopt Northern Songhay, currently barely hanging on in one or two remote towns of northern Niger, as a first language? Again, obviously things must have been different, but it's not easy to see how. My best guess for the moment is that they did so in order to reinforce their identity as religious specialists (ineslemen, "marabouts"), since Songhay was the language of the urban centres where advanced religious studies could be pursued, but there are a lot of question marks over that. To confuse matters further, their neighbours like to claim that Tadaksahak speakers are of Jewish descent - probably just to undermine their religious specialist status, but possibly reflecting some more complex history.
Oral tradition isn't much help; there is no firm consensus within the group on their history, and such genealogies as have been circulated, by themselves or by their neighbours, look very much like efforts to push self-serving agendas. About the only common theme across them is that they came from the west. Genetic testing might give firmer data, but the results could be politically sensitive. More lexical data, both for Tadaksahak and for other minority languages of the region, would certainly help, but the problem is ultimately cross-disciplinary - historians, archeologists, anthropologists, etc. take note! Any ideas?