In 1627, a couple of decades after the expulsion of the Moriscos, a corsair ship from Algiers raided Iceland, capturing a couple of hundred unfortunate villagers, one of whom left a description of his experiences. While the distance travelled in this raid was unusual, the practice itself was less so: the capitals of the Barbary states were full of European slaves captured by state-sponsored pirates, waiting for ransoms that might never come. Likewise, many North Africans were captured and held as slaves in Europe (see eg Wettinger 2002 on Malta): describing Algiers in 1612, Diego de Haedo comments that "there are many Muslims who have been captives in Spain, Italy and France" and hence speak those countries' languages (Vincent 2004:107). To further complicate matters, not all immigration from Europe was involuntary: Haedo adds that "There are also an infinite number of renegades [converts to Islam] from these countries and a large number of Jews who have been there, who speak polished Spanish, French, or Italian. The same holds for all the children of renegades who, having learned their national language from their parents, speak it as well as those born in Spain or in Italy."
In brief, 17th-century North Africa contained plenty of European immigrants - some refugees, some captives, and even some voluntary - learning the language spoken around them while maintaining, for a while, the language they had arrived with. What impact did this have on Maghrebi Arabic and Berber? Unfortunately, it's not easy to date Romance loans into either, but we can safely assume that some of the precolonial loans arrived in this period. A good dialect map, in combination with historical data on where these groups ended up, might help identify such loans more precisely - but that doesn't really exist yet, except to some extent for Morocco (Heath 2002).
Vincent, Bernard. 2004. In Jocelyne Dakhlia ed., Trames de langues. Usages et métissages linguistiques dans l’histoire du Maghreb, Tunis-Paris, IRMC, Maisonneuve & Larose, 2004, 561 p.