the prophet peace be upon him did not wield a sword with which to force people to enter islam, for the qur'an is crystal clear in its refusal of this concept. almighty allah has said:
there shall be no compulsion in [acceptance of] the religion. (2:256) then, would you compel the people in order that they become believers? (10:99) for you isyour religion, and for me is my religion. (109:6) but this does not mean that the state should remain powerless in the face of internal or external aggression, for almighty allah has given the believers permission to defend themselves and to restore their rights in proportion to the injustice done to them, but not more than that.
and almighty allah, said:
so whoever has assaulted you, then assault him in the same way that he has assaulted you. (2:194) fight in the way of allah those who fight you but do not transgress. (2:190) but if they fight you, then kill them. (2:191)
this shows that the legality of war is confined to self defence and protection of the ummah from aggression and conspiracies from within and from without. when we look at the history of armed conflict in islam, this fact is confirmed. for when the makkans' oppression increased, they compelled the prophet peace be upon him to leave his home after having plotted to kill him, and it was they who began aggression against the muslims by unjustly forcing them to leave their homes. so, after the hijrah, almighty allah permitted the emigrants to fight the polytheists of quraysh, saying: permission [to retaliate] has been given to those who are being fought, because they were wronged, and indeed, allah is competent to give them victory. [they are] those who have been evicted from their homes without right – only because they say, "our lord is allah." (22:39-40) thus, the prophet peace be upon him was first to oppose only the quraysh and not the other arabs.
but when other arab polytheists besides the people of makkah turned against the muslims and allied themselves with their enemies, almighty allah ordered them to fight all of the polytheists, and said:
and fight against the disbelievers collectively as they fight against you collectively. (9:36)
this is how jihad became general against all of the pagans who had no divine scripture, and it is the affirmation of the prophet's peace be upon him saying,
"i have been commanded to fight the people until they say, 'la ilaha ill-allah', so if they say it their lives and properties are protected from me by its right, and their account is with allah." (al-bukhari and muslim)
and when the muslims found betrayal from the jews after they broke the treaty and aided the polytheists in their war, almighty allah ordered fighting against them also, saying:
if you fear from a people betrayal, throw [their treaty] back to them, [putting you] on equal terms. indeed, allah does not like traitors. (8:5) fighting them became obligatory until they offered the jizyah
tax in submission so that muslims would be safe from them.
similarly, the prophet peace be upon him did not begin hostilities with the christians. shaykh al-islam ibn taymiyyah said, "he did not fight against any of the christians until after he sent his messengers, following the treaty of hudaybiyyah, to all the rulers, inviting them to islam. he sent them to caesar, chosroes, al-muqawqas, an-najashi and the arab rulers to the east and the north. some of them entered islam, but the christians of syria deliberately killed some of their dignitaries who had embraced islam. so, the christians had first fought the muslims and then killed those of their people who accepted islam out of tyranny and injustice. otherwise, the prophet pbuh had sent his messengers to invite people to islam voluntarily, not forcefully. he pbuh never compelled anyone to accept islam."
thus, the battles of allah's messenger peace be upon him were based on the following principles:
the polytheists of quraysh were considered enemy combatants because they initiated aggression.
the jews were observed to have betrayed and gone over to the side of the polytheists.
any arab tribe that transgressed against the muslims was to be fought until it accepted islam.
jews and christians who committed aggression were to be fought until they accepted islam or paid the jizyah tax.
anyone who enters islam has protected his life and property, and islam cancels what occurred before it.
nur al-yaqeen, p. 84-85.
qa'idah mukhtasirah fi qitaal al-kuffar wa muhadanatihim, p. 135-136.