Igboho had earlier warned of impending doom if the herders did not leave Ibarapa by Friday.
Seriki of the Fulanis in Ibarapa, Oyo State, Saliu Abdukadir, has claimed the Fulanis are peaceful people who obey the government and not criminals.
He disclosed this in an interview with the BBC while responding to the Yoruba freedom fighter's vacation order, Sunday Adeyemo, better known as Sunday Igboho, to herders in Oyo State.
Igboho had earlier warned of impending doom if the herders did not leave Ibarapa by Friday.
He claimed that the Fulani herders have more sophisticated weapons than the Western Nigeria Security Network's operatives, codenamed Amotekun.
Reacting to the order, the Seriki dismissed a claim that the Fulanis working with him are the kidnappers and murderers terrorising the area.
Buttressing his point, the monarch said he and his people are also victims of the attacks.
He, however, admitted that some of the arrested kidnappers are indeed Fulanis.
He said, "When Igboho came here, his men were shooting, he broke this man's hand. I've been here for over 45 years (crying).
"Truly, some of these kidnappers are Fulani, but when they arrest some and ask them where they had come from, they will say Zamfara, Kebbi. We are also victims. They take our cows and us; we just paid the ransom for one of our people."
He alleged that the Yoruba leaders had not looked inwards to solve insecurity in the state.
"What they are supposed to tackle is not yet tackled. At Igangan there, scavengers are plenty there, some of them are not scavengers but thieves.
"We don't go against the government; we do their bidding."
In an earlier report, Igboho had alleged that Fulani people in Ibarapa are killing people.
He said, "Look at this Doctor Fatai, who went abroad for many years to work. He made his money, came back to his father's village in Igangan, built farms and employed people to work with him. Fulani people ruined the farm for him. He approached the Seriki of the Fulani that this ought not to be, and they tied him and butchered him.
"There are many others like that; the Fulani people killed and kidnapped some. The least they accept as ransom is N7 million, N10 million, N11 million on our soil. This hurts me. I said if I should keep quiet because it has not got to me or because I have those who are guarding me, how about people who do not have any form of protection?
"That was why I went there to ask the Seriki if this thing his men are doing is good. They have been living with us on our soil for years, and we accommodate them. I talked to the Seriki, and one of his men pointed a gun at me. I collected the gun. Another one brought out an axe; we collected it. Another one brought out another gun and shot at me. I told them I was not there to fight but to tell them that if they would not find a lasting solution to the problem of kidnapping and killing, they should leave our land."