Though the Pope is the successor of Peter par excellance followed by Alexandria and Antioch, all bishops hold the place of Peter, too and this is reflected in a variety of patristic sources. If you know of additional quotations related to this, please post them below and we will add them.
St. Cyprian of Carthage
“Our Lord, whose precepts and admonitions we ought to observe, describing the honor of a bishop and the order of His Church, speaks in the Gospel, and says to Peter: “I say unto thee, That thou art Peter, and upon this rock will I build my Church; and the gates of hell shall not prevail against it. And I will give unto thee the keys of the kingdom of heaven: and whatsoever thou shalt bind on earth shall be bound in heaven: and whatsoever thou shall loose on earth shall be loosed in heaven.” Thence, through the changes of times and successions, the ordering of bishops and the plan of the Church flow onwards; so that the Church is founded upon the bishops, and every act of the Church is controlled by these same rulers. Since this, then, is founded on the divine law, I marvel that some, with daring temerity, have chosen to write to me as if they wrote in the name of the Church; when the Church is established in the bishop and the clergy, and all who stand fast in the faith.”
“Although it may seem that this power of loosing and binding was given by the Lord only to Peter, we must nevertheless know without any doubt that it was given to the other apostles…Indeed even now the same office is committed to the whole Church in her bishops and priests.”
~Homilies on the Gospels: Book I: Advent to Lent, Hom. I.20, p. 202.
St. Isidore of Seville
“So Peter first received the power of binding and loosing, and he first led people to faith by the power of his preaching. Still, the other Apostles have been made equal with Peter in a fellowship of dignity and power. They also, having been sent out into all the world, preached the Gospel. Having descended from these apostles, the bishops have succeeded them, and through all the world they have been established in the seats of the apostles”
~De Ecclesiasticus, II.5, M.P.L., Vol. 83, Col. 781-782.
St. John Chrysostom
Commenting on the words “feed my lambs, feed my sheep” (which Rome wishes to apply only to St. Peter): “This was not said to the Apostles and bishops only, but also to each one of us, however humble, to whom has been committed the care of the flock.”
~Homilies on St. Matthew, 77th homily
“In speaking of St. Peter, the recollection of another Peter [Flavian, Bishop of Antioch, at the time the discourse was written,] has come to me, the common father and teacher, who has inherited his prowess, and also obtained his chair. For this is the one great privilege of our city, Antioch, that it received the leader of the apostles as its teacher in the beginning. For it was right that she who was first adorned with the name of Christians, before the whole world, should receive the first of the apostles as her pastor. But though we received him as teacher, we did not retain him to the end, but gave him up to royal Rome. Or rather we did retain him to the end, for though we do not retain the body of Peter, we do retain the faith of Peter, and retaining the faith of Peter we have Peter”
~On the Inscription of the Acts
Speaking to friend Basil (not to be confused with St. Basil the Great) who had just been consecrated a bishop, St. John Chrysostom says:
“What advantage, pray, could be greater than to be seen doing those things which Christ with his own lips declared to be proofs of love to Himself? John 21:15-17 For addressing the leader of the apostles He said, Peter, do you love me? and when he confessed that he did, the Lord added, if you love me tend my sheep. The Master asked the disciple if He was loved by him, not in order to get information (how should He who penetrates the hearts of all men?), but in order to teach us how great an interest He takes in the superintendence of these sheep. This being plain, it will likewise be manifest that a great and unspeakable reward will be reserved for him whose labors are concerned with these sheep, upon which Christ places such a high value…For what purpose did He shed His blood? It was that He might win these sheep which He entrusted to Peter and his successors. Naturally then did Christ say, Who then is the faithful and wise servant, whom his lord shall make ruler over His household…Observe at any rate how great the reward is — He will appoint him, He says, ruler over all His goods. Matthew 24:47 2. Will you, then, still contend that you were not rightly deceived, when you are about to superintend the things which belong to God, and are doing that which when Peter did the Lord said he should be able to surpass the rest of the apostles, for His words were, Peter, do you love me more than these? Yet He might have said to him, If you love me practice fasting, sleeping on the ground, and prolonged vigils, defend the wronged, be as a father to orphans, and supply the place of a husband to their mother. But as a matter of fact, setting aside all these things, what does He say? Tend my sheep. For those things which I have already mentioned might easily be performed by many even of those who are under authority, women as well as men; but when one is required to preside over the Church, and to be entrusted with the care of so many souls, the whole female sex must retire before the magnitude of the task, and the majority of men also…”
~“On the Priesthood” Book II, ch. 1 and 2
“But since I have mentioned Peter, I have perceived a fifth crown woven from him, and this is that this man succeeded to the office after him. For just as any one taking a great stone from a foundation hastens by all means to introduce an equivalent to it, lest he should shake the whole building, and make it more unsound, so, accordingly, when Peter was about to depart from here, the grace of the Spirit introduced another teacher equivalent to Peter, so that the building already completed should not be made more unsound by the insignificance of the successor.”
~Homilies on St. Ignatius, Chapter 4.
Pope St. Alexander of Alexandria
“O you, blessed with the gift of healing, in the likeness of Peter, the first of the apostle, your namesake, from whom you have inherited his power of binding and loosing in heaven!”
~An Appraisal of St. Peter of Alexandria
Sourced from: H. Hyvernat. Les actes des martyrs de l’Égypte tirés des manuscripts coptes de la Bibliothèque Vaticane et du Musée Borgia, t. 1, Paris. 1886, pp. 247–262. Translation from French to English mine. (RUSSIAN TRANSLATION)
Life of Shenoute
Shenoute was in conversation with Christ when the bishop came and demanded to see him. When Shenoute sent his servant to tell the bishop he was occupied and could not see him, the bishop threatened excommunication:
“The servant went to our father [Shenoute] and said to him what the bishop had told him. But my father smiled graciously with laughter and said: “See what this man of flesh and blood has said! Behold, here sitting with me is he who created heaven and earth! I will not go while I am with him.” But the Savior said to my father: “O Shenoute, arise and go out to the bishop, lest he excommunicate you. Otherwise, I cannot let you enter heaven because of the covenant I made with Peter, saying ‘What you will bind on earth will be bound in heaven, and what you will loose on earth will be loosed in heaven’ [Matthew 16:19]. When my father heard these words of the Savior, he arose, went out to the bishop and greeted him.”
~Besa, Life of Shenoute 70-72 (trans. Bell). On the context of this story see Behlmer 1998, esp. pp. 353-354. Gaddis, There is No Crime for those who have Christ, p. 296
St. Pope Leo the Great
Commenting on these words, “Whatsoever you shall bind on earth, it shall have been bound in heaven, and whatsoever you shall loose, shall have been loosed in heaven,” he says:
“This power is confided to him in a special manner, because the type (“forma”) of Peter is proposed to all the pastors of the Church. Therefore the privilege of Peter dwells wherever judgement is given with his equity.”
In the following, it appears St. Leo is saying the metropolitans hold the place of Peter but it is not clear to me.
“And so we would have you recollect, brethren, as we do, that the Apostolic See, such is the reverence in which it is held, has times out of number been referred to and consulted by the priests of your province as well as others, and in the various matters of appeal, as the old usage demanded, it has reversed or confirmed decisions: and in this way the unity of the spirit in the bond of peace Ephesians 4:3 has been kept, and by the interchange of letters, our honourable proceedings have promoted a lasting affection: for seeking not our own but the things of Christ Philippians 2:21, we have been careful not to do despite to the dignity which God has given both to the churches and their priests. But this path which with our fathers has been always so well kept to and wisely maintained, Hilary has quitted, and is likely to disturb the position and agreement of the priests by his novel arrogance: desiring to subject you to his power in such a way as not to allow himself to be subject to the blessed Apostle Peter, claiming for himself the ordinations of all the churches throughout the provinces of Gaul, and transferring to himself the dignity which is due to metropolitan priests; he diminishes even the reverence that is paid to the blessed Peter himself with his proud words: for not only was the power of loosing and binding given to Peter before the others, but also to Peter more especially was entrusted the care of feeding the sheep. Yet any one who holds that the headship must be denied to Peter, cannot really diminish his dignity: but is puffed up with the breath of his pride, and plunges himself into the lowest depth.”
Pope St. Gregory the Great
“To Eulogius, Bishop. Gregory to Eulogius, Bishop of Alexandria. Your most sweet Holiness has spoken much in your letter to me about the chair of Saint Peter, Prince of the apostles, saying that he himself now sits on it in the persons of his successors. And indeed I acknowledge myself to be unworthy, not only in the dignity of such as preside, but even in the number of such as stand. But I gladly accepted all that has been said, in that he has spoken to me about Peter’s chair who occupies Peter’s chair.”
~Registrum Epistolarum, Book VII, Letter 40 (New Advent)
St. Gaudentius of Brescia
“I beseech our common father Ambrose [of Milan]…Let him speak from that Holy Spirit with which he is filled, and ‘from his belly shall flow rivers of living water;’ and, as successor of Peter, he shall be the mouth of all the surrounding priests. For when the Lord Jesus asked of the apostles, ‘Whom do you say that I am?’ Peter alone replies, with the mouth of all believers, ‘Thou art the Christ, the Son of the living God.’”
~Tract. 16, De Ordin. Ipsius.; cited by J. Waterworth S.J., A Commentary (London: Thomas Richardson, 1871), pp. 105-107).
Saint Gildas the Wise
“Britain hath priests, but they are unwise; very many that minister, but many of them impudent; clerks she hath, but certain of them are deceitful raveners; pastors (as they are called) but rather wolves prepared for the slaughter of souls (for they provide not for the good of the common people, but covet rather the gluttony of their own bellies), possessing the houses of the church, but obtaining them for filthy lucre’s sake; instructing the laity, but showing withal most depraved examples, vices, and evil manners; seldom sacrificing, and seldom with clean hearts, standing at the altars, not correcting the commonality for their offences, while they commit the same sins themselves; despising the commandments of Christ, and being careful with their whole hearts to fulfil their own lustful desires, some of them usurping with unclean feet the seat of the apostle Peter; but for the demerit of their covetousness falling down into the pestilent chair of the traitor Judas”
~De Excidio et Conquestu Britanniae (On the Ruin and Conquest of Britain), Chapter 3, Section 66, Translated by John Allen Giles, available online at: https://sourcebooks.fordham.edu/basis/gildas-full.asp
St. Theodoret of Cyrhus
“Dioscorus, however, refuses to abide by these decisions; he is turning the see of the blessed Mark (Alexandria) upside down; and these things he does though he perfectly well knows that the Antiochene metropolis possesses the throne of the great Peter, who was teacher of the blessed Mark, and first and coryphæus of the chorus of the apostles.”
St. Hilary of Poitiers
Hilary called those who condemned Saint Athanasius: “O worthy successors of Peter and Paul” (O dignos successores Petro atque Paul)
~Jacques Paul Migne, Patrologia Latina, Volume 10, Column 645A, available online at: https://books.google.co.uk/books?id=G2w3DOjSopAC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false
Translated by Edward Denney, Papalism (London: Rivingtons, 1912), p. 67, available online at: https://archive.org/details/papalismtreatise00denn/page/66/mode/2up
St. Aphrahat the Persian Sage
“The good shepherd giveth himself for the sake of his sheep (John 10:11). And again He said:—I have other sheep and I must bring them also hither. And the whole flock shall be one, and one shepherd, and My Father because of this loveth Me; that I give Myself for the sake of the sheep (John 10:16-17). And again He said;—I am the door of the sheep. Every one that entereth by Me shall live and shall go in and go out and find pasture (John 10:9). O ye pastors, be ye made like unto that diligent pastor, the chief of the whole flock, who cared so greatly for his flock. He brought nigh those that were afar off. He brought back the wanderers. He visited the sick. He strengthened the weak. He bound up the broken. He guarded the fatlings. He gave himself up for the sake of the sheep. He chose and instructed excellent leaders, and committed the sheep into their hands, and gave them authority over all his flock. For He said to Simon Cephas:—Feed My sheep and My lambs and My ewes (John 21:15-17). So Simon fed His sheep; and he fulfilled his time and handed over the flock to you, and departed. Do ye also feed and guide them well. For the pastor who cares for his sheep engages in no other pursuit along with that. He does not make a vineyard, nor plant gardens, nor does he fall into the troubles of this world. Never have we seen a pastor who left his sheep in the wilderness and became a merchant, or one who left his flock to wander and became a husbandman. But if he deserts his flock and does these things he thereby hands over his flock to the wolves.”
~ Demonstration 10 (Of Pastors), Translated by John Gywnne, available online at: http://www.newadvent.org/fathers/370110.htm, accessed 30th November 2019
A special thanks to those who have contributed, especially “St. Dostoyevsky” who contributed several quotations.