Though the Pope of Rome, the Pope of Alexandria, and the Patriarch of Antioch (among others) are the successors of Peter by lineage, all bishops are successors to Peter and this is reflected in a variety of patristic sources. If you know of additional quotations related to this, please post them below and we will add them.

---

St. Cyprian of Carthage (+258)

“Our Lord, whose precepts and admonitions we ought to observe, describing the honor of a bishop and the order of His Church, speaks in the Gospel, and says to Peter: “I say unto thee, That thou art Peter, and upon this rock will I build my Church; and the gates of hell shall not prevail against it. And I will give unto thee the keys of the kingdom of heaven: and whatsoever thou shalt bind on earth shall be bound in heaven: and whatsoever thou shall loose on earth shall be loosed in heaven.” Thence, through the changes of times and successions, the ordering of bishops and the plan of the Church flow onwards; so that the Church is founded upon the bishops, and every act of the Church is controlled by these same rulers. Since this, then, is founded on the divine law, I marvel that some, with daring temerity, have chosen to write to me as if they wrote in the name of the Church; when the Church is established in the bishop and the clergy, and all who stand fast in the faith.” ~Epistle 26:1

St. Cyril of Alexandria (+444)

“And, by dwelling on this passage, instructors in religion may arrive at the knowledge that they cannot please the Chief Shepherd, that is Christ, unless they take thought for the health of the sheep of His fold, and their continuance in well-being. Such was the inspired Paul, who shared the infirmities of his weak brethren, and called those who through him believed, and chose to gain repute by the glory of their deeds, the boast, and joy, and crown of his apostleship.” Commentary on John 21:15-17 in ACCS   

St. Bede (+735)

“Although it may seem that this power of loosing and binding was given by the Lord only to Peter, we must nevertheless know without any doubt that it was given to the other apostles…Indeed even now the same office is committed to the whole Church in her bishops and priests.” ~Homilies on the Gospels: Book I: Advent to Lent, Hom. I.20, p. 202

Alcuin of York (+804)

To feed the sheep is to support the believers in Christ from falling from the faith, to provide earthly sustenance for those under us, to preach and exemplify withal our preaching by our lives, to resist adversaries, to correct wanderers.” Commentary on John 21:15 Found in ACCS

St. Germanus of Constantinople (+733)

The double crown inscribed on the head of the priest through tonsure represents the precious head of the Chief-Apostle Peter. When he was sent out in the teaching and preaching of the Lord, his head was shaved by those who did not believe his word, as if in mockery. The Teacher Christ blessed this head, changed dishonour into honour, ridicule into praise. He placed on it a crown made not of precious stones, but one which shines more than gold, topaz, or precious stones – with the stone and rock of faith. Peter, the most holy, the summit, beauty, and crown of the twelves stones, which are the Apostles, is the hierarch of Christ.” 

St. Isidore of Seville (+636)

“So Peter first received the power of binding and loosing, and he first led people to faith by the power of his preaching. Still, the other Apostles have been made equal with Peter in a fellowship of dignity and power. They also, having been sent out into all the world, preached the Gospel. Having descended from these apostles, the bishops have succeeded them, and through all the world they have been established in the seats of the apostles” ~De Ecclesiasticus, II.5, M.P.L., Vol. 83, Col. 781-782.

St. John Chrysostom (+407)

“Will you that I tell you a fourth proof also of these things? Peter, do you love me, says He; Feed my sheep: (John 21:15-17) and having asked him a third time, declared this to be an infallible proof of love. But not to priests only is this said, but to every one of us also, who are also entrusted with a little flock.” ~Homilies on St. Matthew, 77th homily 

The term “priests” in the Greek patristic literature typically meant “bishops” so perhaps the better interpretation should be “Not to bishops only is this said, but to every one of us also, who are entrusted with a little flock [i.e. parish priests]’

“In speaking of St. Peter, the recollection of another Peter [Flavian, Bishop of Antioch, at the time the discourse was written,] has come to me, the common father and teacher, who has inherited his prowess, and also obtained his chair. For this is the one great privilege of our city, Antioch, that it received the leader of the apostles as its teacher in the beginning. For it was right that she who was first adorned with the name of Christians, before the whole world, should receive the first of the apostles as her pastor. But though we received him as teacher, we did not retain him to the end, but gave him up to royal Rome. Or rather we did retain him to the end, for though we do not retain the body of Peter, we do retain the faith of Peter, and retaining the faith of Peter we have Peter.” ~On the Inscription of the Acts

Speaking to friend Basil (not to be confused with St. Basil the Great) who had just been consecrated a bishop, St. John Chrysostom says:

“What advantage, pray, could be greater than to be seen doing those things which Christ with his own lips declared to be proofs of love to Himself? (John 21:15-17) For addressing the leader of the apostles He said, Peter, do you love me? and when he confessed that he did, the Lord added, if you love me tend my sheep. The Master asked the disciple if He was loved by him, not in order to get information (how should He who penetrates the hearts of all men?), but in order to teach us how great an interest He takes in the superintendence of these sheep. This being plain, it will likewise be manifest that a great and unspeakable reward will be reserved for him whose labors are concerned with these sheep, upon which Christ places such a high value…For what purpose did He shed His blood? It was that He might win these sheep which He entrusted to Peter and his successors. Naturally then did Christ say, Who then is the faithful and wise servant, whom his lord shall make ruler over His household…Observe at any rate how great the reward is — He will appoint him, He says, ruler over all His goods. (Matt 24:47) Will you, then, still contend that you were not rightly deceived, when you are about to superintend the things which belong to God, and are doing that which when Peter did the Lord said he should be able to surpass the rest of the apostles, for His words were, Peter, do you love me more than these? Yet He might have said to him, If you love me practice fasting, sleeping on the ground, and prolonged vigils, defend the wronged, be as a father to orphans, and supply the place of a husband to their mother. But as a matter of fact, setting aside all these things, what does He say? Tend my sheep. For those things which I have already mentioned might easily be performed by many even of those who are under authority, women as well as men; but when one is required to preside over the Church, and to be entrusted with the care of so many souls, the whole female sex must retire before the magnitude of the task, and the majority of men also…” ~“On the Priesthood” Book II, Chapters 1 and 2

Speaking of St. Ignatius of Antioch, St. John Chrysostom says:

“But since I have mentioned Peter, I have perceived a fifth crown woven from him, and this is that this man succeeded to the office after him. For just as any one taking a great stone from a foundation hastens by all means to introduce an equivalent to it, lest he should shake the whole building, and make it more unsound, so, accordingly, when Peter was about to depart from here, the grace of the Spirit introduced another teacher equivalent to Peter, so that the building already completed should not be made more unsound by the insignificance of the successor.”  ~Homilies on St. Ignatius, Chapter 4

Pope St. Alexander of Alexandria (+326)

“O you, blessed with the gift of healing, in the likeness of Peter, the first of the apostle, your namesake, from whom you have inherited his power of binding and loosing in heaven!”  ~An Appraisal of St. Peter of Alexandria

Sourced from: H. Hyvernat. Les actes des martyrs de l’Égypte tirés des manuscripts coptes de la Bibliothèque Vaticane et du Musée Borgia, t. 1, Paris. 1886, pp. 247–262. Translation from French to English mine. (RUSSIAN TRANSLATION)

St. Augustine of Hippo (+430)

“Those who have this purpose in feeding the flock of Christ, that they may have them as their own, and not as Christ’s, are convicted of loving themselves, and not Christ, from the desire either of boasting, or wielding power, or acquiring gain, and not from the love of obeying, serving, and pleasing God. […] Let us, then, love not ourselves, but Him; and in feeding His sheep, let us be seeking the things which are His, not the things which are our own. […] Let not those, then, who feed Christ’s sheep be lovers of their own selves, lest they feed them as if they were their own, and not His […] But if the good Shepherd, who laid down His own life for His sheep, has raised up so many martyrs for Himself out of the very sheep, how much more ought those to contend to death for the truth, and even to blood against sin, who are entrusted by Him with the feeding, that is, with the teaching and governing of these very sheep? And on this account, along with the preceding example of His own passion, who can fail to see that the shepherds ought all the more to set themselves closely to imitate the Shepherd, if He was so imitated even by many of the sheep under whom, as the one Shepherd and in the one flock, the shepherds themselves are likewise sheep? For He made all those His sheep for [all of] whom He died, because He Himself also became a sheep that He might suffer for all.” ~Tractate 123.5 (Commentary on John 21:12-19)

Notice St. Augustine applies the term not only to the Bishop of Rome but onto bishops in general. 

Life of Shenoute (+466)

Shenoute was in conversation with Christ when the bishop came and demanded to see him. When Shenoute sent his servant to tell the bishop he was occupied and could not see him, the bishop threatened excommunication:

“The servant went to our father [Shenoute] and said to him what the bishop had told him. But my father smiled graciously with laughter and said: “See what this man of flesh and blood has said! Behold, here sitting with me is he who created heaven and earth! I will not go while I am with him.” But the Savior said to my father: “O Shenoute, arise and go out to the bishop, lest he excommunicate you. Otherwise, I cannot let you enter heaven because of the covenant I made with Peter, saying ‘What you will bind on earth will be bound in heaven, and what you will loose on earth will be loosed in heaven’ [Matthew 16:19]. When my father heard these words of the Savior, he arose, went out to the bishop and greeted him.”

~Besa, Life of Shenoute 70-72 (trans. Bell). On the context of this story see Behlmer 1998, esp. pp. 353-354. Gaddis, There is No Crime for those who have Christ, p. 296

St. Pope Leo the Great (+461)

Commenting on these words, “Whatsoever you shall bind on earth, it shall have been bound in heaven, and whatsoever you shall loose, shall have been loosed in heaven,” he says: “This power is confided to him in a special manner, because the type (“forma”) of Peter is proposed to all the pastors of the Church. Therefore the privilege of Peter dwells wherever judgement is given with his equity.” ~Sermon III

In the following, it appears St. Leo is saying the metropolitans hold the place of Peter but it is not clear to me.

“And so we would have you recollect, brethren, as we do, that the Apostolic See, such is the reverence in which it is held, has times out of number been referred to and consulted by the priests of your province as well as others, and in the various matters of appeal, as the old usage demanded, it has reversed or confirmed decisions: and in this way the unity of the spirit in the bond of peace has been kept, and by the interchange of letters, our honourable proceedings have promoted a lasting affection: for seeking not our own but the things of Christ, we have been careful not to do despite to the dignity which God has given both to the churches and their priests. But this path which with our fathers has been always so well kept to and wisely maintained, Hilary has quitted, and is likely to disturb the position and agreement of the priests by his novel arrogance: desiring to subject you to his power in such a way as not to allow himself to be subject to the blessed Apostle Peter, claiming for himself the ordinations of all the churches throughout the provinces of Gaul, and transferring to himself the dignity which is due to metropolitan priests; he diminishes even the reverence that is paid to the blessed Peter himself with his proud words: for not only was the power of loosing and binding given to Peter before the others, but also to Peter more especially was entrusted the care of feeding the sheep. Yet any one who holds that the headship must be denied to Peter, cannot really diminish his dignity: but is puffed up with the breath of his pride, and plunges himself into the lowest depth.”  ~Epistle 10

Pope St. Gregory the Great (+604)

“To Eulogius, Bishop. Gregory to Eulogius, Bishop of Alexandria. Your most sweet Holiness has spoken much in your letter to me about the chair of Saint Peter, Prince of the apostles, saying that he himself now sits on it in the persons of his successors. And indeed I acknowledge myself to be unworthy, not only in the dignity of such as preside, but even in the number of such as stand. But I gladly accepted all that has been said, in that he has spoken to me about Peter’s chair who occupies Peter’s chair.” ~Registrum Epistolarum, Book VII, Letter 40

St. Gaudentius of Brescia (+410)

“I beseech our common father Ambrose [of Milan]…Let him speak from that Holy Spirit with which he is filled, and ‘from his belly shall flow rivers of living water;’ and, as successor of Peter, he shall be the mouth of all the surrounding priests. For when the Lord Jesus asked of the apostles, ‘Whom do you say that I am?’ Peter alone replies, with the mouth of all believers, ‘Thou art the Christ, the Son of the living God.’”

~Tract. 16, De Ordin. Ipsius.; cited by J. Waterworth S.J., A Commentary (London: Thomas Richardson, 1871), pp. 105-107).

St. Gildas the Wise (+570)

“Britain hath priests, but they are unwise; very many that minister, but many of them impudent; clerks she hath, but certain of them are deceitful raveners; pastors (as they are called) but rather wolves prepared for the slaughter of souls (for they provide not for the good of the common people, but covet rather the gluttony of their own bellies), possessing the houses of the church, but obtaining them for filthy lucre’s sake; instructing the laity, but showing withal most depraved examples, vices, and evil manners; seldom sacrificing, and seldom with clean hearts, standing at the altars, not correcting the commonality for their offences, while they commit the same sins themselves; despising the commandments of Christ, and being careful with their whole hearts to fulfil their own lustful desires, some of them usurping with unclean feet the seat of the apostle Peter; but for the demerit of their covetousness falling down into the pestilent chair of the traitor Judas”

~De Excidio et Conquestu Britanniae (On the Ruin and Conquest of Britain), Chapter 3, Section 66, Translated by John Allen Giles

Theodoret of Cyrhus (+458)

“Dioscorus, however, refuses to abide by these decisions; he is turning the see of the blessed Mark (Alexandria) upside down; and these things he does though he perfectly well knows that the Antiochene metropolis possesses the throne of the great Peter, who was teacher of the blessed Mark, and first and coryphæus of the chorus of the apostles.”  ~Letter 86

St. Hilary of Poitiers (+367)

Hilary called those who condemned Saint Athanasius: “O worthy successors of Peter and Paul” (O dignos successores Petro atque Paul) ~Jacques Paul Migne, Patrologia Latina, Volume 10, Column 645A

Translated by Edward Denney, Papalism. London: Rivingtons, 1912, p. 67, available online here.

Aphrahat the Persian Sage (+345)

“The good shepherd giveth himself for the sake of his sheep (John 10:11). And again He said:—I have other sheep and I must bring them also hither. And the whole flock shall be one, and one shepherd, and My Father because of this loveth Me; that I give Myself for the sake of the sheep (John 10:16-17). And again He said;—I am the door of the sheep. Every one that entereth by Me shall live and shall go in and go out and find pasture (John 10:9). O ye pastors, be ye made like unto that diligent pastor, the chief of the whole flock, who cared so greatly for his flock. He brought nigh those that were afar off. He brought back the wanderers. He visited the sick. He strengthened the weak. He bound up the broken. He guarded the fatlings. He gave himself up for the sake of the sheep. He chose and instructed excellent leaders, and committed the sheep into their hands, and gave them authority over all his flock. For He said to Simon Cephas:—Feed My sheep and My lambs and My ewes (John 21:15-17). So Simon fed His sheep; and he fulfilled his time and handed over the flock to you, and departed. Do ye also feed and guide them well. For the pastor who cares for his sheep engages in no other pursuit along with that. He does not make a vineyard, nor plant gardens, nor does he fall into the troubles of this world. Never have we seen a pastor who left his sheep in the wilderness and became a merchant, or one who left his flock to wander and became a husbandman. But if he deserts his flock and does these things he thereby hands over his flock to the wolves.”  ~ Demonstration 10 (Of Pastors), Translated by John Gywnne.

Pseudo-Dionysius (Late 5th Century)

­“Similarly, insofar as he makes known the judgments of God he has also the powers of excommunication. Not indeed that the all-wise divinity gives in to his every unthinking impulses, if I may so speak with all reverence. But the hierarch obeys the Spirit which is the source of every rite and which speaks by way of his words. He excommunicates those unworthy people whom God has already judged. It says: ‘Receive the holy Spirit. If you forgive the sins of any, they are forgiven; if you retain the sins of any, they are retained.’ And to the one enlightened by the sacred revelation of the all-holy Father it is said in Scripture, ‘whatever you bind on earth shall be bound in heaven and whatever you loose on earth shall be loosed in heaven.’ Thus [Peter] himself and all the hierarchs like him have had the judgment of the Father revealed to them, and, being themselves men who provide revelation and explanation, they have the task of admitting the friends of God and of keeping away the ungodly. That sacred acknowledgement of God came to him, as Scripture shows, not on his own, not from a flesh-and-blood revelation, but as something from the understanding and under the influence of the God who initiated him into what he knew. Similarly, God’s hierarchs must use their powers of excommunication, as well as all their other hierarchical powers, to the extent that they are moved by the divinity which is the source of every rite. And everyone else must obey the hierarchs when they act as such, for they are inspired by God Himself “He who rejects you, it says, ‘rejects me.’”  Celestial Hierarchy 7:7 (translation by Dr. Colm Luibheid)

St. Theodore the Studite (+826)

“But here it is a question of divine and heavenly decisions and those are reserved only to him to whom the Word of God has said: “Whatsoever you shall bind upon earth, will be bound in Heaven and whatsoever you shall loose on earth, shall be loosed in Heaven.” And who are the men to whom this order was given? –  the Apostles and their successors. And who are their successors? –he who occupies the throne of Rome and is the first; the one who sits upon the throne of Constantinople and is the second; after them, those of Alexandria, Antioch, and Jerusalem. That is the Pentarchic authority in the Church. It is to them that all decision belongs in divine dogmas. The Emperor and he secular authority have the duty to aid them and to confirm what they have decided.”

St. Theodore the Studite (Taken from Fr. Francis Dvornik’s, “Byzantium and the Roman Primacy” p. 101)

The reference is clearly to Matthew 16:17-19 because he states “…only to him…“ (singular). Were St. Theodore initially referring to Matt 18:18, it would have instead been in the plural, not the singular. St. Theodore then, with Matt 18:18 in mind, understands this to apply the apostles and therefore all of the bishops. It goes back to the idea that St. Peter ordained the other Apostles and it is via the apostles that each bishop is Peter.

Blessed Theophylact of Ochrid (+1108)

“He spoke as God, with authority, “I will give unto thee.” For as the Father gave you the revelation, so I give you the keys. By “keys” understand that which binds or looses transgressions, namely, penance or absolution; for those who, like Peter, have been deemed worthy of the grace of the episcopate, hav the authority to absolve or to bind. Even through the worlds “I will give untio thee” were spoke to Peter alone, yet they were given to all the apostles. Why? Because He said “Whosoever sins yet remint, they are remitted.” 

Commentary on Matthew 16:19 in “The Explanation of the Holy Gospel According to Matthew” pages 140-141. Translated by Fr. Christopher Stade. Chrysostom Press 1992

“Therefore he who loves Christ, and has charge of both lambs and sheep, will feed the lambs, meaning provide them simple, gentle care suitable for beginners, and will tend the sheep, meaning, apply stricter care for those more mature in faith. But even the more advanced sometimes need gentler care as well, so the Lord instructs His pastors to feed the sheep also. 

Commentary on John 21:15-18 in “The Explanation of the Holy Gospel According to John” page 309. Translated by Fr. Christopher Stade. Chrysostom Press 2007

---

A special thanks to those who have contributed, especially “St. Dostoyevsky” who contributed several quotations.