When discussing sin, it's important that we not insert modern definitions. That can lead to confusion. Rather the scripture defines sin.

1 John 3:4
Everyone who makes a practice of sinning also practices lawlessness; sin is lawlessness.

Romans 7:7
The Law and Sin
What then shall we say? That the law is sin? By no means! Yet if it had not been for the law, I would not have known sin. For I would not have known what it is to covet if the law had not said, “You shall not covet.”

So what does God's Law say?

Exodus 20:14
“You shall not commit adultery.

Take for instance, the most famous adulterer in the Bible is King David. He is practically the poster boy for this particular sin. Nathan the prophet explains David's sin in a metaphor about sheep.

2Sa 12:1 — 2Sa 12:9
​ And the Lord sent Nathan to David. He came to him and said to him, “There were two men in a certain city, the one rich and the other poor. The rich man had very many flocks and herds, but the poor man had nothing but one little ewe lamb, which he had bought. And he brought it up, and it grew up with him and with his children. It used to eat of his morsel and drink from his cup and lie in his arms, and it was like a daughter to him. Now there came a traveler to the rich man, and he was unwilling to take one of his own flock or herd to prepare for the guest who had come to him, but he took the poor man's lamb and prepared it for the man who had come to him.” Then David's anger was greatly kindled against the man, and he said to Nathan, “As the Lord lives, the man who has done this deserves to die, and he shall restore the lamb fourfold, because he did this thing, and because he had no pity.”
Nathan said to David, “You are the man! Thus says the Lord, the God of Israel, ​‘I anointed you king over Israel, and I delivered you out of the hand of Saul. And I gave you your master's house and your master's wives into your arms and gave you the house of Israel and of Judah. And if this were too little, I would add to you as much more. Why have you despised the word of the Lord, to do what is evil in his sight? You have struck down Uriah the Hittite with the sword and have taken his wife to be your wife and have killed him with the sword of the Ammonites.

Notice how Nathan compares adultery to the sins of theft and murder, by using the symbol of the sheep for Uriah wife? Notice also that the charge Nathan makes against David is his offense against Uriah.

Deuteronomy 22:22
“If a man is found lying with the wife of another man, both of them shall die, the man who lay with the woman, and the woman. So you shall purge the evil from Israel.

Of course David repents, and God forgives him.

2 Samuel 12:13
David said to Nathan, “I have sinned against the Lord.” And Nathan said to David, “The Lord also has put away your sin; you shall not die.

But what should be noted is that this offense is against Uriah the husband of Bathsheeba. The adultery is always referenced this way.

1 Kings 15:5
because David did what was right in the eyes of the Lord and did not turn aside from anything that he commanded him all the days of his life, except in the matter of Uriah the Hittite.

So why wasn't this attributed as an offense against his marriage the way we do today?

It comes down to the word "adultery" which in Hebrew presents the concept of the Hebrew word "naaph". That the strong concordance says literally means, woman that breaketh wedlock.

And that's always what we see, adultery is an offense of one man against another.

I end with the words of Proverbs to further make the point.

Pro 6:20 — Pro 7:27
Warnings Against Adultery
My son, keep your father's commandment,
and forsake not your mother's teaching.
Bind them on your heart always;
tie them around your neck.
When you walk, they will lead you;
when you lie down, they will watch over you;
and when you awake, they will talk with you.
For the commandment is a lamp and the teaching a light,
and the reproofs of discipline are the way of life,
to preserve you from the evil woman,
from the smooth tongue of the adulteress.
Do not desire her beauty in your heart,
and do not let her capture you with her eyelashes;
for the price of a prostitute is only a loaf of bread,
but a married woman hunts down a precious life.
Can a man carry fire next to his chest
and his clothes not be burned?
Or can one walk on hot coals
and his feet not be scorched?
So is he who goes in to his neighbor's wife;
none who touches her will go unpunished.
People do not despise a thief if he steals
to satisfy his appetite when he is hungry,
but if he is caught, he will pay sevenfold;
he will give all the goods of his house.
He who commits adultery lacks sense;
he who does it destroys himself.
He will get wounds and dishonor,
and his disgrace will not be wiped away.
For jealousy makes a man furious,
and he will not spare when he takes revenge.
He will accept no compensation;
he will refuse though you multiply gifts.
Warning Against the Adulteress
​ My son, keep my words
and treasure up my commandments with you;
keep my commandments and live;
keep my teaching as the apple of your eye;
bind them on your fingers;
write them on the tablet of your heart.
Say to wisdom, “You are my sister,”
and call insight your intimate friend,
to keep you from the forbidden woman,
from the adulteress with her smooth words.
For at the window of my house
I have looked out through my lattice,
and I have seen among the simple,
I have perceived among the youths,
a young man lacking sense,
passing along the street near her corner,
taking the road to her house
in the twilight, in the evening,
at the time of night and darkness.
And behold, the woman meets him,
dressed as a prostitute, wily of heart.
She is loud and wayward;
her feet do not stay at home;
now in the street, now in the market,
and at every corner she lies in wait.
She seizes him and kisses him,
and with bold face she says to him,
“I had to offer sacrifices,
and today I have paid my vows;
so now I have come out to meet you,
to seek you eagerly, and I have found you.
I have spread my couch with coverings,
colored linens from Egyptian linen;
I have perfumed my bed with myrrh,
aloes, and cinnamon.
Come, let us take our fill of love till morning;
let us delight ourselves with love.
For my husband is not at home;
he has gone on a long journey;
he took a bag of money with him;
at full moon he will come home.”
With much seductive speech she persuades him;
with her smooth talk she compels him.
All at once he follows her,
as an ox goes to the slaughter,
or as a stag is caught fast
till an arrow pierces its liver;
as a bird rushes into a snare;
he does not know that it will cost him his life.
And now, O sons, listen to me,
and be attentive to the words of my mouth.
Let not your heart turn aside to her ways;
do not stray into her paths,
for many a victim has she laid low,
and all her slain are a mighty throng.
Her house is the way to Sheol,
going down to the chambers of death.