Introducing @duplibot - Reducing rewards on comment spam

Didn't you just say that?

We've all seen, "Great post!", "Nice share", and even, "Follow me" and most of us just look past these insignificant comments on our way to the real content. What some don't realize is that many times the account making these comments is literally copying and pasting the exact same thing again and again, cluttering this place with vague, often complimentary remarks with the sole intent of grabbing a few upvotes. Easy money, really. Especially if you automate it.

These sort of posts are all comment spam.

Mission Statement

To discourage the rewarding of duplicate comments by increasing visibility of these behaviors to effectively eliminate comment spam.

A secondary mission is to stabilize the bandwidth issues that have appeared along with our significant growth.

Bandwidth

The Steem Whitepaper lays out the entire foundation upon which Steem was built and offers this core principle.

Blockchain technology currently depends upon transaction fees to prevent spam.

When a blockchain has increased transaction volume, the fees will increase and people will pause before continuing to transact at those higher fees. Basically, higher fees result in lower volume. Instead of fees, Steem has implemented bandwidth as a way to handle transaction volume.

This post as well as this post offer an excellent overview of bandwidth and the challenges we face.

The Whitepaper continues.

...the blockchain will automatically adjust the reserve ratio for the network during times of congestion...any time the surges are sustained the blockchain reduces the maximum bandwidth-per-share. When a surge is over and there is surplus capacity the blockchain can slowly increase the bandwidth-per-share.

Specifically, this is referring to the current_reserve_ratio which is graphed online so we can see how the reserve ratio changes over time.

No matter when you look, you'll probably see something like this.

Every single day, the reserve ratio plummets.

The reserve ratio is adjusted any time the average size of the new blocks is greater than 25% of the maximum block size. The more traffic on the blockchain, the larger the average size of new blocks. If the new blocks get too big, the current_reserve_ratio is adjusted, which lowers bandwidth for everyone. And this is happening every single day, because a LOT of people are posting a LOT of content.

Steem is a Dynamic Fractional Reserve which adjusts itself to respond to changing conditions. If too many people are posting, the bandwidth is limited to make it harder to post.

This basically cripples new minnows with no SP and only a fighting chance of catching on around here, who are shutdown by the universal current_reserve_ratio even if they have not posted in a while.

Comment spam stops the rest of us from posting because of bandwidth.

Why are they spamming?

In a word, money. Every comment you make has an upvote button, and these comments do get a meaningful amount of upvotes, which only encourages further spamming.

And comment spam is the easiest. You need content for an actual post, but a comment can be just a few words. Quite often, the spam blends right in because it will often compliment and endear the author to entice an upvote.

These comments look completely normal so upvotes come along freely, more comments are made, and the cycle continues. We need to disrupt that cycle.

Introducing @duplibot

There are many different types of comment spam, most of which require some actual thinking to identify, but the exact same comment repeated by an account is easy to identify and can be handled by a bot. That's where I come in.

My job is to identify accounts posting duplicate comments and reply to let the community know like this.

^--- You made this exact same comment at least 21 times in the last 2 hours. View these duplicate comments

Upvote this comment to support @duplibot and help reduce spam and superfluous comments.

My intent is to point out that the account reposted the same thing over a few hours and provide a quick link to the comments to verify with the sole intention of discouraging upvotes. If we upvote this comment spam it will persist, but if the entire community joins together to reject these messages they are more likely to cease.

Getting started

As you can see my account is only a few days old. I wanted to take some time to get on my feet and see how the community responded before I made this intro post.

When looking at duplicate comments, there are TONS of false positives from other legitimate bots, @utopian-io moderators, and the painful to see, "My account was hacked, can you please remove your downvote..." Replying to any of these is beyond annoying and nearing offensive.

To avoid anything like that, today every comment I make is handled manually and I plan to keep it that way for a while. My credibility and actual reputation would be destroyed if I blindly replied to all duplicate comments, especially those legitimate and well-meaning replies.

I am in learning mode, trying to figure out what works and what I need to adjust.

First downvote

So far, the response has been quite positive. Then I had a downvote, which I then commented on.

Not only did I receive 8 upvotes (plus 1 on my second comment), the original spammer received 3 downvotes. That was validation I really appreciated.

At the time of this writing, I've made 285 comments and received 80 upvotes across 55 of those comments and I've received 19 replies that were directly complimentary and only 3 that spoke out against me.

A spammer replies

I recently came across this account which looks for posts discussing Bitcoin and posts the current price, "For future readers" to know what the price was at the time of writing.

Since this account posts a different price for most posts, I don't see as many exact duplicates but there are many close variations and this account revealed this metric to me. So, I capture the number of times for that exact comment, but then I started looking for close variations to give the full context.

My post started as usual, now also noting the close variations.

Shortly thereafter, this account replied.

As expected, defensive posturing declaring the posts as 'useful' and pronouncing me as the spammer.

I put together what I thought was a very thoughtful response, pointing out some clear metrics that tell a far different story, and opening the dialog and even offering to connect on steem.chat or Discord.

Then I got another reply.

A few key points here. I legitimately appreciate this opening remark.

Meybe [sic] we should hear each other.

Absolutely, I'm at the table listening and ready to talk. And this was the line that brought it home for me.

I am prepared to limit my spam if I can improve quality.

Wow, when you say, "limit my spam" I take that as evidence I have reached you well enough that you have cast off your denial and you actually hear the points I'm making.

And then I looked at the recent comments from this account and saw this.

This account has stopped posting after my last reply!

This is a perfect example because I believe this account really does have generally good intentions. I have had at least one reply to my comment saying the price is useful to add context, so it's not rejected by everyone, but there has to be a middle ground. I think I've opened the door to finding that middle ground and I couldn't be more excited!

On the whole I feel a great deal of support from the community so I'm excited to continue my work.

Why are you doing this?

To eliminate comment spam.

In the simplest sense, it is clutter, but the larger picture is that this community will thrive if we have real engagement and apropos comments. The learning curve here is no joke so for a minnow to arrive and see some of this nonsense, it is first confusing and later it ultimately drives attrition as users abandon our platform for safer ground.

Another larger issue we've already touched on is bandwidth. All of these superfluous comments contribute to the bandwidth problems we see every day. This is not some hypothetical, we get roughly 250,000 posts and comments every single day.

To sustain this blockchain we will need to either reduce spam comments or accept the bandwidth limitations. The latter realty is easy for those with 100+ SP, but for everyone who shows up with barely a clue of how things work this bandwidth limitation shuts them down completely. We cannot grow as a community with this reality persisting so I hope to make a measurable impact on spam.

I maintain a complete local copy of the blockchain in a database so I can easily sift through the data to find these duplicate comments. More importantly, I can look to the data to identify trends in comment spam generally and try to correlate my efforts to any changes in these data. To put it plainly, I'll look to the data for validation that I am reducing spam.

If I'm not having a measurable impact, then I'm just adding transactions to the blockchain that disrupts the reserve ratio, making me part of the problem I'm trying to solve. I won't keep piping out more comments if there isn't some quantitative metric to back my efforts, as well as the qualitative feedback from the community, which is perhaps the most important validation.

Aren't you creating the very thing you're fighting against?

Indeed I am, and this is the true poetry of life.

These comment spoke to the point exactly.

In short, my script identified my own duplicate comments and I could have easily replied to myself! Another reason I'm doing everything manually today.

And to be fair, you can track my followers here and you'll see I've got a fair amount of mutes showing up. I can't blame them one bit, I'm the sort of stuff you just don't wanna see.

To be honest, I continue to struggle with this point, particularly because of bandwidth so I'm continually evaluating this and will adjust as necessary. From a purely bandwidth perspective, I believe I can reduce the total bandwidth usage far above the amounts I'm using for my comments.

For now I'm making very limited posts, only 1 or 2 per account each round of posting. Some of the major abusers making hundreds of posts have seen me getting 10 or more posts out, but I'm trying to limit myself.

Will you downvote?

The flag wars around here are real and I will not participate in that. Today my purpose is to raise awareness of the disingenuous remarks being repeated everywhere so that people decide not to upvote this spam.

From a broad view, if we reduce upvotes on spam content, it will go away. These comments are produced to generate earnings and if that money disappears, so will the spammers.

This is a very community oriented place, so when someone stops by and says, "Awesome post, following, can't wait for more!" it's basic instinct to express your thanks with an upvote. If I point out that this has been posted 13 times in the last 2 hours, very few people will still vote.

What's next?

As I mentioned, I'm still figuring everything out. I will definitely continue manually for a while and I'm exploring ways to validate my work with data. I'm going to look at comment patterns of accounts before and after I start replying to see if I have any impact. And then look to the data overall to see if spam is generally decreasing.

Long term, I hope to establish a site for users to submit links that they see as duplicates. I will check the account and comment as necessary, adding the reporting account as a beneficiary to any rewards on those comments.

I have many thoughts and ideas of who I am and where I'm going but I start from this point with an open mind and I plan to continue exploring to determine exactly what's next.

I expect that I will continue to evolve as I learn more from both the data and the community as well.

What do you think?

I'm here to support and build the community so I want to validate everything I'm doing by directly asking for your feedback. What do you like about @duplibot and where am I falling short? Do my comments annoy you to no end or do you see the value in my work? Have you muted me or are you following me?

Let me know your thoughts about @duplibot and let's all work together to end comment spam!

And since I'm totally new and can easily fade into the dust around here, please consider resteeming this post to increase visibility and let the community--and spammers!--know that I'm out here working hard to clean up spam.