Twitter Thinks I'm A Bot

In September 2020 I rather unknowingly started my biggest project yet. Its idea was born while being slightly drunk and it was implemented within a day - and yet generated more money than my blog did in over 10 years while being completely free.

Disclaimer

This post is not about the technical implementation of my bot and will cover everything around it. In case you're interested in writing your own stock tracking bot please head over to the guide.

Motivation

When Nvidia last year announced its new RTX 3000 graphic card series I got kinda excited. I used to build a decent amount of custom PCs in my free time some years ago for my friends and always enjoyed the experience of getting my hands on brand-new hardware. For my 30th birthday I wanted to make myself a present as a tribute to these times: I wanted to build the shiniest, most gorgeous and powerful computer I ever had. I always went for a good price-performance ratio (especially since most people that got a computer from me didn't really care about aesthetics), but this time I wanted to do something stupid solely focus on performance and aesthetics.

That being said I was ready on launch day to buy one of these shiny 3080 cards. But little did I know: this release was kinda... weird.

Within seconds the entire stock was gone on every shop. While this meme might seem like an exaggeration, it wasn't - stock lasted literally for just a few seconds.

Unfortunately I wasn't one of the lucky like seven people that got a new GPU that day. The next day I had to give my frustration some room: I went for a few drinks with a friend and complained what feels an entire evening about this GPU launch.

After some Spritzers I said something to my friend which would have a relatively huge impact on the next months: "You know, I just don't want to sit around and press F5 all day long to know when something's back in stock" - and while saying that my remaining sober brain was like "wait a minute! this sounds like something a software developer could solve - and we are a software developer!".

Following the advice of my brain I knew what to do: I ordered more Spritzer and put that idea on "let's think about that tomorrow".

Fortunately I didn't drink too much that day and was capable of remembering this idea the next day. After waking up I started to think about technical details and went to work.

While implementing the first version I realized what I was doing there was kinda unfair. There were probably more people out there that wanted to a have new GPU that weren't a developer and had no chance of creating such a service which would give me an unfair advantage. As I'm not a huge fan of creating unfair advantages I improved my plan a bit: instead of making a bot for myself I should make a bot for everyone - and Twitter would be a perfect place for that.

The next moral topic was polling frequency - or in less technical terms - how often would my bot check a shop for stock changes. What plays a role here is my daytime job: part of it is maintaining shop systems - and I'd be pretty upset if someone would cause trouble to one of these shops with a stupid crawler. That thought ultimately led to a very simple principle: I didn't want to hurt shop servers, which is why to this very day the polling frequency of my bot is a whopping 90 seconds.

Then there was the good old question: should this be open source? And, honestly, I didn't think much about this and straight went for "sure!" - which turned out to be a mistake later on - but more on that later.

All that was left was a name - and I'm terrible at naming. Since it was series "3000" I remembered the "I love you 3000" scene from the Avengers movie, slightly changed it and ended with: ilovertx3000. Again, I'm just not good at naming.

Since I wanted a new GPU fast I wanted to implement this entire thing as fast as possible - and to this very day I'm still confused about how I did it: without over-engineering and by piling up tons of tech debt it was done in less than 4 hours.

Release history

To roughly summarize what's happened within the last few months let's briefly hop through the (more important) releases without going into technical detail.

Version 0.1 - The first release

With a total of 390 lines of code (and support for five shops) I released version 0.1 on the 20th September. I posted it on Reddit after I was done with its implementation and went to bed.

When I woke up the next I was overwhelmed: my phone showed more than 500 notifications. At this point in time that was probably more notifications than it has shown in its entire life.

The @ilovertx3000 account gained more than 4.000 followers within a single night, I had hundreds messages on Reddit, pull requests on GitHub, feature requests and all that kind of fun stuff. Obviously people liked what I did - which would completely skyrocket my motivation.

I set up a static site blog to write about updates within the next day which is hosted on Netlify to this very day and started improving the service by adding more shops, going through the pull requests and worked through countless feature requests (which were most likely just "please add shop X").

Version 0.2 - Affiliate links

Two days later I released version 0.2 which would now have a total of 466 lines of code and support a total of 9 shops.

Version 0.2 also introduced something I hadn't even thought about yet: affiliate links. Since my bot would reach hundreds of people I thought that I might be able to fund my own GPU with the money I can make from affiliate marketing.

Around this time I took some money into hand and bought a dedicated server to run the bot from my home:

Until this point it was hosted on AWS, but unfortunately many sites would be able to detect that pretty easily and blocked the IP. While there are obvious disadvantages to this approach (like power outages, connectivity issues, bandwidth usage, etc.) running the server from my home - rotating between proxies and my own IP - drastically decreased error rate and made my life a lot easier. The bot is running on this server to this very day - and still uses DynamoDB, hosted on AWS, for data persistence.

Version 0.3 - the PS5 bot

Five days later I released version 0.3 which would initialize the second Twitter account: @iloveps_5 (sticking to the ilove naming scheme made it a lot easier to find a name). Little did I know, this account would cause this entire project to really take off in a few weeks.

Another big thing of 0.3 was the refactoring to a configuration-based setup. So far every crawler "knew" what products to crawl which wouldn't scale that well, especially when adding more products to monitor. As a consequence I refactored it to a configuration-based approach, where I could simply spin up new bots by adding new configurations. Think of it like this: the PlayStation 5 bot is just a configuration file which states where the bot should look for the PS5 - and crawlers became simple workers that would deal with configuration files.

I kept adding shops, improving the blog, improving the service and so on. As said earlier the bot was built on a pile of tech debt (just rudimentary error handling, no concurrent crawling, etc.) and at this point my technical todo list was pretty much endless.

But at this point I realized an important thing: nobody, except me, cared about the tech debt. Yes, having to physically go to the server and check the log files for errors every day was annoying - but that was just on my end. This realization would lead to an important lesson from this project we'll talk more about later.

Version 0.4 - Closed source

With version 0.4 I set the GitHub repository to private which meant the bot was no longer open source because more and more people kept using the bot at a complete ridiculous polling rate (like one second) which was something I didn't want to support.

In mid-November I posted an update about the future of this project. As I've spend an insane amount of time working on it and already have been able to purchase my GPU at this point I thought it was time to announce its end. The plan at this point was to take it down by the end of December - hoping that the stock situation of GPUs and consoles would stabilize until this point. But, as always with this project, little did I know...

In early December I ran a total of four bots (for RTX 3000, PlayStation 5, RX 6000 and Ryzen 5000 processors) and my bots had a cumulated amount of around ~10.000 followers - an already insane amount which was going to drastically increase within the next days.

A few days into December some friends suddenly started to send me links from reputable IT magazines with "you made it into the news". I was completely baffled and had no idea what was happening. With that the motivation of my little weekend project was reinforced again. Within a few days the cumulated amount of followers tripled and I knew: taking down the bot by the end of December wouldn't be the smartest thing to do. Instead, my todo list grew longer again.

Version 0.11 - The Discord server

Something I've completely neglected until this point was Discord, even though a lot of people requested it. I had so much work with the bot alone that I spend entire weekends solely working on the bot - and having a dedicated Discord server would increase the required maintenance even more.

But with 0.11 the bot got to a point where weekly maintenance was sufficient which allowed to make my life a bit harder again. Having absolutely no experience in how Discord actually works (except my personal server with like seven people) there was only one thing to do: implement Discord notifications for the bot, start a server and announce it to ~30.000 people.

Within minutes the server went from 0 to more than a thousand members. Quickly I realized my first major mistake: my channels did not specify a specific language, and all of a sudden hundreds of people were talking english, german, french and probably klingonian in the same channels.

As time passed by I fixed all my rookie mistakes and the server kept growing. The server kept growing and I realized: I should have started a Discord server right at the beginning of this project.

The Discord server was (and still is) such a perfection complement to the bot. Shops that aren't tracked by the bot are monitored by thousands of Discord members which would mean the ilox project wouldn't miss a single restock - no matter where.

After this realization I started implementing additional features for the Discord server like a dedicated Discord bot which gave the community the capability to post to the Twitter accounts to sync up the dynamic Discord server with the static Twitter accounts. Thanks to an amazing community the Twitter account was now able to post things that weren't even tracked by the bot.

To this very day the Discord server is publicly available and completely free. Even though it caused a shit ton of additional work, I'm incredibly happy with the community that formed on this server - and hugely thankful for all the community drop posters, my amazing mod team and all the people that selflessly helped others in getting new hardware.

Lessons learned

Throughout the months I've learned a lot. Fast growth and the temporary nature of the project (since scarcity is not meant to last) lead to a lot of good things - and things I've probably done wrong.

Shipping is more important than perfectionism

While being a perfectionist most of the time I completely neglected the aspect of striving for perfection on this project - which, in my opinion, made the entire project possible in the first place.

To this very day the bots error handling is subpar at best, lots of duplicated code and I could probably find better solutions in terms of architecture within a few minutes.

Every time I worked on the bot I knew these things. When I saw my IDE complain about "duplicated code!" I had this brief "I should refactor that"-moment - but I simply had no time for that because there were so many other things to do.

But the important thing is: nobody, except me, really cared. People cared about the result. Supporting new shops is great, tracking more products is great. But removing duplicated code? Improving the architecture? Nobody gives a shit.

Not that I've changed my mind about shipping high-quality software - but sometimes the result is more important that the good feeling a proper architecture or decent implementation gives you. Not that the implementation is particularly bad - but it could be a lot better.

Shipping was a lot more important than striving for perfectionism - which is something I miss in a lot of my other projects and I'll try to implement more often from now.

Not everyone is on your side - no matter what you do

When I started this project it had no "business" side to it. I just wanted to help people. Adding affiliates later was a "win-win", since it didn't change prices for customers and simultaneously supported me.

But no matter how good my intention was - some people didn't like what I do. I got tons of hateful or negative messages, people accused me of things I had absolutely no influence on and constantly told me to stop my work and so on.

My personal favorite to this day was a message I got on Twitter from some random internet citizen that told me: "Because of you my son will get no PS5 for christmas!"

I cared a lot about all this negativity in the first place. I just couldn't get my head around the idea of people seemingly hating me for something I do in order to help them.

But at some point I realized: no matter what I'd do - some people will not like me or my work. And that's fine. I stopped giving a fuck about every single message and started to view things from a different perspective.

Reading Twitter replies nowadays is pretty funny: I'm not the idiot who's replying to every single Tweet - made by an automated piece of software that couldn't care less - of an account I don't even want to follow. Accusing a bot that has literally zero intelligence of spreading fake news and trolling just doesn't put you in the perspective of telling me how stupid I am I guess.

Trying to read and reply to all comments was a bad idea. Not giving a fuck about negativity was a good idea.

Affiliate marketing sucks

While affiliate marketing is a valid way of generating money it also can suck quite hard. Some affiliate partners cancelled my partnership without ever telling me why, some never replied and others treat you like dogshit.

Amazon, for example, cancelled my partnership one month after accepting my application because it's against their terms of service to spread affiliate links via client-side software. It took more than 20 mails to explain them that my bot doesn't require client-side software - but in the end they told me that their "decision is final and that all my commission is gone". Two months after accepting this bullshit they suddenly re-opened my account, restored my commission and apologized. To this very day I have no idea what happened there.

Other affiliates took more than half a year to even reply to my application just to tell me I'm in the wrong country which isn't stated in their terms of service once.

Since the idea behind this project always was to help people and making money from affiliate marketing was more of accompanying phenomenon most shops my bot tracks are no affiliate partners. More accurately the only remaining affiliate partner to this day is Amazon (while still supporting more than 20 shops).