This morning, an e-commerce site (built on Laravel and well-developed, hence quite efficient) started showing signs of slowing down. This had also happened a few weeks ago, and we partially managed the situation by increasing the VPS power and freeing up the physical machine from other loads.
An analysis of the nginx log reveals that the server is being bombarded with requests from Bytedance. As often happens in these cases, I attempted to firewall the IPs associated with the bots.
However, as soon as I block one IP, (or entire class) the crawling resumes (violently, almost like triggering a DoS attack) from another IP on another class.
They don't respect the robots.txt file.
The IPs they use online don't match the ones from which the requests originate; they probably constantly acquire and change IP blocks.
It makes me wonder: if everyone online behaved like this, everything would collapse in a matter of minutes.
#webdevelopment #serverissues #DDoS #Bytedance #firewall
Stefano Marinelli /
npub14c…j0nj8
2024-01-04 11:20:01
Author Public Key
npub14calwd6xg349ahf3nnhhyqem2w2e3gs66p7zctz2sna74u3tsddq7j0nj8Published at
2024-01-04 11:20:01Event JSON
{
"id": "ff52ecc0e4d36ca396b949b0ee97f42538e68b3551005d85aee60f34f8d94d7c",
"pubkey": "ae3bf73746446a5edd319cef72033b539598a21ad07c2c2c4a84fbeaf22b835a",
"created_at": 1704367201,
"kind": 1,
"tags": [
[
"t",
"webdevelopment"
],
[
"t",
"serverissues"
],
[
"t",
"ddos"
],
[
"t",
"bytedance"
],
[
"t",
"firewall"
],
[
"proxy",
"https://mastodon.bsd.cafe/users/stefano/statuses/111697408916258875",
"activitypub"
]
],
"content": "This morning, an e-commerce site (built on Laravel and well-developed, hence quite efficient) started showing signs of slowing down. This had also happened a few weeks ago, and we partially managed the situation by increasing the VPS power and freeing up the physical machine from other loads. \nAn analysis of the nginx log reveals that the server is being bombarded with requests from Bytedance. As often happens in these cases, I attempted to firewall the IPs associated with the bots. \nHowever, as soon as I block one IP, (or entire class) the crawling resumes (violently, almost like triggering a DoS attack) from another IP on another class. \nThey don't respect the robots.txt file. \nThe IPs they use online don't match the ones from which the requests originate; they probably constantly acquire and change IP blocks. \n\nIt makes me wonder: if everyone online behaved like this, everything would collapse in a matter of minutes. \n\n#webdevelopment #serverissues #DDoS #Bytedance #firewall",
"sig": "da1bdc8489a7e5f02aff732491f34dbfe1e9c71816206c0b8d99bae46f4592964fc99ea66a6e6ea136e656aff99420c79832ce7c9fbb9876f4e35bf8adf3b375"
}