I have just recovered from a DDOS/Brute force attack on my server from a botnet running on Amazon EC2 instances. (Technical)

[Gromett]

Just FYI:
330 open AWS tickets for the period Friday to now with "DDoS" as a keyword {for cases CREATED in that time window}
And I'm not sure how deep I can access AWS IT support tickets

edit: clarity & so ~142 other cases were resolved in that period

I didn't mention DDOS in my ticket. The DDOS was incidental to the attack. My subject line was.
"Abuse from many amazonaws servers"

 

[Gromett]

You have some patience Gromett. You need to give them a prize. Leave them an easy access server with a payload, I sure some one like Sophos would oblige if they knew what you have achieved so far.

Can't do that. they are directly attacking a single URL.

 

[Gromett]

To what extent do you reckon this activity is “State Sponsored”?

It's not. My server has no value to any states.

 

[Gromett]

Why not just subscribe to a cloud edge traffic protection services? Stops all DOS attacks and cleans traffic, so improving the service time for real users. They have the scale for any size of attack.

Also DDOS attacks are ever increasingly being used for very specifically targeted distraction purposes, rather than just disruption purposes. The real breach ‘stuff’ going on in the background whilst the distraction is going on and the attacked party focusing resources and time because of the business impact.

Finally, it’s always about not being the slowest wilderbeast in the herd, which creates protection in itself as the ‘bad’ people are not going to try very hard if the return is poor.

I have DDOS protection on my server by the server provider. I refuse to pay for "edge services" as they are always slower than how I configure my servers directly.
This is their target audience.
visitor -> edge service -> badly configured server

This is my situation.
visitor -> well configured server.

Adding edge services of any kind will just slow down page loads as it is adding an additional step.

This is not a DDOS attack, it is a brute forcing attack that happened to act as a temporary DDOS because of my mitigation actions.

I was successfully blocking their small brute force attack, so they ramped up the brute force attack which caused a minor DDOS for a short period of time while I mitigated it.

I hope that makes sense?

I have been doing this since 1997 (25 years). Dealing with hackers directly on server I own. So I am never concerned about them.
I have stuff in place to deal with everything from DDOS, to brute forcing, to full blown crypto ransom attacks to my data centre being blown up by terrorists. I plan for the very worst possible events and have responses for each of them..
You wouldn't believe some of the things I have dealt with over the years.

 

[Gromett]

Finally, it’s always about not being the slowest wilderbeast in the herd, which creates protection in itself as the ‘bad’ people are not going to try very hard if the return is poor.

Generally you are correct, but this attack was by a script kiddie type. When I successfully blocked them a professional attacker would have ceased attempts and moved on.
Either this guy is dumb as a bag of spanners and can't read the data that is being returned. The server is responding with 403 errors to every single one of his attacks and has been for almost 24 hours now. He is not getting a single 200 response.
The other option which is more likely is that he is either using a script he downloaded and it is either crap or he doesn't know how to configure it correctly. Or he is paying someone for the attacks and they don't care so long as they are getting paid.

A good script would use curl to post the request, then check the response. If the response is not 200, then it should keep a tally and if it makes up all the response for a period of time should pause the attack. After one or 2 retries, it should cease the attack completely as it is pointless and they could point the attacking resources at different target that is perhaps going to have better potential.

^{_{Subscribers  do not see these advertisements}}

 

[Carpmart]

I have DDOS protection on my server by the server provider. I refuse to pay for "edge services" as they are always slower than how I configure my servers directly.
This is their target audience.
visitor -> edge service -> badly configured server

This is my situation.
visitor -> well configured server.

Adding edge services of any kind will just slow down page loads as it is adding an additional step.

This is not a DDOS attack, it is a brute forcing attack that happened to act as a temporary DDOS because of my mitigation actions.

I was successfully blocking their small brute force attack, so they ramped up the brute force attack which caused a minor DDOS for a short period of time while I mitigated it.

I hope that makes sense?

I have been doing this since 1997 (25 years). Dealing with hackers directly on server I own. So I am never concerned about them.
I have stuff in place to deal with everything from DDOS, to brute forcing, to full blown crypto ransom attacks to my data centre being blown up by terrorists. I plan for the very worst possible events and have responses for each of them..
You wouldn't believe some of the things I have dealt with over the years.

Makes absolute sense mate…

Was just thinking that a tiny bit of latency in serving content through the edge protection (and the capacity these providers bring in the arms race with the hackers), versus disruptive brute force, low and slow or old fashioned DDOS, disruption, time to handle etc, could be very acceptable to some people. I appreciate this is your world, just looking at it if I was the commercial owner of a service under attack.

 

[Carpmart]

I’m also way out of touch these day Gromett having not looked at this space for six years now!

Sounds like you’ve had some fun and also achieved the outcome you were after! Mere mortals may need the type of help I suggested, or indeed employ your services!

 

[Gromett]

Makes absolute sense mate…

Was just thinking that a tiny bit of latency in serving content through the edge protection (and the capacity these providers bring in the arms race with the hackers), versus disruptive brute force, low and slow or old fashioned DDOS, disruption, time to handle etc, could be very acceptable to some people. I appreciate this is your world, just looking at it if I was the commercial owner of a service under attack.

A service such as cloudflare which is what I suspect you were thinking about wouldn't have helped in this situation. I would have had to escalate a ticket to them and wait for a response. In the meantime I could have resolved it myself.
The total period of time the side effect DDOS took for me to identify and resolve was 15 minutes.

Here you can see the critical warning from my monitoring system.
***** Nagios *****

Notification Type: PROBLEM

Service: CPU Load
Host: ********
Address: **********
State: CRITICAL

Date/Time: Sun Jan 23 22:30:37 GMT 2022

Additional Info:

CRITICAL - load average: 35.13, 21.20, 14.19

---

Here you can see the back to ok.
***** Nagios *****

Notification Type: PROBLEM

Service: CPU Load
Host: ***********
Address: **********
State: WARNING

Date/Time: Sun Jan 23 22:45:37 GMT 2022

Additional Info:

WARNING - load average: 0.23, 10.87, 17.10

---

I know for a fact that cloudflare for example would have had zero impact on this attack and my response was far quicker.

 

[Millcourt]

It's not. My server has no value to any states.

Apologies Gromett ……. I was not suggesting your server had any value to a malevolent State……. I was just trying to get a better understanding of why such activities take place and if you (or anyone else reading this thread) had any views.

 

[robnchris]

Gromett and myself will never agree on the electric car debate but I take my hat of to him with all this hacker, computer knowledge.

Well done sir, you have my respect.

^{_{Subscribers  do not see these advertisements}}

 

[The Dotties]

Gromett
I you’re fast a binkies, would you receive a warning message, enough to wake you, before any damage was done?

 

[Gromett]

Apologies Gromett ……. I was not suggesting your server had any value to a malevolent State……. I was just trying to get a better understanding of why such activities take place and if you (or anyone else reading this thread) had any views.

In my case the attacker is trying to get access to the admin panel of an ecommerce site. You can imagine what they are trying to get. This is a purely financial play.

State based attackers will be trying to get one of the following.
* access to a companies network to launch supply chain attacks.
* ability to cripple a company should they wish it. Hitting a logistics company for example would affect military supplies.
* Intellectual property. Trade secrets or product designs for example.

Many other reason. But the website I host have no international or national value strategically to a foreign state.

Most "hackers" are not actually hackers but script kiddies. They scatter gun their attacks with no specific targetting. Literally they will buy a bit of software, google for website running the targetted software then let rip.
State actors tend to target a single company and work diligently and covertly without raising any attention.

I hope that makes sense?

 

[Gromett]

Gromett
I you’re fast a binkies, would you receive a warning message, enough to wake you, before any damage was done?

Yes. I got the first warning at 12:59 in the afternoon that something was going off.
I added an additional rule to my fail2ban setup to mitigate the attack shortly after.
By 6pm the attack was mitigated completely and I was seeing zero entries in my logs.

At 21:50 I started getting warnings again, and started monitoring to ensure my mitigations were holding.
They were effective but not fast enough to keep up with the ramp up of their attack, which is why I started seeing the effects as a DDOS (although they hadn't intended a ddos).
The attacks ramped up until it was effectively a DDOS starting at 22:30..
Which is why I made additional efforts and by 22:45 I had shut them down completely although the attack was still ongoing it was having zero effect on me.

I have a monitoring system that checks the servers every 5 minutes across all services and resources. It emails me a message to two email addresses. Any email sent to my phone causes a Klaxon alarm that I cannot sleep through. I can be up and on the computer in 2 minutes (boot time and pee).

 

[kevenh]

I didn't mention DDOS in my ticket. The DDOS was incidental to the attack. My subject line was.
"Abuse from many amazonaws servers"

OK, as half expected, my view of amazon case tickets is only high level & maybe only internals too. Definitely not granular enough to drill into customer raised cases.
nvm, may've been interesting to get some internal comms.
(I've 2 days left at AWS / )

 

[Gromett]

OK, as half expected, my view of amazon case tickets is only high level & maybe only internals too. Definitely not granular enough to drill into customer raised cases.
nvm, may've been interesting to get some internal comms.
(I've 2 days left at AWS / )

I wasn't looking for you to chase it up for me Nice of you to offer though.

You are leaving Amazon? What is your new job? Anything interesting?

^{_{Subscribers  do not see these advertisements}}

 

[kevenh]

I wasn't looking for you to chase it up for me Nice of you to offer though.

You are leaving Amazon? What is your new job? Anything interesting?

The team I've been in at Amazon for the last 5 years is a bit of an oddity. Our products are hardware based. AWS have made console versions of them and now sales of the hardware are too weak to need installation/site support engineers.
I'm taking some time away from employment. It may roll into retirement.

edit: Re. your case, I was intrigued to see if there'd be any inside info to share but as I kind of expected, I don't have the right tools.

 

[Silver-Fox]

I you’re fast a binkies,

Ehhh

The Dotties

 

[The Dotties]

Ehhh

The Dotties

Everybody but you understood
Binkies = sleepies!

 

[artona]

I would have turned it off and back on again

I miss the good old days, when you could just kick something, or use a bigger hammer to get it going again

 

[Gromett]

Everybody but you understood
Binkies = sleepies!

To be fair even after reading it a few time I had no clue what you were talking about there. Later on in the sentence you said this;

enough to wake you

So I presumed you meant alseep

^{_{Subscribers  do not see these advertisements}}

 

[Silver-Fox]

Everybody but you understood
Binkies = sleepies!

Must be an age thing, after all you must be thirty years older than me

 

[Gromett]

This is interesting, they are trying to hit my DNS servers now. Definately into DDOS territory. Shame they don't appear to realise I have firewalled every single one of their servers.

 

[Realist]

DDoS attacks have increased over the last few weeks and their are a few countries involved or one can point fingers at.

Not been in the game over the last few years but I’m a certified system engineer but no longer have the need or wish to go back to that game, been away far to many years.

 

[EML]

It's hard to believe that your client is being attacked from compromised servers if he's just a low-level e-commerce site. It would be expensive. Does the client do anything interesting? Crypto?

Have you checked the IP addresses against the current list of Tor exit nodes? It seems to be fairly common to block both Tor and AWS - see https://library.hud.ac.uk/pages/restrictednetworks/, for example.

I'm also curious that you're running your own DNS server. Seems like it's asking for trouble - I gave this up and started paying for it years ago.

 

[Gromett]

It's hard to believe that your client is being attacked from compromised servers if he's just a low-level e-commerce site. It would be expensive. Does the client do anything interesting? Crypto?

No the client sells a physical product that they manufacture themselves.

Have you checked the IP addresses against the current list of Tor exit nodes? It seems to be fairly common to block both Tor and AWS - see https://library.hud.ac.uk/pages/restrictednetworks/, for example.

No, pretty much every attacking IP is on EC2.

I'm also curious that you're running your own DNS server. Seems like it's asking for trouble - I gave this up and started paying for it years ago.

I have been doing this a very long time. DNS is pretty straight forward if you know what you are doing. I currently manage something like 30 DNS servers and have never had an issue or security breach in 20+ years.

Up until 2015 I ran a hosting company. I had around 100 servers and managed DNS, email, SQL and web services for many thousands of sites. What I do is small scale now, but I still apply the same techniques and principles.

It would be pointless paying someone else to do what I am perfectly capable of doing and doing well.

PS: This is what I do to make a living.

^{_{Subscribers  do not see these advertisements}}

 

[bobnick]

I've got one of those brains that cannot, will not take in anything tech. I read but think this may as well be written in Chinese or that stuff on an Egyption tomb for all i can understand of it.
Am i one of the only survivin'g

 

[Diabalo]

This is interesting, they are trying to hit my DNS servers now. Definately into DDOS territory. Shame they don't appear to realise I have firewalled every single one of their servers.

10 out of 10 for persistance

 

[The Dotties]

.

 

[The Dotties]

Must be an age thing, after all you must be thirty years older than me

You Sir are a damned sight older than 25!

 

[Gromett]

I think this may be the final update. Last night a client tried to email me. He uses a spam filtering service for his incoming and outgoing email (also does end to end encryption) and an email he sent last night didn't arrive.
So I had to update my firewall rules to just block port 80 and 443 rather than block the entire IP address.

This means if an ec2 instance accesses my server for any reason they get blocked on the web server. The DNS server has to remain open for MX record look ups for the email service.

Attacked by an Amazon EC2 hosted botnet (Part 2) DNS server under attack. – Gray.me.uk – Tech and Linux Blog

www.gray.me.uk

I am tempted to change the filter itself to only filter on port 443/80 then do the Log Drop on that. Hmm. some thought is required

^{_{Subscribers  do not see these advertisements}}