A Large Credential Stuffing Attack - How We Respond and Mitigate

Credential stuffing attacks are a growing threat that organizations must be prepared to face in the dynamic world of web cybersecurity. Recently, we at amazee.io needed to tackle such an attack head-on when one of our valued government clients in Australia was targeted. 

In this post, we will share the story of our response and mitigation strategies against an onslaught of 3.7 million requests, providing insights and tactics that helped maintain the integrity and availability of our client's services.

What is a Credential Stuffing Attack?

Before delving into the specifics of our response, it's essential to understand what a credential stuffing attack is and how it operates. 

Credential stuffing is a type of cyber attack where attackers use stolen account credentials (usernames and passwords) obtained from data breaches to gain unauthorized access to user accounts on various platforms. 

This is typically done through automated scripts or bots that attempt to use these credentials on different websites, exploiting the common practice of users repeating passwords across multiple sites. The primary aim is to access accounts fraudulently, often to steal personal data, commit fraud, or spread spam and malware.

The Attack Background

In July 2023, our client faced a formidable credential stuffing attack. Around 13,000 unique IP addresses were used to flood an API endpoint responsible for validating customer email addresses. The attack lasted about 19 hours, involving an astounding 3.7 million requests. Despite the scale of this challenge, our team's rapid response and robust cybersecurity measures ensured no interruption in application availability.

The Challenge

Addressing a credential stuffing attack of this magnitude presented several unique challenges, each requiring careful consideration and a strategic approach:

Volume and Velocity of Requests: The sheer volume of 3.7 million requests from 13,000 unique IP addresses in a span of only 19 hours was colossal. Handling this massive influx without disrupting service was a significant challenge. The high velocity and frequency of requests meant that our systems and defenses had to be robust and responsive in real-time to keep up with the attack's pace.

Distinguishing Between Legitimate and Malicious Traffic: A critical challenge was differentiating between legitimate user requests and those generated by the attack. Any misidentification could lead to blocking genuine users or allowing malicious traffic through.

Rapidly Evolving Attack Patterns: The attackers continuously evolved their strategies, using different IP addresses and varying their request patterns to bypass our defenses. 

Resource Allocation and Management: Ensuring our infrastructure could handle the additional load without compromising performance for legitimate users was crucial. We knew early on that we would have to allocate and manage resources efficiently to bolster our defenses while maintaining optimal service levels for our clients' users.

Coordination and Communication: Effective communication and coordination within our team and with our technology partners would be vital. Ensuring everyone was on the same page and acting swiftly was critical to our successful response. Rapid decision-making and implementation of strategies in a high-pressure environment add to the situation's complexity. 

Post-Attack Analysis and Prevention: We knew that after mitigating the immediate threat, analyzing the attack to understand its nature and prevent future occurrences would be topical. This would involve meticulous examination of logs, patterns, and attack vectors, requiring both time and expertise.

These challenges highlighted the need for a proactive, agile, and multi-layered cybersecurity response. 

Credential Stuffing Attack Mitigation Strategy

Our approach to mitigating this large-scale credential stuffing attack was multifaceted, involving a combination of rapid response, technical expertise, and strategic decision-making.

Initial Assessment and Application Layer Response: As soon as the attack was detected, our team conducted a thorough assessment to understand its scope and the specific target - an API endpoint for email validation. At the application layer, the customer already had in place an immediate defensive measure (in the form of rate limiting).

Implementing Signal Sciences Rate Limit Rule: Recognizing the need for more robust control, we swiftly deployed a new rate limit rule using Signal Sciences. This rule was designed to identify and throttle the rate of requests from IP addresses that exhibited patterns typical of credential stuffing attacks.

This measure effectively reduced the number of malicious requests reaching the application, allowing legitimate traffic to flow unhindered.

Escalating to Fastly's Edge Protection: With the attack persisting, we escalated our response to include Fastly's edge protection services. By moving the defense perimeter to the network edge, we could intercept and filter attack traffic before it even reached our infrastructure.

We utilized Fastly's advanced tools to set up ASN blocks, explicitly targeting the sources identified as part of the credential stuffing attack.

Agility, Dynamic Adaptation, and Continuous Monitoring: Our team continuously monitored the situation throughout the attack, adapting our real-time strategies as the attack patterns evolved. This dynamic approach allowed us to stay ahead of the attackers, rapidly adjusting our defensive measures as needed.

Strategies Validated and Lessons Learned

This incident was not only a test of our cyber defense capabilities but also a valuable learning opportunity:

• Early detection of anomalies and proactive monitoring are crucial in identifying and responding to such attacks. Implementing systems for real-time monitoring and alerting proved invaluable.
• Adapting defense strategies in real-time as the attack evolves is essential. This incident highlighted the need for flexibility and agility in our cybersecurity approach.
• Implementing advanced filtering and rate limiting at different layers (application, network edge) effectively manages high-volume cyber threats. These measures help distinguish between legitimate and malicious traffic, thereby protecting genuine users.
• Having a well-structured incident response plan that can be rapidly executed is critical. This experience underscored the importance of regular updates and drills to our response protocols.
• The role of teamwork and collaboration within the organization and with external partners cannot be overstated. A unified approach was vital to our success in mitigating the attack.
• Conducting a thorough post-attack analysis to understand attack vectors and methodologies is imperative for future prevention and strengthening of defenses.

Successes

Several key achievements characterize the successful management of this large-scale credential stuffing attack:

Maintaining Application Availability: Throughout the 19-hour ordeal, we maintained 100% application availability. This is a significant success, considering the scale and intensity of the attack.

Effective Attack Neutralization: The attack was successfully neutralized with minimal success on the attacker's part. Our timely and effective response strategies played a pivotal role in this accomplishment.

Sustained Protection Post-Attack: The rate limit rules and other defensive measures we implemented continue to protect the application against similar threats. This ongoing protection is a testament to the durability and effectiveness of our cybersecurity strategies.

Efficient Resource Management: Efficient allocation and management of resources ensured that our client's services remained uninterrupted and performed optimally even under the stress of the attack.

Strong Partner Collaboration: Our close collaboration with technology partners and expert support teams contributed significantly to successfully mitigating the attack, highlighting the importance of strategic partnerships in crisis situations.

In facing this large-scale credential stuffing attack, our team demonstrated not just technical proficiency, but also strategic agility. By rapidly deploying a layered defense strategy and leveraging our partnerships, we ensured uninterrupted service for our client. This experience reinforces our commitment to staying at the forefront of application delivery and cybersecurity, ready to protect our clients against the most sophisticated threats.

If you want to learn more about our fully open source application delivery and hosting platform, get in touch with us. We’d love to chat with you.