AI agent instructions
AI agent instructions
Resolving Bot Detection Issues
Bot detection is a common challenge for web agents. This guide provides strategies to bypass these detection mechanisms using Notte’s stealth configuration options. Common Bot Detection Challenges:- Accessing e-commerce sites with anti-bot measures
- Scraping content from news or social media platforms
- Interacting with banking or financial websites
- Accessing sites with geographic restrictions
Stealth Configuration Strategies
1. Proxy Configuration
Proxies are one of the most effective ways to bypass bot detection. Different proxy configurations can help you appear as legitimate traffic from various locations.Using Default Proxies
Notte’s built-in residential proxies are enabled by default (proxies=True), so every session already routes through them unless you set proxies=False:
Country-Specific Proxies
For sites with geographic restrictions, use proxies from specific countries:2. CAPTCHA Solving
Automatic CAPTCHA solving is enabled by default (solve_captchas=True) for sites that use these challenges:
solve_captchas.py
3. Advanced Stealth
For sites with sophisticated bot detection, start the session withadvanced_stealth=True to use Notte’s highest-fidelity browser environment. Advanced stealth is available to approved workspaces; contact us to request access.
Complete Stealth Configuration Example
Here’s a comprehensive example combining all stealth techniques:Troubleshooting Tips
- Start Simple. Begin with basic configurations and gradually add complexity:
- Keep the defaults first (
proxies=True,solve_captchas=True) - Pick a proxy country that matches the target site
- Add custom
user_agentif still detected - Try
advanced_stealth=Trueif your workspace has access
- Keep the defaults first (
- Test Incrementally. Test each configuration change individually to identify what works:
- Monitor for Patterns. Keep track of which configurations work for different types of sites:
- E-commerce sites often respond well to residential proxies
- Social media sites may require specific user agents
- Banking sites may need country-specific proxies
Best Practices
- Rotate Configurations: Don’t rely on a single configuration: it makes it easier to track you
- Monitor Success Rates: Verify which configurations work best for different site types
- Respect Rate Limits: Implement delays between requests to avoid triggering rate limiting
- Keep Configurations Updated: Bot detection methods evolve, so regularly test and update your configurations