Servers & HostingAdvanced

Website Opens Normally in Browsers but Returns HTTP 500 to Bots, Crawlers or External Services – Complete Technical Guide to WAF, ModSecurity, User-Agent, Bot Protection, CDN, Cache, PHP, APIs and Server Logs

A confusing website problem occurs when a page or endpoint opens perfectly in normal browsers such as Google Chrome, Microsoft Edge, or Mozilla Firefox, but ...

Bison Technical Team Enterprise IT specialists

Updated 18 Aug 2026 23 min read 0 total views

A confusing website problem occurs when a page or endpoint opens perfectly in normal browsers such as Google Chrome, Microsoft Edge, or Mozilla Firefox, but automated systems receive an error such as:

HTTP 500 Internal Server Error

This situation can affect many types of web resources, including:

PHP pages
XML sitemaps
RSS feeds
REST APIs
AJAX endpoints
Webhooks
File download URLs
Search pages
Login endpoints
Cron-trigger URLs
Monitoring endpoints
Dynamic images
JSON services
Search-engine crawlers

At first, administrators often assume that the PHP code itself is broken.

However, if the same URL works normally for human visitors but fails only for bots, crawlers, API clients, monitoring tools, or other automated software, the actual problem may be somewhere else.

The difference may come from:

User-Agent handling
IP reputation
Web Application Firewall rules
ModSecurity
Bot protection
CDN security
Rate limiting
Request headers
Server cache
PHP OPcache
Hosting resource limits
Reverse proxy rules
PHP-FPM limits
Database behaviour
Security false positives

This article explains how to diagnose this type of problem systematically.

1. What Does HTTP 500 Internal Server Error Mean?

HTTP status code 500 means that the server encountered an unexpected condition while processing the request.

A browser or client may display:

500 Internal Server Error

or:

HTTP ERROR 500

The important point is that HTTP 500 is a server-side failure.

It does not automatically tell you whether the fault is in:

PHP
MySQL
Apache
Nginx
LiteSpeed
PHP-FPM
.htaccess
CDN
WAF
ModSecurity
Bot protection
Hosting configuration
Application code

Further investigation is required.

2. Why Can the Same URL Work in a Browser but Fail for a Bot?

A browser and an automated crawler do not necessarily send identical HTTP requests.

For example, a normal browser may send:

GET /page.php HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 ...
Accept: text/html,...
Accept-Language: en-US,en;q=0.9

An automated client may send:

GET /page.php HTTP/1.1
Host: example.com
User-Agent: ExampleBot/1.0
Accept: */*

The URL may be identical, but the request is not.

Security and application systems may examine:

User-Agent
Source IP
HTTP method
Headers
Cookies
Referrer
Request frequency
Geographic origin
Query parameters

As a result:

Same URL ≠ Same server treatment

3. User-Agent Is One of the First Things to Check

The User-Agent header identifies the requesting client.

Examples include:

Mozilla/5.0
Googlebot
Bingbot
curl
Python-requests
RSS Reader
SEO crawler
Monitoring service

Some websites or security systems treat different User-Agents differently.

For example:

Chrome        → HTTP 200
Firefox       → HTTP 200
Edge          → HTTP 200
Googlebot     → HTTP 200
Unknown bot   → HTTP 500

If this pattern appears, User-Agent-based filtering or a security rule may be involved.

4. What Is a Web Application Firewall?

A Web Application Firewall, or WAF, inspects HTTP requests before or while they reach the web application.

A common traffic path is:

Visitor
   ↓
CDN
   ↓
WAF
   ↓
Web Server
   ↓
PHP
   ↓
Database

A WAF may check:

Request URL
HTTP method
User-Agent
Query string
POST body
IP reputation
Request rate
Attack patterns
Suspicious headers
SQL injection signatures
Cross-site scripting patterns

If a request matches a security rule, it may be blocked or handled differently.

5. What Is ModSecurity?

ModSecurity is a widely used web application firewall technology.

Many shared-hosting providers use ModSecurity or similar security systems.

It is designed to detect attacks such as:

SQL injection
Cross-site scripting
Remote code execution
Directory traversal
Malicious file access
Suspicious bot activity

However, like any automated security engine, false positives can occur.

A legitimate automated request may sometimes match a rule that was intended to block malicious traffic.

6. Bot Protection Can Cause Request-Specific Problems

Modern hosting and CDN platforms often include bot protection.

These systems try to distinguish between:

Human user

and:

Automated software

That is useful because many attacks are automated.

However, legitimate software is automated too.

Examples include:

Googlebot
Bingbot
RSS readers
Uptime monitors
SEO audit tools
API clients
Backup services
Webhook systems
Monitoring agents

An aggressive bot-protection rule may accidentally interfere with legitimate automated requests.

7. HTTP 500 Is Not the Only Possible Security Response

Security systems can return many different status codes.

For example:

403 Forbidden

may indicate explicit blocking.

429 Too Many Requests

may indicate rate limiting.

503 Service Unavailable

may indicate server overload or temporary protection.

500 Internal Server Error

may indicate a PHP/server failure or an improperly handled security event.

Therefore, do not assume that every blocked bot will receive HTTP 403.

8. CDN Can Affect Requests

A Content Delivery Network such as Cloudflare or a hosting provider's own CDN may sit between visitors and the origin server.

The request path becomes:

Client
   ↓
CDN Edge
   ↓
Security Layer
   ↓
Origin Server

The CDN may perform:

Caching
Bot filtering
Rate limiting
Firewall filtering
DDoS protection
Header modification
IP reputation checks

Therefore, a failure seen by one external system may originate before the request even reaches PHP.

9. Browser Cache and Server Cache Are Different

When troubleshooting, people often say:

"I cleared the cache."

But there can be multiple cache layers.

For example:

Browser Cache
     ↓
CDN Cache
     ↓
Reverse Proxy Cache
     ↓
Hosting Cache
     ↓
Application Cache
     ↓
PHP OPcache

Clearing Chrome cache only clears the browser's local cache.

It does not necessarily clear:

CDN cache
Host-level cache
PHP OPcache
Reverse proxy cache
Security cache
Bot reputation data

10. PHP OPcache

PHP OPcache stores compiled PHP code in memory.

This improves website performance because PHP does not need to compile the same script repeatedly.

Normally OPcache works transparently.

However, during troubleshooting it is useful to remember that application code can be cached at the server level even after a file is modified.

Hosting providers usually handle this automatically, but not always immediately.

11. PHP Errors Can Still Be the Cause

Even when a page works in a browser, PHP can fail under particular request conditions.

Common examples include:

PHP Fatal error

Uncaught Exception

Call to undefined function

Allowed memory size exhausted

Maximum execution time exceeded

Too many connections

The request itself may trigger a code path that normal browser requests do not.

12. Database Errors Can Be Intermittent

Dynamic pages often rely on MySQL or another database.

A PHP page may execute:

$result = $conn->query($sql);

Possible failure conditions include:

Temporary database outage
Connection limit reached
Slow query
Lock contention
Timeout
Invalid query under certain parameters
Missing data
Character-set problems

Therefore, intermittent HTTP 500 errors can sometimes be caused by database conditions rather than permanent coding errors.

13. Hosting Resource Limits

Shared hosting often imposes limits such as:

CPU time
PHP processes
PHP workers
Memory
Entry processes
Concurrent connections
I/O usage

If the website temporarily exceeds a limit, some requests may fail while others continue to work.

This can produce behaviour such as:

Normal browser request → works
Monitoring request     → fails
Next request            → works again

14. PHP-FPM Worker Limits

Web servers often use PHP-FPM to execute PHP scripts.

If all PHP workers are busy, additional requests may wait or fail depending on configuration.

Possible symptoms include:

Slow PHP pages
Intermittent HTTP 500
HTTP 502
HTTP 503
Timeout errors

The hosting provider's server logs are usually required to confirm this.

15. Reverse Proxy Behaviour

A reverse proxy may sit in front of the web server.

Examples include:

Nginx
LiteSpeed
CDN proxy
Load balancer

The proxy may apply its own:

Timeout
Request-size limits
Header restrictions
Rate limiting
Caching
Security rules

Therefore, not every HTTP 500 necessarily comes directly from PHP.

16. API Endpoints Are Commonly Affected

An API endpoint might work when opened manually in a browser:

https://example.com/api/status.php

but fail when called by an application.

Possible reasons include:

Different method such as POST instead of GET
Missing header
Invalid JSON
Authentication differences
Rate limit
WAF rule
User-Agent filtering

Always compare the complete requests, not only the URL.

17. Webhooks Can Fail Even When the URL Opens

A webhook destination may open normally in a browser but fail when the remote service sends a POST request.

For example:

Browser:
GET /webhook.php
→ 200 OK

Remote service:
POST /webhook.php
→ 500 Internal Server Error

This may be because the PHP code processes POST data differently.

Therefore:

Browser test alone does not fully validate a webhook.

18. RSS Feeds Can Show the Same Pattern

An RSS feed such as:

https://example.com/rss.php

may open in normal browsers but fail for a feed reader or crawler.

Potential reasons include:

User-Agent rule
WAF
Content negotiation
Bot protection
Server security rule
Encoding error triggered by one article
Database issue

RSS is only one example of the broader problem.

19. XML Sitemaps Can Also Be Affected

A sitemap may be dynamically generated using PHP:

https://example.com/sitemap.php

Search engines may request it differently from normal browsers.

If the sitemap works in the browser but Search Console reports a server error, investigate:

Googlebot access
WAF rules
Rate limits
Server logs
Content-Type
XML validity

Do not assume the sitemap is valid merely because it displays in Chrome.

20. AJAX Requests Can Fail Differently

AJAX requests may include headers such as:

X-Requested-With: XMLHttpRequest

and may request JSON rather than HTML.

A PHP page that works directly in a browser might still fail when called asynchronously by JavaScript.

This is another example of why request context matters.

21. Cron URLs Can Behave Differently

Some websites use URL-based cron tasks.

Example:

https://example.com/cron.php?key=ABC123

If a monitoring service calls this URL automatically, security software may classify the request differently.

This can produce a bot-specific failure even though manually opening the URL works.

22. Download URLs May Trigger Security Rules

File-download endpoints can sometimes trigger hosting security systems.

Examples:

download.php?id=125
export.php?format=csv
backup.php?file=...

Security systems may inspect query parameters and block requests containing patterns that resemble attacks.

23. Search Pages Are Particularly Sensitive

Search endpoints often accept arbitrary text.

For example:

search.php?q=windows+server

Security systems may inspect the query value for SQL injection or XSS patterns.

A legitimate search-engine crawler requesting unusual query parameters may therefore trigger a false positive.

24. Authentication Can Change the Result

A browser may have:

Login cookie
Session cookie
CSRF token
Remember-me token

An automated client usually does not.

Therefore, a URL that works in your logged-in browser may fail for external software.

Always test in:

Incognito mode
Logged-out mode
Another device

25. Cookies Can Affect PHP Behaviour

PHP applications often use sessions.

A browser may send:

PHPSESSID=...

while a crawler does not.

Application code may incorrectly assume that a session exists.

This can lead to request-specific failures.

26. Source IP Can Matter

Security systems often evaluate IP reputation.

One IP may be allowed while another is challenged or blocked.

For example:

Office ISP      → 200
Mobile network  → 200
Bot data center → 500

This can make a website appear healthy locally while still failing for external services.

27. Geographic Filtering

Some security systems apply country-based rules.

If an external monitoring service operates from another country, its request may be treated differently.

This can affect:

APIs
Monitoring services
Search crawlers
Remote integrations

28. Rate Limiting

Automated services often make repeated requests.

A WAF or application may enforce a limit such as:

100 requests per minute

After the limit is exceeded, requests may fail.

Normally the correct response is HTTP 429, but poorly configured systems may produce other errors.

29. Request Method Matters

Always check whether the request is:

GET
POST
PUT
DELETE
HEAD
OPTIONS

A browser address bar normally sends GET.

An API client may send POST.

A monitoring service may send HEAD.

If PHP code expects GET only, other methods may behave differently.

30. HEAD Requests Can Reveal Hidden Problems

Monitoring tools frequently use:

HEAD /page.php

instead of:

GET /page.php

A badly configured application or server may not handle HEAD requests properly.

Testing only in a browser will not reveal this.

31. Request Headers Should Be Compared

Compare successful and failing requests.

Important headers include:

User-Agent
Accept
Accept-Encoding
Accept-Language
Content-Type
Authorization
Cookie
Referer
Origin

Differences may explain why the server behaves differently.

32. Use curl for Testing

A simple test:

curl -I https://example.com/page.php

This shows response headers.

For a full request:

curl https://example.com/page.php

33. Test With a Browser User-Agent

Use:

curl -A "Mozilla/5.0" -I https://example.com/page.php

Then test with:

curl -A "TestBot/1.0" -I https://example.com/page.php

If results differ, investigate bot filtering or application logic.

34. Test GET vs HEAD

Compare:

curl -I https://example.com/page.php

and:

curl -X GET -I https://example.com/page.php

If one fails and the other works, method handling may be involved.

35. Test From Another Network

Try the URL from:

Office broadband
Mobile hotspot
Another ISP
VPN
Remote server

This can help identify IP-specific filtering.

36. Check Browser Developer Tools

Open:

F12 → Network

Reload the page and inspect:

Status code
Request headers
Response headers
Redirects
Response time

This provides more information than merely seeing the page load.

37. Check Access Logs

Access logs can show:

Date/time
URL
Status
User-Agent
Source IP
Method
Response size

Example:

GET /page.php HTTP/1.1 200

versus:

GET /page.php HTTP/1.1 500

Compare the two entries.

38. Check PHP Error Logs

Error logs are often the most important evidence.

Look for:

PHP Fatal error
PHP Warning
Uncaught Exception
Memory exhausted
Timeout
mysqli_sql_exception

Match the timestamp with the failed request.

39. Check WAF or ModSecurity Logs

If available, look for:

Rule ID
Request URL
Source IP
Message
Severity
Action

A blocked request may reveal a specific ModSecurity rule.

Do not disable the entire WAF unless you understand the security impact.

40. Check Hosting Logs

Hosting panels may provide:

Access logs
Error logs
CPU usage
Entry process usage
PHP usage
Resource limit events

On shared hosting, these can be critical for diagnosing intermittent errors.

41. Do Not Leave display_errors Enabled in Production

Developers sometimes temporarily use:

ini_set('display_errors', '1');
error_reporting(E_ALL);

This can help during testing.

However, public error messages may reveal:

File paths
SQL information
Server configuration
Code details

Use error logs instead for production environments.

42. Check .htaccess

Apache-compatible hosting may use .htaccess.

Rules may include:

Redirects
Rewrite rules
Bot blocking
IP restrictions
User-Agent rules
Security directives

A problematic rule could affect some requests but not others.

43. Look for User-Agent Blocking Rules

Example:

RewriteCond %{HTTP_USER_AGENT} bot [NC]
RewriteRule .* - [F,L]

A broad rule like this can accidentally block legitimate crawlers.

Avoid overly general bot-blocking rules.

44. Check Security Plugins

CMS platforms may have security plugins that perform:

Bot blocking
Rate limiting
Country blocking
IP blocking
User-Agent filtering

The web server may be healthy while a plugin rejects the request.

45. CMS Caching Can Also Interfere

WordPress and other CMS systems often use caching plugins.

Possible problems include:

Stale cache
Incorrect cache key
Bot-specific caching
Dynamic page cached incorrectly

Exclude truly dynamic endpoints from inappropriate caching when necessary.

46. API Authentication Errors Should Not Become 500

If authentication fails, the application should ideally return:

401 Unauthorized

or:

403 Forbidden

If instead it returns HTTP 500, application error handling should be improved.

47. Error Handling Matters

Application code should handle failures cleanly.

For example:

if (!$result) {
    http_response_code(500);
    exit('Temporary server error');
}

Production systems should also log the actual internal error.

48. Do Not Expose Database Errors to Visitors

Avoid displaying raw MySQL errors publicly.

Instead:

Log detailed error internally
Return generic error externally

This improves security.

49. Check Character Encoding

Some failures may occur only when certain database content is processed.

Examples include:

Emoji
Smart quotes
Non-English text
Invalid UTF-8
Special XML characters

This is especially relevant for:

RSS
XML
JSON
APIs

50. XML Requires Proper Escaping

Dynamic XML must escape special characters.

For example:

htmlspecialchars($value, ENT_XML1, 'UTF-8');

Without proper escaping, one article containing & or < could break the feed.

51. JSON Endpoints Have Similar Requirements

JSON APIs should use:

header('Content-Type: application/json; charset=utf-8');

and safely encode data using:

json_encode(...)

A malformed record can cause problems if encoding errors are not handled.

52. Check Memory Usage

A dynamic endpoint may load large amounts of data.

If PHP reaches its memory limit:

Allowed memory size exhausted

HTTP 500 may result.

Reduce query size or increase memory only if justified.

53. Check Execution Time

A slow operation may exceed:

max_execution_time

This can affect:

Reports
Exports
Large XML feeds
Database queries
API processing

54. Check SQL Query Performance

Use indexes where appropriate.

A slow query may work when server load is low but fail during peak periods.

55. Check Database Connection Limits

If the database reaches maximum connections, PHP may fail intermittently.

Symptoms may appear random.

Hosting logs are important here.

56. Check DNS and Proxy Configuration

If a CDN or proxy is used, confirm that:

DNS points correctly
SSL works
Origin is reachable
Proxy settings are correct

A proxy misconfiguration can produce server errors.

57. Check SSL/TLS

Automated tools may enforce TLS rules more strictly than browsers.

Check:

Valid certificate
Complete certificate chain
Supported TLS versions
Correct hostname

TLS errors normally do not present as HTTP 500, but reverse proxies can sometimes translate backend failures.

58. Check Redirect Chains

A browser may automatically follow redirects.

Some bots may not.

Example:

http://example.com
→ https://example.com
→ https://www.example.com
→ /page.php

Too many or incorrect redirects can affect automated clients.

59. Confirm Canonical Hostname

Avoid inconsistent combinations such as:

example.com
www.example.com
http
https

Choose one primary host and redirect others consistently.

60. robots.txt and HTTP 500 Are Different Issues

robots.txt may tell compliant crawlers not to access a path.

Example:

User-agent: *
Disallow: /private/

But this is not the same as the server returning HTTP 500.

Do not confuse crawl policy with server failure.

61. Search Engines Need Reliable Server Responses

Repeated HTTP 500 errors can affect crawling.

If a search engine repeatedly sees server errors, it may reduce crawl activity temporarily.

Therefore, public endpoints should return stable HTTP responses.

62. Monitoring Services Are Valuable

Uptime monitoring can detect intermittent server errors that you may not see manually.

Useful checks include:

Homepage
API endpoint
Sitemap
Login page
Important PHP endpoint

63. Monitor From Multiple Regions

Some monitoring tools support multiple geographic locations.

This can reveal:

India → 200
Europe → 500
USA → 200

which suggests routing or security differences.

64. Security False Positives Are Real

A security system may incorrectly classify a legitimate request as malicious.

This is known as a false positive.

The correct solution is not to disable security entirely.

Instead:

Identify the exact rule.
Verify the request is legitimate.
Create the smallest safe exception.
Retest.

65. Do Not Whitelist Unknown Bots Blindly

If an unknown automated client fails, do not immediately whitelist it.

First verify:

Service identity
IP range
User-Agent
Purpose

Security exceptions should be narrowly scoped.

66. Verify Search-Engine Crawlers Properly

Do not trust a User-Agent string alone.

Anyone can send:

User-Agent: Googlebot

When security decisions matter, verify official crawler identity using the search engine's documented verification methods.

67. Be Careful With IP Blocking

Blocking entire data-center ranges may interfere with:

Search engines
Monitoring services
APIs
Cloud integrations

Use precise rules whenever possible.

68. Check Rate Limits Before Raising Them

If a legitimate service is hitting a rate limit, first confirm that its request volume is expected.

Do not simply remove rate limiting globally.

69. HTTP 500 Should Be Investigated Even If Intermittent

A site that works 99% of the time can still have a real problem.

Intermittent HTTP 500 errors may indicate:

Worker exhaustion
Database limits
Memory limits
Security rules
Server overload
Application exceptions

70. Repeated Code Changes Can Make Troubleshooting Worse

Do not modify several components at once.

Avoid simultaneously changing:

PHP
SQL
.htaccess
WAF
Cache
CDN
DNS

Change one thing at a time and retest.

71. Establish a Known-Good Baseline

If an endpoint works correctly in multiple normal browsers, save the working code before making changes.

This gives you a rollback point.

72. Back Up Before Editing Production Files

Before modifying:

page.php
api.php
rss.php
.htaccess
config.php

create a backup copy.

For example:

rss.php.backup

Store it securely and avoid leaving sensitive backups publicly accessible.

73. Do Not Keep Backup PHP Files Publicly Executable

Files such as:

config-old.php
db-backup.php
test.php

may expose sensitive information.

Place backups outside the public web root where possible.

74. Avoid Debug Files in Production

Temporary diagnostic scripts such as:

phpinfo.php
testdb.php
debug.php

should be removed after troubleshooting.

They can reveal valuable information to attackers.

75. Compare Successful and Failed Requests

This is one of the strongest troubleshooting techniques.

Create a comparison:

Successful request:
IP:
User-Agent:
Method:
Headers:
Status:
Time:

Failed request:
IP:
User-Agent:
Method:
Headers:
Status:
Time:

Look for differences.

76. Test the Exact Endpoint

Do not assume the homepage proves the whole site works.

A website may return:

Homepage → 200
RSS      → 500
API      → 500
Sitemap  → 200

Each endpoint should be checked separately.

77. HTTP 200 Does Not Guarantee Correct Content

A server may return HTTP 200 but display:

Database error

or an empty page.

Always inspect both:

Status code
+
Response content

78. HTTP 500 Does Not Always Mean Permanent Failure

A temporary resource condition may produce a short-lived 500.

Retest and correlate with logs.

79. Error Timestamp Is Critical

Record the exact date and time when the failure occurs.

For example:

18 August 2026
15:32 IST

Then search logs around that timestamp.

This makes diagnosis much faster.

80. Hosting Support Needs Evidence

When contacting hosting support, provide:

Affected URL
Exact time
Status code
Source IP if available
User-Agent
Whether browser works
Whether curl fails
Relevant error-log entry

This is far better than simply reporting:

My website sometimes gives error.

81. Example Hosting Support Request

A useful support request might say:

The URL works normally in Chrome, Edge and Firefox but some automated requests receive HTTP 500.

Please check the corresponding server, PHP, WAF and ModSecurity logs at the supplied timestamp and confirm whether any security or resource rule is being triggered.

82. Common Root Causes

The most common causes of browser-versus-bot differences include:

WAF rule
ModSecurity false positive
Bot protection
IP reputation
Rate limiting
Request method
Missing header
Session dependency
Resource limit
PHP exception
Database issue

83. Recommended Troubleshooting Sequence

Use this order:

1. Test in normal browser
2. Test in private/incognito mode
3. Test from another browser
4. Test another network
5. Check HTTP status
6. Test with curl
7. Compare User-Agents
8. Compare GET/HEAD/POST
9. Check access logs
10. Check PHP error logs
11. Check WAF/ModSecurity logs
12. Check hosting resource limits
13. Check CDN
14. Check cache
15. Change code only when evidence points to code

84. Practical Example: RSS

Suppose:

/rss.php

opens normally in several browsers and produces valid XML.

However, an external crawler receives HTTP 500.

Do not immediately rewrite the RSS generator.

Instead check:

User-Agent
WAF
access logs
PHP logs
source IP
bot protection
rate limit
hosting resource usage

RSS is only one application of this broader troubleshooting method.

85. Practical Example: Sitemap

If:

/sitemap.php

opens in browsers but Search Console reports a server error, compare crawler access and server logs.

Also verify:

Content-Type: application/xml

and valid XML.

86. Practical Example: API

If:

/api.php

works when opened manually but an application receives HTTP 500, compare:

Method
Headers
Authentication
JSON
WAF
Rate limits

87. Practical Example: Webhook

If a webhook endpoint works with GET but fails with POST, inspect POST-processing code and security rules.

88. Practical Example: Monitoring Tool

If an uptime monitor receives HTTP 500 but users do not, test from the monitor's region or IP range and inspect security logs.

89. Security Should Not Be Weakened Without Evidence

Do not solve the issue by globally disabling:

WAF
ModSecurity
Firewall
Bot protection

unless absolutely necessary and properly assessed.

Security features exist for a reason.

90. Use Narrow Exceptions

If a legitimate request is confirmed as a false positive, create the smallest possible exception.

For example:

Specific URL
Specific rule ID
Specific trusted IP

rather than disabling protection site-wide.

91. Production Error Handling Should Be Clean

Visitors should receive a generic message.

Administrators should receive detailed logs.

This is safer than exposing internal information publicly.

92. Log Important Failures

Applications should log:

Database failure
API exception
Authentication error
External service failure
Unexpected input

Logs are essential for diagnosing intermittent problems.

93. Use Health-Check Endpoints Carefully

A dedicated health endpoint can help monitoring systems verify server availability.

For example:

/health.php

It should perform a minimal safe check and return a simple result.

Do not expose sensitive server information.

94. Monitor Server Resources

Track:

CPU
RAM
PHP workers
Database connections
Disk I/O
Entry processes

Resource exhaustion can produce intermittent errors that look application-specific.

95. Keep Software Updated

Outdated:

PHP
CMS
plugins
web server software

may contain bugs or security problems.

However, always test updates before applying them to production.

96. Avoid Unsupported PHP Versions

Older PHP versions may produce compatibility problems with modern code and libraries.

Use a supported PHP branch appropriate for your application.

97. Test After Every Major Change

After changing:

PHP version
WAF
CDN
hosting
DNS
security rules

test important endpoints again.

98. Document the Resolution

When the root cause is identified, document:

Symptom
Cause
Evidence
Fix
Security impact
Verification result

This helps future troubleshooting.

99. Core Troubleshooting Principle

The most important principle is:

Do not assume that a working browser page and a failed automated request are contradictory. They may be different requests processed by different rules.

Always inspect the complete request path.

100. Final Diagnosis Approach

When a website works for users but returns HTTP 500 to certain external systems, investigate the entire chain:

Client
   ↓
DNS
   ↓
CDN
   ↓
Firewall/WAF
   ↓
Web Server
   ↓
PHP
   ↓
Database
   ↓
Application

Any layer can cause the failure.

The correct solution is evidence-based troubleshooting using logs, request comparison and controlled testing.

FAQ

1. Why does my website work in Chrome but fail for a bot?

Bots can send different User-Agents, headers, methods and originate from different IPs. Security systems may therefore treat them differently.

2. Does HTTP 500 always mean PHP is broken?

No. It can also involve the web server, database, WAF, CDN, hosting limits or other server-side systems.

3. Can ModSecurity cause a false positive?

Yes. Legitimate requests can occasionally match a security rule.

4. Should I disable ModSecurity?

Not without evidence. Identify the exact rule first.

5. Can bot protection block legitimate crawlers?

Yes.

6. Can a CDN cause HTTP 500?

A CDN or reverse proxy can contribute to server-side errors or expose origin failures.

7. Is browser cache responsible for HTTP 500?

A genuine 500 is a server-side response. Browser cache may affect what you see, but it is not normally the root cause.

8. Does clearing browser cache clear PHP OPcache?

No.

9. What is PHP OPcache?

It stores compiled PHP bytecode in memory for better performance.

10. Can MySQL cause HTTP 500?

Yes. Database failures or exceptions can cause PHP requests to fail.

11. What is the best place to find the real cause?

PHP error logs, web-server logs and WAF/ModSecurity logs.

12. How can I test an HTTP response?

Use:

curl -I https://example.com/page.php

13. How do I test a different User-Agent?

Use:

curl -A "TestBot/1.0" -I https://example.com/page.php

14. Can GET work while POST fails?

Yes.

15. Can HEAD fail while GET works?

Yes.

16. Why does a webhook fail although its URL opens?

The browser normally sends GET while the webhook may send POST with a different payload and headers.

17. Can a sitemap work in the browser but fail for Google?

Yes, particularly if crawler access, WAF or server stability differs.

18. Can RSS work in the browser but fail for a feed reader?

Yes.

19. Can an API work manually but fail from software?

Yes, because authentication, headers, methods or rate limits may differ.

20. Can IP reputation affect website access?

Yes.

21. Can hosting limits cause intermittent 500 errors?

Yes.

22. Can PHP worker exhaustion cause errors?

Yes.

23. Can memory limits cause HTTP 500?

Yes.

24. Can execution time limits cause HTTP 500?

Yes.

25. Can malformed UTF-8 cause endpoint errors?

Yes, especially with XML or JSON generation.

26. Can special XML characters break a feed?

Yes if they are not escaped correctly.

27. Should I display PHP errors publicly?

No, not on a production site.

28. What should I do instead?

Log errors privately and display a generic message.

29. Can `.htaccess` cause request-specific problems?

Yes.

30. Can User-Agent blocking rules cause false positives?

Yes.

31. Can a security plugin cause HTTP 500?

Potentially, especially if it handles blocked requests incorrectly.

32. Does robots.txt cause HTTP 500?

Normally no.

33. Can rate limiting cause HTTP 500?

The ideal response is 429, but misconfigured systems may behave differently.

34. Should I whitelist a bot immediately?

No. Verify its identity first.

35. Should I disable WAF globally?

No. Use the narrowest safe exception.

36. Why should I note the exact time of the failure?

It allows you to correlate the request with logs.

37. Can a VPN help diagnose the issue?

Yes, because it changes the request's source IP and often geographic path.

38. Can different countries receive different results?

Yes if geographic filtering or CDN routing is involved.

39. Can cookies make a difference?

Yes.

40. Can login sessions make a difference?

Yes.

41. Can a browser hide a server problem?

A browser may follow redirects, retry requests or use cached content, so inspect developer tools and status codes.

42. Does HTTP 200 always mean the page is healthy?

No. Inspect the actual content too.

43. Should I rewrite PHP every time a crawler reports an error?

No. Gather evidence first.

44. What is the safest troubleshooting method?

Change one variable at a time.

45. Should I back up files before troubleshooting?

Yes.

46. Can backup PHP files be a security risk?

Yes if they are publicly accessible.

47. What should hosting support check?

Server logs, PHP logs, WAF logs, ModSecurity, resource limits and failed request timestamps.

48. Can monitoring help find intermittent errors?

Yes.

49. Should important endpoints be tested separately?

Yes.

50. What is the main lesson?

A website can work for normal users while failing for automated clients because the server may process those requests differently.