To monitor, or not to monitor — that is the question

API monitoring matters more than testing. Here's how to combine active checks, passive listening, and traffic log analysis with the right alerting strategy.

Oct 14, 2024 · 4 min read · Apinizer Team, Operations · Operations

Tags: #api-monitoring · #observability · #uptime · #alerting · #api-gateway

What API monitoring is, why it matters, and how to do it: active / passive monitoring, notifications, and logging.

What do Batman and Heimdall have in common? Both watch their domains so they can intervene when necessary. When you apply the same watchfulness to your APIs and web services — informing responsible parties of problems rather than letting them escalate — that practice is called API Monitoring.

Throughout this discussion, the terms API and Web Service are used interchangeably. Similarly, REST "endpoints" and SOAP "operations" are collectively referred to as "methods."

Ideally, developers would have dedicated superheroes monitoring their APIs and systems. In reality, organizations must implement monitoring themselves rather than waiting for problems to become critical.

What is API monitoring? Is it really necessary?

While developers thoroughly test code before release, testing alone cannot catch all real-world issues. Consider these scenarios:

Unexpected conditions. Clients send data that triggers errors not covered in tests.
Server slowdown. Performance degradation causes timeouts despite correct API logic.
Dependent services. Upstream service failures make your service appear broken.
Network changes. Firewall or network modifications render services unreachable.
Server failure. Infrastructure becomes unable to serve requests.
Shared resource changes. Other applications alter shared databases, breaking methods.

Many web services lack thorough automated testing — relying instead on manual checks. These scenarios are difficult to catch through testing alone.

What is API monitoring?

API monitoring involves continuously verifying that APIs return expected responses within acceptable timeframes, while executing predefined actions (email alerts, SMS notifications, ticket creation) when issues occur — ensuring responsible parties are informed immediately.

Is API monitoring really necessary?

Consider these questions:

Do other organizations, teams, or projects depend on your APIs?
Are your APIs critical to business operations?
Have you experienced outages, timeouts, or unexpected responses?

Answering "yes" to any of these indicates monitoring is necessary — often mandatory. Undetected failures lead to escalations, complaints, and potential loss of trust, reputation, or customers.

How to do API monitoring

API monitoring encompasses two components: problem detection and notification.

Monitoring and troubleshooting

Three primary monitoring approaches exist.

Active monitoring

Send predefined requests at regular intervals to validate API responses against expected behavior.

Step 1 — sending requests:

Call a dedicated endpoint. Add endpoints like /health, /healthcheck, or /status returning HTTP 200. This indicates whether the service is up but cannot detect individual method failures.
Call selected methods. Choose critical methods and define monitoring checks with specific request payloads and expected response validations. A few requests every few minutes have minimal performance impact while catching issues early. Exercise caution when monitoring methods that modify data.
Call methods in sequence. Test complete workflows by calling method sequences, using one method's output as input to the next. More complex and less common, but valuable for specific scenarios.

Step 2 — response validation:

Timeout. Responses must arrive within defined limits (e.g. 30 seconds). Alert immediately on failure or retry several times (3–5) to reduce false alarms from temporary network issues.
HTTP status code. Verify returned codes match expectations. Receiving 500 when expecting 201 signals method failure.
Response content and structure. Beyond status codes, validate body content and structure to catch bugs returning correct codes with incorrect data.

Passive monitoring

Better understood as listening, passive monitoring does not send requests. Instead, you expect monitored systems to initiate contact. This approach monitors systems unable to initiate service calls.

For each monitored system, expose a small web service or endpoint instructing that system to call it periodically (via cron + curl). Continuous requests indicate system health; you may require minimal response content. Missing requests within expected intervals signal system problems.

Monitoring message traffic logs

Inspect logs of incoming and outgoing messages to monitor APIs or clients. Filter traffic by HTTP codes, response time, content, client IP, user, error code, method, and message size. Compare findings to thresholds, detecting anomalies and notifying appropriate personnel.

Notification

Service unreachability, server downtime, method failures, or stopped passive monitoring requests typically cannot be fixed programmatically. The best response is raising alerts and notifying responsible people.

Email or SMS notifications alone may prove insufficient. Creating alerts directly in call and alert management platforms like Opsgenie or PagerDuty proves more effective. Consider also integrating with internal ticketing or logging systems. Design your setup for flexibility, enabling new notification channels to be added easily.

Logging

Store all monitoring, validation, and alert events for practical troubleshooting. You'll need to reference:

When problems occurred
Which services and methods were affected
Request details
Received responses
Why they were flagged as incorrect
Which personnel received notifications

Since logs grow quickly, establish retention policies and purge older records based on storage capacity and organizational policy.

Conclusion

API monitoring is indispensable for maintaining smooth operations in many situations. Rather than building custom monitoring applications, leveraging existing solutions proves more efficient and cost-effective for most organizations.

Apinizer offers an API Monitoring product with direct integration into the Apinizer API Gateway platform. Define API proxies once, then easily create monitoring definitions, convert proxy definitions into monitoring specifications, and reuse analytics queries for monitoring purposes.

All posts · Book a Demo · Read the docs