A Log Story

Customer said: “We want to log all access details and message content of our Web Services to be able to find out which data we sent to whom, and when. Additionally, only authorized users must be able to see this data. We tried to log everything on our relational database, but soon the data became too big to manage or query. Since it will not be possible to filter the logs, we decided not to use txt files for logging, either. What can we do?”

Problem: Partly for legal reasons, the institute wanted to log all access details and content of Web Services, and be able to filter these logs to find out some specific data. The potentially huge size of data was a big problem in terms of performance of both logging and querying. Restricted access on this log data was another topic to be considered.

What Apinizer Team did: We explained that Apinizer keeps logs in Elasticsearch, and it is scalable so that even full-text search on log content is possible. There is no need to be experienced users on Elasticsearch, because all log configuration and query definitions are done via Apinizer’s form-based user interfaces, agnostic to Elasticsearch. Additionally, customizable logging, querying, reporting and visualization capabilities of Apinizer were also impressive for the institute’s personnel. We showed that Apinizer can backup the log data as well, and all these stuff can be done by authorized users only. After we configured a few APIs, produced load on those APIs to create log content, and built up example queries, features were clear enough to the staff. Finally, we showed them how to handle privacy in log data for sensitive information.

Result: After seeing Apinizer in action and how easy it is to log APIs’ traffic, and filter, report and visualize the logged data, the institute decided to manage all APIs’ logs via Apinizer. As a case study, they created a custom query that will filter all requests/responses to/from selected APIs from a specific IP, and defined a report that will periodically send the results of the query to specified e-mail addresses. We helped them to configure Apinizer to backup log data in a managed shared folder and clear log data from active log database monthly for performance issues. The institute manages all APIs and their logs via Apinizer now.