Ocean implements automatic, extremely scalable and failsafe aggregation of distributed log data. The Log Service collects and stores log data, which can be requested and analysed using the Log Service API.
The distributed logging system has the following properties:
- It can receive and store several million log lines per second, given a suitably dimensioned Redis backend
- It is extremely scalable, as it is lightweight, brokerless and distributed
- Services can join, leave, go down and reappear at any time without incurring wait states or losing messages
- The logging system is redundantly deployed. There is no single point of failure.
- The Log Service API provides means to tabulate, filter and process log data in real time, for instance per application, service, resource, or server instance.
The key technologies to the above functionality are
- ZeroMQ: extremely fast transport layer with built-in messaging, queuing, reconnection, etc.
- Redis: a properly tuned Redis installation can easily receive and store millions of log lines per second.
Each worker process in the bottom row of log servers interfaces directly to Redis, where log data is stored. Each log service instance runs a number of log worker processes (five by default) and one reception queue process which distributes log messages between them. All these processes have supervisor processes which monitor them and restart them if necessary. The worker processes are small and light-weight; a typical log server instance can support many more if necessary.
Every log service instance is also a Rails server implementing a RESTful JSON API used to obtain log excerpts and to manage and process log data in bulk. The Admin web application is used to view log data.
- The LogExcerpt Resource
- an excerpt from the aggregated log, consisting of zero or more log lines ordered chronologically.