Create a new match rule against those “flowcount” messages. Fluent-plugin-flowcounter will then emit new messages with metrics for bytes/sec and messages/sec as key/value pairs under a new tag, ex: “flowcount”. AWS Kinesis, Google BigQuery, etc), use Fluentd’s built-in copy plugin to also send those messages to fluent-plugin-flowcounter. For all messages that are forwarded to our typical output destination (ex. In order to answer that operational question, we had to build a creative solution:ġ. While the above metrics for CPU, memory, buffer size, and retries are helpful, none of them allow us to answer one of the most basic questions one would hope to answer: How many messages/sec or bytes/sec of log messages is a given host (or cluster) forwarding? You can determine which process name the dd-agent would match against by running the following snippet: #!/opt/datadog-agent/embedded/bin/python If this integration isn’t working for you, you likely have the wrong process name listed for your Fluentd process. Just configure the Datadog agent (dd-agent) on your host(s) to collect stats on your Fluentd process: # /etc/dd-agent/conf.d/process.yaml This integration is also very easy to setup. We can use this to monitor the CPU & memory overhead of the Fluentd process itself. The Datadog Process Check captures metrics from specific running processes on a system, such as CPU %, memory usage, and disk I/O.

To setup this integration: # /etc/fluent/config.d/src/nf

fluentd.buffer_total_queued_size (gauge).

The Datadog-Fluentd integration provides several metrics from Fluentd: Here are the two primary integrations that we use: Datadog-Fluentd Integration Datadog’s Builtin Integrationsĭatadog provides several built-in, easy to setup integrations that we can use for monitoring Fluentd. This blog post is a write-up of how we monitor Fluentd at scale, using both Datadog and Rollbar.

Since the reliability of Fluentd is crucial to our operations, we have extensive monitoring and alerting around the Fluentd process running on each host.

For part 2, see: Part 2 : Visualizing & Alerting)Īt Twilio, we use the open source log-forwarder Fluentd to forward billions of log messages per day from thousands of instances in Amazon AWS into Google BigQuery.

Datadog process monitoring