Log Management Graylog: How to Ship Data
In few easy steps
This is the continuation of my previous post. In this post, I’m going to explain how do we send data to Graylog. For this, I will be using graypy which is a python wrapper written over a python logger. graypy sends logs in GELF(Graylog Extended Log Format) format a type of log format accepted by Graylog.
Below is the list of log forwarding defined by graypy:
- GELFUDPHandler — It uses UDP to ship logs to Graylog.
- GELFTCPHandler — It uses TCP to ship logs to Graylog.
- GELFTLSHandler — It uses TCP with TLS support to ship logs to Graylog.
- GELFHTTPHandler — It uses HTTP lto ship logs to graylog.
- GELFRabbitHandler — It pushes log to RabbitMQ and Graylog pulls the logs from RabbitMQ.
Let me explain this one by one:
GELFUDPHandler
It uses UDP to send logs to Graylog. The advantage of UDP is that it’s a connectionless protocol, sends the data without affecting the performance of the application. There is no error checking in UDP, if there is a packet drop, it remains un noticed to the application.
Example:
handler = graypy.GELFUDPHandler('<graylog-ip>', 12201) my_logger.addHandler(handler) my_logger.debug('Hello Graylog.')the above example sends a plain string “Hello Graylog” via UDP. we can also send JSON data as well.
Example 2:
handler = graypy.GELFUDPHandler('<graylog-ip>', 12201)
json_data = {"name":"abcd","message":"error"}
my_logger.addHandler(handler)
my_logger.debug(json.dumps(json_data))Now that we know how to send the data, we shall see how to receive this data in Graylog. Assuming that you have Graylog setup ready, in the top right go to `systems->input`, there will be a dropdown to select an input from that select GELF UDP and click `Launch Input`, fill in the necessary details which are pretty straight forward and hit ‘save’
Here on-words for the following handler I’m going to keep the content small as it’s going to be very straightforward to implement and test it out.
GELFTCPHandler
It uses TCP to send logs to Graylog. Here the errors and packet drops can be identified easily as it throws errors if it has connection failure or is unable to send a packet due to various reasons. This definitely has an impact on the application’s performance.
Example:
handler = graypy.GELFTCPHandler('<graylog-ip>', 12202)
my_logger.addHandler(handler)
my_logger.debug('Hello Graylog.')Similarly, you can also send JSON logs like the one I have shared above in UDP.
To set up a TCP input in the top right go to `systems->input`, there will be a dropdown to select an input from that select GELF TCP and click `Launch Input`, fill in the necessary details which are pretty straightforward and hit ‘save’
GELFTLSHandler
It uses TCP with TLS to send logs to Graylog. This is similar to TCP, the only addition is that It requires public and private certs to be attached while creating the TCP input in graylog.
GELFHTTPHandler
This is also way similar to the above just that it uses HTTP to send logs to graylog. Here also there is an option to add TLS certs to enable an extra layer of security.
GELFRabbitHandler
We can also use RabbitMQ as a queue system, use GELFRabbitHandler to send messages to RabbitMQ, and configure your Graylog server to consume messages via AMQP. The advantage of doing this it prevents log messages from being lost due to dropped UDP packets as the packets will be available in the queue.
handler = graypy.GELFRabbitHandler('amqp://guest:guest@localhost/', exchange='logging.gelf')
my_logger.addHandler(handler)
my_logger.debug('Hello Graylog.')You will need to configure RabbitMQ with a gelf_log queue and bind it to the logging.gelf exchange so messages are properly routed to a queue that can be consumed by Graylog.
Note: The queue and exchange name can be changed as per our will.
To set up a RabbitMQ input in the top right go to `systems-> input`, there will be a dropdown to select an input, from that select GELF AMQP and click `Launch Input`, fill in the necessary details which are pretty straightforward and hit ‘save’.
Make sure to have a RabbitMQ setup running before creating an input in Graylog.
There are lot more inputs supported by Graylog which a user can explore based on their requirements.
