Logstash

Logstash adalah data processing pipeline yang mengumpulkan data dari berbagai sumber, melakukan transformasi, lalu mengirimkannya ke Elasticsearch sebagai output.

Prerequisites

Komponen	Versi
OS	RHEL / CentOS / Rocky Linux 8+
Java	OpenJDK 11
Logstash	8.x
JDBC Driver	mssql-jdbc 12.8.1.jre8

---

Instalasi (VPS)

Step 1: Install Logstash

# Install Java 11
sudo yum install java-11-openjdk java-11-openjdk-devel

# Import GPG Key Elasticsearch
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# Tambah Elastic Repository
sudo tee /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

# Install Logstash
sudo dnf install logstash

Step 2: Download JDBC Driver

JDBC driver diperlukan agar Logstash bisa terkoneksi ke SQL Server (MSSQL).

# Buat folder vendor
sudo mkdir -p /usr/share/logstash/vendor/jars/

# Download MSSQL JDBC Driver
cd /usr/share/logstash/vendor/jars/
sudo wget https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.8.1.jre8/mssql-jdbc-12.8.1.jre8.jar

# Set permission
sudo chmod 644 /usr/share/logstash/vendor/jars/mssql-jdbc-12.8.1.jre8.jar
sudo chown logstash:logstash /usr/share/logstash/vendor/jars/mssql-jdbc-12.8.1.jre8.jar

# Verifikasi
ls -la /usr/share/logstash/vendor/jars/

Step 3: Konfigurasi

JVM Options — sesuaikan heap size dengan RAM server:

sudo nano /etc/logstash/jvm.options

-Xms512m
-Xmx512m

Pipelines:

sudo nano /etc/logstash/pipelines.yml

/etc/logstash/pipelines.yml

- pipeline.id: products_pipeline
  path.config: "/etc/logstash/conf.d/products.conf"
  pipeline.workers: 1

- pipeline.id: users_pipeline
  path.config: "/etc/logstash/conf.d/users.conf"
  pipeline.workers: 1

- pipeline.id: orders_pipeline
  path.config: "/etc/logstash/conf.d/orders.conf"
  pipeline.workers: 1

- pipeline.id: transactions_pipeline
  path.config: "/etc/logstash/conf.d/transactions.conf"
  pipeline.workers: 1

- pipeline.id: analytics_pipeline
  path.config: "/etc/logstash/conf.d/analytics.conf"
  pipeline.workers: 1

Transfer file konfigurasi ke VPS:

# Dari lokal ke VPS
scp -P 22 ./conf.d/*.conf user@your-server:~

# Pindah ke direktori Logstash
sudo mv ~/*.conf /etc/logstash/conf.d/

# Set permission
sudo chown logstash:logstash /etc/logstash/conf.d/*.conf
sudo chmod 644 /etc/logstash/conf.d/*.conf

# Verifikasi
ls -la /etc/logstash/conf.d/

Systemctl

sudo systemctl daemon-reload
sudo systemctl enable logstash
sudo systemctl start logstash
sudo systemctl status logstash
sudo systemctl stop logstash

# Realtime log
sudo journalctl -u logstash -f

---

Docker

Untuk environment lokal / development, jalankan Logstash via Docker. Pastikan Elasticsearch sudah berjalan sebelumnya.

Explorer

logstash

config

logstash.yml

pipelines.yml

pipeline

products.conf

drivers

mssql-jdbc-12.8.1.jre8.jar

docker-compose.yml

.env

docker-compose.yml

version: "3.8"

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.4
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ./logstash/drivers:/usr/share/logstash/vendor/jars:ro
    environment:
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=${ELASTIC_HOSTS}
    mem_limit: 1073741824

Jalankan:

docker compose up -d

# Log realtime
docker compose logs -f logstash

# Restart
docker compose restart logstash

---

Troubleshooting

# Kill paksa proses Logstash (VPS)
sudo pkill -9 -f logstash

# Cek port Elasticsearch
sudo netstat -tlnp | grep 9200

# Log container (Docker)
docker compose logs logstash --tail=100

# Masuk ke shell container
docker exec -it <container-name> bash