Ana içeriğe geç

Olcum Gecikiyor

Semptomlar​

  • Sensor verileri beklenen 5 saniyelik araliklar yerine dakikalarca gecikiyor
  • Bazi cihazlarda veri gelirken diger cihazlarda veri eksik
  • Dashboard grafikleri bosluklu gorunuyor (missing data points)
  • Measurement API yanitlari normalden yavas donuyor

Olasi Sebepler​

  1. Modbus polling suresi fazla cihaz nedeniyle uzamis
  2. MQTT kuyrugu sismis, mesajlar birikmis
  3. TimescaleDB insert performansi dusmus (chunk, index sorunlari)
  4. Celery worker'lari mesgul veya cokmus
  5. Ag gecikmesi — broker ile backend arasi yuksek latency
  6. ESP32 tarafinda polling dongusu yavaslamis (Modbus timeout)

Teshis Adimlari​

1. Modbus Polling Suresini Kontrol Et​

# Modbus poller loglarinda polling cevrim suresini kontrol et
docker logs --tail 200 zeus-modbus-poller 2>&1 | grep -i "cycle\|duration\|poll"

# Ortalama polling suresini hesapla
docker logs --tail 1000 zeus-modbus-poller 2>&1 \
  | grep "poll_duration" \
  | awk -F'=' '{sum+=$2; count++} END {print "Ort:", sum/count, "ms"}'

# Timeout olan cihazlari bul
docker logs --tail 500 zeus-modbus-poller 2>&1 | grep -i "timeout\|error"

2. MQTT Kuyruk Boyutunu Kontrol Et​

# EMQX mesaj kuyruk istatistikleri
curl -s -u admin:public http://localhost:18083/api/v5/stats \
  | python3 -c "
import json, sys
stats = json.load(sys.stdin)
for s in stats:
    if 'queue' in s.get('key','').lower() or 'message' in s.get('key','').lower():
        print(f\"{s['key']}: {s['value']}\")
"

# EMQX subscription sayisi ve mesaj orani
curl -s -u admin:public http://localhost:18083/api/v5/metrics \
  | python3 -m json.tool | grep -i "messages\|delivery\|received"

3. TimescaleDB Insert Performansini Kontrol Et​

# Aktif sorgulari ve surelerini kontrol et
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
SELECT pid, state, query_start, NOW() - query_start AS suresi,
       LEFT(query, 100) AS sorgu
FROM pg_stat_activity
WHERE datname = 'zeus_db'
  AND state != 'idle'
ORDER BY query_start;
"

# Son 1 saatteki insert performansini kontrol et
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
SELECT date_trunc('minute', created_at) AS dakika,
       COUNT(*) AS kayit_sayisi
FROM measurements
WHERE created_at > NOW() - INTERVAL '1 hour'
GROUP BY 1
ORDER BY 1 DESC
LIMIT 20;
"

# Chunk boyutlarini kontrol et
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
SELECT hypertable_name, chunk_name,
       pg_size_pretty(total_bytes) AS boyut,
       range_start, range_end
FROM timescaledb_information.chunks
WHERE hypertable_name = 'measurements'
ORDER BY range_end DESC
LIMIT 10;
"

4. Celery Kuyruk Durumunu Kontrol Et​

# Celery worker durumunu kontrol et
docker exec zeus-backend celery -A app.core.celery.app inspect active

# Kuyruk boyutlarini kontrol et
docker exec zeus-redis redis-cli LLEN celery

# Bekleyen gorev sayisi
docker exec zeus-redis redis-cli LLEN celery:measurement_processing
docker exec zeus-redis redis-cli LLEN celery:default

5. Ag Gecikmesini Olc​

# Backend'den EMQX'e ping suresi
docker exec zeus-backend ping -c 5 emqx

# Backend'den PostgreSQL'e ping suresi
docker exec zeus-backend ping -c 5 postgres

# Redis latency testi
docker exec zeus-redis redis-cli --latency -c 100

6. Veri Akis Zamanlamalarini Karsilastir​

# MQTT mesaj zaman damgasi ile DB kayit zamani arasindaki farki kontrol et
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
SELECT device_id,
       received_at,
       created_at,
       EXTRACT(EPOCH FROM (created_at - received_at)) AS gecikme_saniye
FROM measurements
WHERE created_at > NOW() - INTERVAL '10 minutes'
ORDER BY gecikme_saniye DESC
LIMIT 20;
"

Cozum Adimlari​

Modbus Timeout Ayarini Duzelt​

# Modbus poller konfigurasyonunu kontrol et
docker exec zeus-backend cat /app/config/modbus_config.yaml

# Timeout degerini guncelle (eger environment variable ise)
# docker-compose.yml'de:
# MODBUS_TIMEOUT_MS=1000  (varsayilan)
# MODBUS_TIMEOUT_MS=500   (hizlandirilmis)

# Degisiklik sonrasi poller'i yeniden baslat
docker compose restart modbus-poller

EMQX Rate Limit Ayarla​

# EMQX rate limit ayarlarini kontrol et
docker exec emqx cat /opt/emqx/etc/emqx.conf | grep -A5 "rate_limit"

# Gerekirse rate limit'i artir
docker exec emqx emqx_ctl conf load /opt/emqx/etc/emqx.conf
docker compose restart emqx

DB Connection Pool Ayarla​

# Mevcut baglanti sayisini kontrol et
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
SELECT count(*) AS toplam_baglanti,
       state,
       count(*) FILTER (WHERE state = 'active') AS aktif,
       count(*) FILTER (WHERE state = 'idle') AS bosta
FROM pg_stat_activity
WHERE datname = 'zeus_db'
GROUP BY state;
"

# max_connections ayarini kontrol et
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "SHOW max_connections;"

# Backend'de pool boyutunu artir (environment variable)
# DATABASE_POOL_SIZE=20 (varsayilan: 10)
# DATABASE_MAX_OVERFLOW=30 (varsayilan: 20)
docker compose restart backend

Batch Size Ayarini Optimize Et​

# Toplu insert boyutunu artir
# Environment variable: MEASUREMENT_BATCH_SIZE=500 (varsayilan: 100)
# MEASUREMENT_FLUSH_INTERVAL=2 (saniye, varsayilan: 5)

# Degisiklik sonrasi backend'i yeniden baslat
docker compose restart backend

TimescaleDB Bakim​

# Gecmis chunk'lari sıkıştır
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
SELECT compress_chunk(i) FROM show_chunks('measurements', older_than => INTERVAL '2 days') i;
"

# Istatistikleri guncelle
docker exec -it zeus-postgres psql -U zeus -d zeus_db -c "
VACUUM ANALYZE measurements;
"

Eskalasyon​

Asagidaki durumlarda eskalasyon yapin:

  • Gecikme 5 dakikayi asiyorsa ve yukaridaki adimlar ise yaramadiysa
  • TimescaleDB disk alani kritik seviyeye ulasmissa (%90+)
  • Toplu veri kaybi suphesi varsa (chunk'lar bozulmus olabilir)
  • Modbus donanim sorunu — birden fazla cihazda ayni anda timeout
  • Performans sorunu kronik hale gelmisse (gunlerce devam ediyorsa) DBA ekibine bildirin