☰

» »

Fix skripte: nginx log + group by user-agent + count + sort desc

Fix skripte: nginx log + group by user-agent + count + sort desc

HotBurek ::

Evo, iz interneta skripta, ki v prvem koraku iz default nginx log fajla pobere ven vse user-agent stringe, v drugem pa naredi uniq, sort po dolžini in asc.

Če kdo pozna te kul tool-e (cut, cat, awk, sort, ...) in bi znal zadevo pomodificirat tako, da bi output bil sum po user-agent stringu, ter order by count in desc. Se pravi, da bi videl, koliko request-ov je naredil posamičen user-agent, razverščeno podajoče po skupnem številu?

cut -f 6 -d'"' /var/log/nginx/access.log > /home/user1/useragent/ualist.txt
cat /home/user1/useragent/ualist.txt | awk '{ print length, $0 }' | sort -n -s | cut -d" " -f2- | uniq > /home/user1/useragent/out-list.txt
root@debian:/# iptraf-ng
fatal: This program requires a screen size of at least 80 columns by 24 lines
Please resize your window
  • poskusilo klonirati: HotBurek ()

joggi79 ::

Daj par vrstic access.log da vidim kako izgleda, ker na podlagi tega je odvisno katere stolpce cutas itd.

HotBurek ::

Tole je iz nginx.conf
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';


Tole pa sample log file:

1.2.3.4 - - [01/Aug/2019:15:02:03 +0200] "GET / HTTP/1.1" 404 146 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"
1.2.3.4 - - [01/Aug/2019:15:02:03 +0200] "GET /favicon.ico HTTP/1.1" 404 146 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"
1.2.3.4 - - [01/Aug/2019:15:02:20 +0200] "GET / HTTP/1.1" 200 9 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"
1.2.3.4 - - [01/Aug/2019:15:02:51 +0200] "GET / HTTP/1.1" 200 9 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
1.2.3.4 - - [01/Aug/2019:15:02:54 +0200] "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0" 400 0 "-" "SSL Labs (https://www.ssllabs.com/about/assessmen...
1.2.3.4 - - [01/Aug/2019:15:18:03 +0200] "GET / HTTP/1.1" 403 146 "-" "-"
1.2.3.4 - - [01/Aug/2019:15:24:49 +0200] "GET / HTTP/1.1" 200 9 "-" "Mozilla/5.1 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
1.2.3.4 - - [01/Aug/2019:17:14:40 +0200] "GET / HTTP/1.1" 200 9 "-" "Mozilla/5.1 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
1.2.3.4 - - [01/Aug/2019:19:02:13 +0200] "GET / HTTP/1.1" 403 146 "-" "-"
root@debian:/# iptraf-ng
fatal: This program requires a screen size of at least 80 columns by 24 lines
Please resize your window

joggi79 ::

cut -f 6 -d'"' /var/log/nginx/access.log > /home/user1/useragent/ualist.txt
cat /home/user1/useragent/ualist.txt | awk '{ print length, $0 }' | cut -d" " -f2- | uniq -c | sort -n -r > /home/user1/useragent/out-list.txt

bo?

HotBurek ::

Kul, sem dodal še manjši fix. Če prvi ukaz ostane, kot je, dela "group by" dokler se isti ponavljajo. Vsaj tako izgleda in output je potem takle:

567 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
375 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
299 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
294 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
229 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
227 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
181 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
178 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
151 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
148 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
138 Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Če pa je prvi ukaz takole:

cut -f 6 -d'"' /var/log/nginx/access.log | sort -n -r > /home/user1/useragent/ualist.txt

Potem pa je output res group by in se ne ponavlja.

S tem se da pogledat, približno kdo in koliko delal requeste na web strežnik. Naprimer koliko % ima posamičen browser in OS. To bi lahko potem uvozil v bazo in delal SQL stavke.

Thank u.
root@debian:/# iptraf-ng
fatal: This program requires a screen size of at least 80 columns by 24 lines
Please resize your window

Zgodovina sprememb…

  • spremenilo: HotBurek ()

Invictus ::

Inštaliraj logstash, butni v bazo, in delaj pol tam sumarizacijo.

Pa resno briši bazo, po možnosti razbij bazo na dnevne particije, ki jih potem zbrišeš.

Več kot en teden logov večinoma ne rabiš.
"Life is hard; it's even harder when you're stupid."

http://goo.gl/2YuS2x

Vredno ogleda ...

TemaSporočilaOglediZadnje sporočilo
TemaSporočilaOglediZadnje sporočilo
»

Skripta za Bolho.

Oddelek: Programiranje		304429 (1841) planina91
»

"Port scan" stanje na IPv4 omrežju (strani: 1 2 )

Oddelek: Omrežja in internet		559323 (8243) AštiriL
»

Fake traffic generator

Oddelek: Omrežja in internet		192242 (1438) HotBurek
»

Enolično prepoznavanje brez piškotkov in naslova IP

Oddelek: Novice / Zasebnost		287374 (5147) Horejšio
»

Mozilla po 10 letih sodelovanja z Googlom podpisala z Yahoojem, ki bo nov privzet isk (strani: 1 2 )

Oddelek: Novice / Brskalniki		5321372 (18196) johnnyyy

Več podobnih tem