Question

如何挖掘nginx日志中隐藏的金矿

Answer 1

　　需求是这样，提取出nginx日志中，GET或者POST的数据中，名称为’shell’字段的数据　　nginx日志的format配置：
　　'$proxy_add_x_forwarded_for - $remote_user [$time_local] "$request" '

'$status $request_body "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time';
　　nginx日志实际的内容，大致如下：
　　61.164.xxx.xxx, 10.16.xx.x – - [13/Aug/2014:00:00:02 +0800] “GET /xxx/xxx?stepid=32&tid=U%2Bo3c0S&&output=json&language=zh_CN&session=114047349&dip=10920&diu=00343B30-9EB8-4B43-A978-FF838587E989&diu3=e9c3afaa4134d678&dia=8E72-1B19E16C0B8E&shell=504c450000000000000000000000004100&compress=false&channel=&adcode=310000&pagenum=1&pagesize=10&sign=C3833277A0DE5C071A799AB5E8AE41E2 HTTP/1.1″ 200 4382 “-” “xxx-iphone” “61.164.xxx.xxx” 0.204
　　或
　　218.202.xxx.xxx, 123.103.xxx.xxx, 10.16.xxx.xxx – - [2014-08-19 22:17:08.446671] “POST /xxx/xxx-web/xxx HTTP/1.1″ 200 stepid=15&tid=U753HifVPE0DAOn%2F&output=json&language=zh_CN&session=114099628&dip=10920&diu=DBDBF926-3210-4D64-972A7&xxx=056a849c70ae57560440ebe&diu2=2DFDB167-1505-4372-AAB5-99D28868DCB5&shell=e3209006950686f6e65352c3205004150504c450000000000000000000000000000&compress=false&channel=&sign=438BD4D701A960CD4B7C1DE36AA8A877&wua=0&appkey=0&adcode=150700&t=0 “-” “xxx-iphone” “218.202.xxx.xxx, 123.103.xxx.xxx” 0.001 20.001
　　grep -P 'shell' access.log | sed 's/\(.*\)&shell=\(.*\)&\(.*\)/\2/g' | awk -F '&' '{print $1}' > output.txt