很久不用linux命令，好多都忘记了，只是常用的记得比较好比如tail、 grep、 top、 touch、 ps-ef、sed等，用的少的日久失修，人脑不像计算机，计算机磁盘满了手动清除才能存下新的文件。但是人脑毕竟不是计算机，遗忘是自然界赋予人的脑的自动清除算法。。。 下面回忆一下awk的统计日志用法:
1.下面我有一个rockmq的日志
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@1990a65e
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@150c158
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@4524411f
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@401e7803
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@1990a65e
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@150c158
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@4524411f
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@401e7803
统计@后面字符串出现的数量：
比如:
2 | 401e7803
2 | 4524411f
下面是执行命令:
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log | awk -F '(into:|@)' '{print $3}'| sort | uniq -c
2 150c158
2 1990a65e
2 401e7803
2 4524411f
hxr-mac:rocketmqlogs houxiurong$
总结
命令说明:
1.从每一行取出需要的字符串并分割
awk -F '(into:|@)' '{print $1}' 按照into: 和 @符号分割，输出结果
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log |awk -F '(into:|@)' '{print $1}'
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set
2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set
2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set
hxr-mac:rocketmqlogs houxiurong$
2.分割后第二个字段值
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log |awk -F '(into:|@)' '{print $2}'
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
sun.nio.ch.KQueueSelectorImpl
hxr-mac:rocketmqlogs houxiurong$
3.分割后第三个字段值,这个字段才是需要的字段
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log |awk -F '(into:|@)' '{print $3}'
1990a65e
150c158
4524411f
401e7803
1990a65e
150c158
4524411f
401e7803
hxr-mac:rocketmqlogs houxiurong$
4.第三个字段但是里面有重复的值，要怎么处理了？使用管道| sort排序
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log |awk -F '(into:|@)' '{print $3}' | sort
150c158
150c158
1990a65e
1990a65e
401e7803
401e7803
4524411f
4524411f
hxr-mac:rocketmqlogs houxiurong$
通过==分割三个字段的值并展示:
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log |awk -F '(into:|@)' '{print "=="$1,"=="$2,"=="$3}' | sort |uniq -c
1 ==2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==150c158
1 ==2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==1990a65e
1 ==2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==401e7803
1 ==2019-09-27 17:51:49 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==4524411f
1 ==2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==150c158
1 ==2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==1990a65e
1 ==2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==401e7803
1 ==2019-09-27 17:53:02 INFO main - instrumented a special java.util.Set == sun.nio.ch.KQueueSelectorImpl ==4524411f
sort 的 -n：依照数值的大小排序；-r 按照相反顺序排列
排序后还不是我需要的值，需要统计这个值出现的次数，可以使用
uniq常用命令字段:
-c 在输出行前面加上每行在输入文件中出现的次数。
-d 仅显示重复行。
-u 仅显示不重复的行。
5.将结果输出到文件中
hxr-mac:rocketmqlogs houxiurong$ cat namesrv_default.log |awk -F '(into:|@)' '{print $3}' | sort |uniq -c | sort -nr > testfile
hxr-mac:rocketmqlogs houxiurong$ ls
namesrv_default.log testfile
hxr-mac:rocketmqlogs houxiurong$ cat testfile
2 4524411f
2 401e7803
2 1990a65e
2 150c158
复习awk命令结束。。。
