10일간 로그 분석

10일간 웹로그에서 가장 많이 찍힌 문자열이 다음과 같습니다.

$ head -20 part-r-00000.sorted

-       38368124

+0900]  15687677

"GET    15639179

HTTP/1.1"       15460537

.NET    12153058

CLR     12152759

NT      12104459

"Mozilla/5.0    10849691

(compatible;    10664745

Windows 9383937

MSIE    9348230

200     8631840

6.1;    7713934

304     5998624

like    4948562

(KHTML, 4430859

Gecko)  4422071

"Mozilla/4.0    4289783

WOW64;  3823166

2.0.50727;      3712273

추출한 데이터 목록과 용량입니다.

[hadoop@hadoop input]$ ls -alh && du -h

total 3.6G

drwxr-xr-x.  2 hadoop hadoop  12K Nov 14 20:01 .

drwxr-xr-x. 12 hadoop hadoop 4.0K Nov 14 20:11 ..

-rw-r--r--.  1 hadoop hadoop 455M May 15  2013 localhost_access_log.2013-05-14.txt

-rw-r--r--.  1 hadoop hadoop 467M May 16  2013 localhost_access_log.2013-05-15.txt

-rw-r--r--.  1 hadoop hadoop 426M May 17 00:00 localhost_access_log.2013-05-16.txt

-rw-r--r--.  1 hadoop hadoop 153M May 18 00:00 localhost_access_log.2013-05-17.txt

-rw-r--r--.  1 hadoop hadoop 138M May 19 00:00 localhost_access_log.2013-05-18.txt

-rw-r--r--.  1 hadoop hadoop 154M May 20 00:00 localhost_access_log.2013-05-19.txt

-rw-r--r--.  1 hadoop hadoop 474M May 21 00:00 localhost_access_log.2013-05-20.txt

-rw-r--r--.  1 hadoop hadoop 474M May 22 00:00 localhost_access_log.2013-05-21.txt

-rw-r--r--.  1 hadoop hadoop 483M May 23 00:00 localhost_access_log.2013-05-22.txt

-rw-r--r--.  1 hadoop hadoop 437M May 24 00:00 localhost_access_log.2013-05-23.txt

3.6G    .

싱글 노드에서 돌렸을 때 20:11:27~20:19:28 (8'01") 걸렸습니다. sort는 1분 이하로 걸렸습니다.

date && hadoop jar hadoop-mapreduce-examples-2.2.0.jar wordcount input output && date

part-r-00000 파일을 정렬한 linux 명령은 다음과 같습니다.

sort -r -n -k2 part-r-00000 > part-r-00000.sorted

virtualbox 가상머신에 centos 6.4 minimal 설치하고, 실행했습니다.

$ cat /proc/cpuinfo /proc/meminfo

processor       : 0

model name      : Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz

cache size      : 6144 KB

fpu             : yes

cpuid level     : 5

wp              : yes

bogomips        : 6585.72

clflush size    : 64

cache_alignment : 64

address sizes   : 36 bits physical, 48 bits virtual

MemTotal:        1020532 kB

$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_hadoop-lv_root

                       14G  5.4G  7.4G  43% /

tmpfs                 499M     0  499M   0% /dev/shm

/dev/sda1             485M   52M  408M  12% /boot

13/11/14 20:19:27 INFO mapreduce.Job:  map 100% reduce 100%

13/11/14 20:19:27 INFO mapreduce.Job: Job job_local1139720980_0001 completed successfully

13/11/14 20:19:27 INFO mapreduce.Job: Counters: 27

        File System Counters

                FILE: Number of bytes read=248170094651

                FILE: Number of bytes written=17401238525

                FILE: Number of read operations=0

                FILE: Number of large read operations=0

                FILE: Number of write operations=0

        Map-Reduce Framework

                Map input records=15687677

                Map output records=376186366

                Map output bytes=5336272958

                Map output materialized bytes=135763805

                Input split bytes=16874

                Combine input records=376186366

                Combine output records=4875302

                Reduce input groups=1665531

                Reduce shuffle bytes=0

                Reduce input records=4875302

                Reduce output records=1665531

                Spilled Records=19718920

                Shuffled Maps =0

                Failed Shuffles=0

                Merged Map outputs=0

                GC time elapsed (ms)=10430

                CPU time spent (ms)=0

                Physical memory (bytes) snapshot=0

                Virtual memory (bytes) snapshot=0

                Total committed heap usage (bytes)=17564585984

        File Input Format Counters 

                Bytes Read=3831605158

        File Output Format Counters 

                Bytes Written=59875419