flume单agent安装

官网下载[较慢] | 蓝奏云盘下载

1、创建flume文件夹，安装telnet远程协议软件
 复制代码
  1. mkdir /usr/apps/flume
  2. yum install -y telnet
2、上传安装包到flume文件夹下
 复制代码
  1. #rz命令，如果没有就安装：yum install -y lrzsz
  2. rz
3、flume-env.sh环境变量配置
 复制代码
  1. cd  /usr/apps/flume/apache-flume-1.8.0-bin/conf
    • （1）创建配置文件
 复制代码
  1. cp flume-env.sh.template flume-env.sh
    • （2）编辑配置文件
 复制代码
  1. vim  flume-env.sh

#vim和vi命令同理，都是编辑文件，在Xshell中vim更能多彩【有颜色】

    • （3）在flume-env.sh中添加
 复制代码
  1. export JAVA_HOME=/usr/apps/java/jdk1.8.0_221
  2. #你的jdk路径
4、配置flume文件系统环境变量
 复制代码
  1. vim /etc/profile

在最后两行添加以下内容

 复制代码
  1. #你的flume文件夹路径
  2. export FLUME_HOME=/usr/apps/flume/apache-flume-1.8.0-bin
  3. export PATH=$PATH:$FLUME_HOME/bin

使环境变量立即生效

 复制代码
  1. source /etc/profile
5、配置flume采集方案
    • (1)进入目录flume的conf目录
 复制代码
  1. cd /usr/apps/flume/apache-flume-1.8.0-bin/conf
    • (2)copy配置文件
 复制代码
  1. cp flume-conf.properties.template  flume-conf.properties
    • (3)编辑配置文件
 复制代码
  1. vim  flume-conf.properties
    • (4)更改内容：

29行更改为“netcat”，并添加属性👇
#查看行号命令为：set nu[esc+英文冒号+set nu]小写

 复制代码
  1. agent.sources.seqGenSrc.type = netcat
  2. agent.sources.seqGenSrc.bind = localhost
  3. agent.sources.seqGenSrc.port = 44444

#源文件如下，自行对比

 复制代码
  1. # Licensed to the Apache Software Foundation (ASF) under one
  2. # or more contributor license agreements.  See the NOTICE file
  3. # distributed with this work for additional information
  4. # regarding copyright ownership.  The ASF licenses this file
  5. # to you under the Apache License, Version 2.0 (the
  6. # "License"); you may not use this file except in compliance
  7. # with the License.  You may obtain a copy of the License at
  8. #
  9. #  http://www.apache.org/licenses/LICENSE-2.0
  10. #
  11. # Unless required by applicable law or agreed to in writing,
  12. # software distributed under the License is distributed on an
  13. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  14. # KIND, either express or implied.  See the License for the
  15. # specific language governing permissions and limitations
  16. # under the License.
  17.  
  18.  
  19. # The configuration file needs to define the sources, 
  20. # the channels and the sinks.
  21. # Sources, channels and sinks are defined per agent, 
  22. # in this case called 'agent'
  23.  
  24. agent.sources = seqGenSrc
  25. agent.channels = memoryChannel
  26. agent.sinks = loggerSink
  27.  
  28. # For each one of the sources, the type is defined
  29. agent.sources.seqGenSrc.type = netcat
  30. agent.sources.seqGenSrc.bind = localhost
  31. agent.sources.seqGenSrc.port = 44444
  32. # The channel can be defined as follows.
  33. agent.sources.seqGenSrc.channels = memoryChannel
  34.  
  35. # Each sink's type must be defined
  36. agent.sinks.loggerSink.type = logger
  37.  
  38. #Specify the channel the sink should use
  39. agent.sinks.loggerSink.channel = memoryChannel
  40.  
  41. # Each channel's type is defined.
  42. agent.channels.memoryChannel.type = memory
  43.  
  44. # Other config values specific to each type of channel(sink or source)
  45. # can be defined as well
  46. # In this case, it specifies the capacity of the memory channel
  47. agent.channels.memoryChannel.capacity = 100
6、启动单个agent
NOTE:注意自己的flume路径
 复制代码
  1. flume-ng agent --conf conf --conf-file /usr/apps/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties --name agent -Dflume.root.logger=INFO,console
7、再新打开一个master窗口
    • (1)执行Telnet命令
 复制代码
  1. telnet localhost 44444
    • (2)测试
NOTE:输入hello 然后回车
8、切换到原始的master节点
NOTE:可以查看到flume采集到 刚才输入的hello信息
 复制代码
  1. 20/12/07 14:51:14 INFO node.Application: Starting Sink loggerSink
  2. 20/12/07 14:51:14 INFO node.Application: Starting Source seqGenSrc
  3. 20/12/07 14:51:14 INFO source.NetcatSource: Source starting
  4. 20/12/07 14:51:15 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
  5. <!--上方为启动信息，下方为采集到的信息-->
  6. 20/12/07 14:52:03 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }
  7. 20/12/07 14:52:09 INFO sink.LoggerSink: Event: { headers:{} body: E4 BD A0 E5 A5 BD 0D                            ....... }
  8. 20/12/07 14:52:15 INFO sink.LoggerSink: Event: { headers:{} body: 77 6F 72 08 08 08 0D                            wor.... }
  9. 20/12/07 14:52:20 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }
  10. 20/12/07 14:52:22 INFO sink.LoggerSink: Event: { headers:{} body: 77 6F 72 6C 64 0D                               world. }
  11. 20/12/07 14:52:31 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 2C 77 6F 72 6C 64 21 0D          hello,world!. }

单flume总结

安装flume、telnet服务

在conf文件夹中统一修改文件：

cp flume-env.sh.template flume-env.sh：jdk、cp flume-conf.properties.template flume-conf.properties添加修改三行

修改profile配置文件，添加flume环境变量

启动agent服务

复制一个master窗口，连接接口【telnet localhost 44444】

测试

查看

多agent安装

首先启动hadoop集群三个节点。master用来汇聚日志，slave1，slave2用来采集日志

1、master主机下载安装包：

官网下载[较慢] | 蓝奏云盘下载

2、创建flume文件夹
 复制代码
  1. mkdir /usr/apps/flume
3、上传安装包到flume文件夹下
 复制代码
  1. cd /usr/apps/flume
 复制代码
  1. rz

解压：tar -zxvf apache-flume-1.8.0-bin.tar.gz

4、flume-env.sh环境变量配置
 复制代码
  1. cd  /usr/apps/flume/apache-flume-1.8.0-bin/conf
    • (1)创建配置文件
 复制代码
  1. cp flume-env.sh.template flume-env.sh
    • (2)编辑配置文件
 复制代码
  1. vim  flume-env.sh
    • (3)在flume-env.sh中添加

注意自己的jdk路径

 复制代码
  1. export JAVA_HOME=/usr/apps/java/jdk1.8.0_221
5、配置flume文件系统环境变量
 复制代码
  1. vim  /etc/profile
    • (1)在最后两行添加

注意你的flume文件路径

 复制代码
  1. export FLUME_HOME=/usr/apps/flume/apache-flume-1.8.0-bin
  2. export PATH=$PATH:$FLUME_HOME/bin
    • (2)使环境变量立即生效
 复制代码
  1. source /etc/profile
6、将创建好的配置远程复制到slave1、slave2节点
    • (1)复制安装包
 复制代码
  1. scp -r /flume/  root@slave1:/
  2. scp -r /flume/  root@slave2:/
    • (2)复制环境变量
 复制代码
  1. scp -r /etc/profile root@slave1:/etc/
  2. scp -r /etc/profile root@slave2:/etc/
    • (3)到slave1、slave2执行source命令
 复制代码
  1. source  /etc/profile
7、slave1、slave2创建日志文件
    • (1)slave1执行：
 复制代码
  1. mkdir  /usr/apps/flume/logs
  2. cd  /usr/apps/flume/logs
 复制代码
  1. vim  access.log   添加内容：slave1  access.log 
  2. vim  nginx.log  添加内容：slave1  nginx.log 
  3. vim   web.log  添加内容：slave1  web.log
    • (2)slave2执行：
 复制代码
  1. mkdir /usr/apps/flume/logs
  2. cd /usr/apps/flume/logs
 复制代码
  1. vim  access.log   添加内容：slave2  access.log 
  2. vim  nginx.log  添加内容：slave2  nginx.log 
  3. vim   web.log  添加内容：slave2  web.log
8、master、slave1、slave2创建采集日志配置文件
    • (1)slave1配置文件：
 复制代码
  1. cd  /usr/apps/flume/apache-flume-1.8.0-bin/conf
  2. vim  flume-conf.properties

添加内容如下：

 复制代码
  1. #命名
  2. a1.sources = r1 r2 r3
  3. a1.sinks = k1
  4. a1.channels = c1
  5. #资源1
  6. a1.sources.r1.type = exec
  7. a1.sources.r1.command = tail -F /usr/apps/flume/logs/access.log
  8. a1.sources.r1.channels = c1
  9. #由于3个资源同时发送给node03，node03不认识都是哪个路径资源发送的，所以需要拦截器，标识一下
  10. a1.sources.r1.interceptors = i1
  11. a1.sources.r1.interceptors.i1.type = static
  12. a1.sources.r1.interceptors.i1.key = key
  13. a1.sources.r1.interceptors.i1.value = access_log
  14. #资源2
  15. a1.sources.r2.type = exec
  16. a1.sources.r2.command = tail -F /usr/apps/flume/logs/nginx.log
  17. a1.sources.r2.channels = c1
  18. #拦截器
  19. a1.sources.r2.interceptors = i2
  20. a1.sources.r2.interceptors.i2.type = static
  21. a1.sources.r2.interceptors.i2.key = key
  22. a1.sources.r2.interceptors.i2.value = nginx_log
  23. #资源3
  24. a1.sources.r3.type = exec
  25. a1.sources.r3.command = tail -F /usr/apps/flume/logs/web.log
  26. a1.sources.r3.channels = c1
  27. #拦截器
  28. a1.sources.r3.interceptors = i3
  29. a1.sources.r3.interceptors.i3.type = static
  30. a1.sources.r3.interceptors.i3.key = key
  31. a1.sources.r3.interceptors.i3.value = web_log
  32. #sink
  33. a1.sinks.k1.type = avro
  34. a1.sinks.k1.channel = c1
  35. a1.sinks.k1.hostname = master
  36. a1.sinks.k1.port = 41414
  37. #channel
  38. a1.channels.c1.type = memory
  39. a1.channels.c1.capacity = 10000
  40. a1.channels.c1.transactionCapacity = 1000
  41. a1.channels.c1.byteCapacityBufferPercentage = 20
  42. a1.channels.c1.byteCapacity = 800000
    • (2)将slave1的采集配置文件远程复制到slave2

注意路径

 复制代码
  1. scp -r /usr/apps/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties root@slave2:/usr/apps/flume/apache-flume-1.8.0-bin/conf/
    • (3)创建master采集配置文件
 复制代码
  1. cd  /flume/apache-flume-1.8.0-bin/conf
  2. vim  flume-conf.properties

添加内容如下：

 复制代码
  1. #命名
  2. a1.sources = r1
  3. a1.channels = c1
  4. a1.sinks = k1
  5. #资源
  6. a1.sources.r1.type = avro
  7. a1.sources.r1.channels = c1
  8. a1.sources.r1.bind = 192.168.1.4//注意IP
  9. a1.sources.r1.port = 41414
  10. #定义通道
  11. a1.channels.c1.type = memory
  12. a1.channels.c1.capacity = 10000
  13. a1.channels.c1.transactionCapacity = 10000
  14. a1.channels.c1.byteCapacityBufferPercentage = 20
  15. a1.channels.c1.byteCapacity = 800000
  16. #定义sink，注意（修改）IP【master】
  17. a1.sinks.k1.type = hdfs
  18. a1.sinks.k1.channel = c1
  19. a1.sinks.k1.hdfs.path =hdfs://master:9000/flume/logs/%{key}/%y-%m-%d/
  20. a1.sinks.k1.hdfs.filePrefix = events-
  21. a1.sinks.k1.hdfs.round = true
  22. a1.sinks.k1.hdfs.useLocalTimeStamp = true
  23. a1.sinks.k1.hdfs.roundValue = 10
  24. a1.sinks.k1.hdfs.roundUnit = minute
  25. a1.sinks.k1.hdfs.fileType = DataStream
  26. a1.sinks.k1.hdfs.writeFormat = Text
9、master、slave1、slave2启动flume agent
    • 三个节点分别执行：

注意路径

 复制代码
  1. flume-ng agent --conf conf --conf-file /usr/apps/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console
10、查看日志采集结果
在Xshell中新开启一个master窗口

查看hdfs上产生文件路径：

 复制代码
  1. hadoop  fs   -lsr /

查看文件内容：

 复制代码
  1. hadoop fs -cat  /文件路径

多agent总结

flume-env.sh环境变量配置，用到jdk路径

配置flume文件系统环境变量【profile】

将创建好的配置远程复制到slave1、slave2节点

三台节点初始化环境变量【source /etc/profile】

slave1、slave2创建日志文件夹logs

slave1、slave2创建编辑日志文件

master【独立】、slave1、slave2创建采集日志配置文件【slave1 = slave2】(slave1用scp命令发送给slave2)

三个节点分别启动agent

查看日志采集结果

NOTE:经测试无误，如有错误请认真查看

如有问题请点击下方联系笔者

联系笔者QQ | 联系笔者邮箱