关于canal同步数据到kafka的kerberos认证,操作过程:
1、修改canal.properties
canal.mq.kafka.kerberos.enable = true
canal.mq.kafka.kerberos.krb5FilePath = ../conf/kerberos/krb5.conf
canal.mq.kafka.kerberos.jaasFilePath = ../conf/kerberos/jaas.conf
2、在conf目录下创建kerberos文件夹
添加krb5.conf文件
添加jaas.conf文件
添加kafka.keytab文件
-----------在kerberos下有三个文件
3、krb5.conf文件和jaas.conf文件
##jaas.conf文件
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="../conf/kerberos/kafka.keytab"
principal="kafka/master1-dev.hadoop.com.cn@HADOOP.COM.CN";
};
##krb5.conf文件
includedir /etc/krb5.conf.d/
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = HADOOP.COM.CN
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
[realms]
HADOOP.COM.CN = {
kdc = master1-dev.hadoop.com.cn
admin_server = master1-dev.hadoop.com.cn
}
[domain_realm]
.hadoop.com.cn = HADOOP.COM.CN
hadoop.com.cn = HADOOP.COM.CN
4、启动报错
[main] ERROR com.alibaba.otter.canal.server.CanalMQStarter - ## Something goes wrong when starting up the canal MQ workers:
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:456) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:303) ~[kafka-clients-1.1.1.jar:na]
at com.alibaba.otter.canal.kafka.CanalKafkaProducer.init(CanalKafkaProducer.java:81) ~[canal.server-1.1.4.jar:na]
at com.alibaba.otter.canal.server.CanalMQStarter.start(CanalMQStarter.java:51) ~[canal.server-1.1.4.jar:na]
at com.alibaba.otter.canal.deployer.CanalStarter.start(CanalStarter.java:101) [canal.deployer-1.1.4.jar:na]
at com.alibaba.otter.canal.deployer.CanalLauncher.main(CanalLauncher.java:115) [canal.deployer-1.1.4.jar:na]
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /data/home/apps/canal_1.1.4_local/bin/../conf/kerberos/jaas.conf
at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:133) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:98) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:119) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:413) ~[kafka-clients-1.1.1.jar:na]
... 5 common frames omitted
确定配置了jass并且也确定没有问题,但就是启动失败。
最后在启动脚本./bin/startup.sh里面添加一句:
JAVA_OPTS=" $JAVA_OPTS -Djava.security.auth.login.config=$base/conf/kerberos/jaas.conf"
如果还是不行,再加一句。
JAVA_OPTS=" $JAVA_OPTS -Djava.security.krb5.conf=$base/conf/kerberos/krb5.conf"
就OK了!