kafka 0.8增加了重新分配partition功能,进行扩容,增加减少replica,移动partition
kafka-reassign-partitions.sh脚本
Option Description
------ -----------
--broker-list <brokerlist> The list of brokers to which the
partitions need to be reassigned in
the form "0,1,2". This is required
if --topics-to-move-json-file is
used to generate reassignment
configuration
--execute Kick off the reassignment as specified
by the --reassignment-json-file
option.
--generate Generate a candidate partition
reassignment configuration. Note
that this only generates a candidate
assignment, it does not execute it.
--reassignment-json-file <manual The JSON file with the partition
assignment json file path> reassignment configurationThe format
to use is -
{"partitions":
[{"topic": "foo",
"partition": 1,
"replicas": [1,2,3] }],
"version":1
}
--topics-to-move-json-file <topics to Generate a reassignment configuration
reassign json file path> to move the partitions of the
specified topics to the list of
brokers specified by the --broker-
list option. The format to use is -
{"topics":
[{"topic": "foo"},{"topic": "foo1"}],
"version":1
}
--verify Verify if the reassignment completed
as specified by the --reassignment-
json-file option.
--zookeeper <urls> REQUIRED: The connection string for
the zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
narkhed$ cat partitions-to-move.json
{"partitions":
[{"topic": "foo",
"partition": 1,
"replicas": [1,2,3] }],
"version":1
}
bin/kafka-reassign-partitions.sh --topics-to-move-json-file dirk_topic_move.json --zookeeper host34/kafka-dirktest --broker-list "0341,0361" --generate
生成将topic dirkzhang放到341和361上的json,用json执行execute就可以了
Current partition replica assignment
{"version":1,"partitions":[{"topic":"dirkzhang","partition":6,"replicas":[361,371]},{"topic":"dirkzhang","partition":0,"replicas":[361,371]},{"topic":"dirkzhang","partition":5,"replicas":[341,371]},{"topic":"dirkzhang","partition":4,"replicas":[371,361]},{"topic":"dirkzhang","partition":3,"replicas":[361,341]},{"topic":"dirkzhang","partition":7,"replicas":[371,341]},{"topic":"dirkzhang","partition":1,"replicas":[371,341]},{"topic":"dirkzhang","partition":2,"replicas":[341,361]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"dirkzhang","partition":6,"replicas":[361,341]},{"topic":"dirkzhang","partition":5,"replicas":[341,361]},{"topic":"dirkzhang","partition":0,"replicas":[361,341]},{"topic":"dirkzhang","partition":4,"replicas":[361,341]},{"topic":"dirkzhang","partition":3,"replicas":[341,361]},{"topic":"dirkzhang","partition":1,"replicas":[341,361]},{"topic":"dirkzhang","partition":7,"replicas":[341,361]},{"topic":"dirkzhang","partition":2,"replicas":[361,341]}]}
-----------------------------------------------------------------
bin/kafka-reassign-partitions.sh --zookeeper host34/kafka-dirktest --reassignment-json-file dirk_move_partition.json --execute
将partition0,1的replica放到341,371上(brokerid前边不能有零,是int)
dirk_move_partition.json:
{"version":1,"partitions":
[{"topic":"dirkzhang",
"partition":0,
"replicas":[341,371]
},
{"topic":"dirkzhang",
"partition":1,
"replicas":[341,371]
}]
}
------------------------------------------------
用verify检查是否迁移成功
bin/kafka-reassign-partitions.sh --reassignment-json-file dirk_move_partition.json --zookeeper host34/kafka-dirktest --verify
Status of partition reassignment:
Reassignment of partition [dirkzhang,1] completed successfully
----------------------------------------------------
收集replica迁移,选leader的日志
bin/kafka-run-class.sh kafka.tools.StateChangeLogMerger
--end-time <end timestamp in the The latest timestamp of state change
format java.text. log entries to be merged (default:
SimpleDateFormat@f17a63e7> 9999-12-31 23:59:59,999)
--logs <file1,file2,...> Comma separated list of state change
logs or a regex for the log file
names
--logs-regex <for example: /tmp/state- Regex to match the state change log
change.log*> files to be merged
--partitions <0,1,2,...> Comma separated list of partition ids
whose state change logs should be
merged
--start-time <start timestamp in the The earliest timestamp of state change
format java.text. log entries to be merged (default:
SimpleDateFormat@f17a63e7> 0000-00-00 00:00:00,000)
--topic <topic> The topic whose state change logs
should be merged
bin/kafka-run-class.sh kafka.tools.StateChangeLogMerger --topic dirkzhang --partition 0,1,2 --start-time "2015-07-02 00:00:00,000" --end-time "2015-07-03 00:00:00,000" --logs /data/kafka_2.10-0.8.2.1/logs/state-change.log
相关推荐
kafkatool 国内可能下载不了,搞32 64位的都压缩在一起了Kafka Tool 2.0.7( 32\64) To download the Kafka UI Tool for your operating system, use the links below. All versions of Kafka Tool come with a ...
kafka连接工具
1、图形化界面可以直观地查看 Kafka 的 Topic 里的内容 2、自由设置 Kafka 数据展示格式 3、使用 Kafka Tool 创建/删除 Topic 4、使用 Kafka Tool 模拟发送 Messages
kafka(包可zookeeper,kafka,kafkatool套装)
kafkatool 2.0.9 Windows & maxOS & Linux
To download the Kafka UI Tool for your operating system, use the links below. All versions of Kafka Tool come with a bundled JRE with the exception of the Linux version. For Linux, you must have Java ...
kafka的开源客户端连接工具,简单易用,业内公认好用。使用教程网上众多,方便快速入门,使用门槛低,kafka的开源客户端连接工,kafka的开源客户端连接工,kafka的开源客户端连接工,kafka的开源客户端连接工
kafkatool客户端工具,查询kafka中的信息。
Kafka Tool 2.0.7 kafka可视化管理工 使用Kafka的小伙伴,有没有为无法直观地查看 Kafka 的 Topic 里的内容而发过愁呢? 下面推荐给大家一款带有可视化页面的Kafka工具:Kafka Tool (目前最新版本是 2.0.4)
Kafka Tool is a GUI application for managing and using Apache Kafka ® clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages ...
Kafka Tool是一个用于管理和使用Apache Kafka集群的GUI应用程序。它提供了一个直观的UI,允许用户快速查看Kafka群集中的对象以及存储在群集主题中的消息
Kafka是一种高吞吐量的分布式发布订阅消息系统,它可以处理消费者规模的网站中的所有动作流数据。 这种动作(网页浏览,搜索和其他用户的行动)是在现代网络上的许多社会功能的一个关键因素。 这些数据通常是由于...
kafka客户端,方便查看提供者和消费者的数据,有关消费者偏移量,一目了然,有二级制和UTF-8模式查看消费者内容 查看消费者组
Kafka Tool是一个用于管理和使用Apache Kafka集群的GUI应用程序。它提供了一个直观的UI,允许用户快速查看Kafka群集中的对象以及存储在群集主题中的消息。
Kafka Tool是一个用于管理和使用Apache Kafka集群的GUI应用程序。它提供了一个直观的UI,允许用户快速查看Kafka群集中的对象以及存储在群集主题中的消息。它包含面向开发人员和管理员的功能。一些主要功能包括 快速...
kafkatool mac 版,linux版本 Kafka Tool 2.0.7 To download the Kafka UI Tool for your operating system, use the links below. All versions of Kafka Tool come with a bundled JRE with the exception of the...
kafka tool kafka 可视化工具 mac,亲测可用,mac 10.14版本
查看kafka上面数据的可视化工具, kafkatool-1.0.exe, 可以查看kafka 0.11以下版本:0.8, 0.9, 0.10
是查看kafka上面数据的可视化...也可以去kafka工具官网地址下载:http://www.kafkatool.com/download.html,速度是很慢。 具体安装步骤可以参考该博主的博客:https://www.cnblogs.com/miracle-luna/p/11299345.html
Kafka Tool 1.0.3 版本 64bit exe+dmg [For Kafka version 0.10 and earlier]