📓
BK Tricks
<- back to the bodik.tech
  • BK Tricks
  • AI\ML
    • Using PostgreSQL with pgvector extension to store and search embeddings
    • JavaScript-based tools
  • Databases
    • PostgreSQL
      • How to Use HOT Updates in PostgreSQL to Boost Your UPDATE Queries
      • Maximizing the Potential of Embeddings with PGvector: A Comprehensive Guide
    • Elasticsearch
      • The impact of Elasticsearch segments on search speed
      • How to aggregate Elasticsearch results by the timestamp on day interval?
      • Mastering Vector Search in Elasticsearch: Mapping Index for KNN
    • Prometheus
      • What is Prometheus and what is it usually used for?
      • How to create a simple Prometheus exporter?
    • Clickhouse
      • How to make a difference of two arrays in ClickHouse?
      • Materialized View that parses JSON or nested JSON from the string field of another table
      • Attach S3 as a storage engine to ClickHouse
      • Table with TTL to S3
      • Partition ClickHouse table based on unique string hash
      • Create a connection to other ClickHouse instance
  • Infrastructure
    • AWS
      • SAM
        • Schema to create a resource of DynamoDB table using SAM
        • Example of enabling streaming functionality of DynamoDB table using SAM resource
        • Schema to create Lambda function resource using SAM
        • Adding environment variables to Lambda function resource using SAM
        • Adding a scheduled timer to start Lambda function resource using SAM
        • Reading stream from DynamoDB by Lambda function declared using SAM
        • Schema to create a resource of SQS queue using SAM
      • CloudFront
        • How to make redirect using Lambda@Edge?
  • Programming
    • JavaScript
      • Convert hexadecimal string to an array of Uint8 values
      • How to decrypt data encoded with NaCl cryptographic algorithm using NodeJS?
      • How to add a property to JavaScript object if it is not empty or null?
  • Tools
    • Kafka
      • Useful tools
      • How to view all of the offsets for some consumer groups in Kafka using CLI?
      • How to set Kafka offsets of a consumer group to a predefined list for all partitions at once?
  • Math
    • Transform the number from one numeric range to another
  • Frontend
    • Useful resources
Powered by GitBook
On this page
  1. Tools
  2. Kafka

How to set Kafka offsets of a consumer group to a predefined list for all partitions at once?

I often face the situation when something happens with topic processing or I simply created duplicate of the topic, and I need to restore all the offsets for some consumer group.

PreviousHow to view all of the offsets for some consumer groups in Kafka using CLI?NextTransform the number from one numeric range to another

Last updated 2 years ago

You can use the docker image called 'wurstmeister/kafka' pulled from DockerHub to not install and build Kafka on my environment.

docker run -v ~:/mydata wurstmeister/kafka /opt/kafka/bin/kafka-consumer-groups.sh  --bootstrap-server --bootstrap-server b-1.localhost:9092,b-2.localhost:9092 --group js_companies_reimport_group_prod --topic topic_name --reset-offsets --from-file /mydata/temp.csv --execute

Your CSV file should have the next structure where the first column is the name of your topic, the second is the index of your partition and the third is the offset you want to set for it:

topic_name,1,93757
topic_name,2,99793
topic_name,3,93382
topic_name,4,93767
topic_name,5,102769
topic_name,6,93335
How to view all of the offsets for some consumer groups in Kafka using CLI?