Comprehensive Guide to Kafka Load Testing with LoadStrike

Start with a concrete evaluation: can your order worker turn an orders.in event into an orders.completed event while meeting the agreed completion window? LoadStrike makes this a code-first test by pairing source and result through the same business ID. The useful feature is the connection between accepted work and an observed outcome.

The walkthrough below uses illustrative topic names and payloads. It is a setup recipe checked against the LoadStrike endpoint and run-mode documentation, not an executed Kafka benchmark. Your application worker, broker access, runner key and reporting configuration are required before a real result exists.

Set up a LoadStrike Kafka test with isolated traffic

Install the SDK for the language your team maintains. LoadStrike supports C#, Go, Java, Python, TypeScript and JavaScript; use the linked installation guide for the current package and runtime requirements. For Python native Kafka endpoints, install loadstrike[kafka]. Native Kafka endpoint access requires a valid runner key on Pro or Enterprise; this recipe is not a promise of native Kafka execution on the Free plan.

Prepare test input and completion topics on your own broker, with an application worker that consumes input, performs the operation and publishes the result. Give the LoadStrike observer a dedicated ConsumerGroupId instead of joining the application worker group. For traditional Kafka consumer groups, joining the worker group can change which consumer receives a partition and alter the workload you intended to measure.

Use a run-specific tracking ID. An example input is {"trackingId":"run-a-order-1","tenantId":"tenant-a"}; the completion carries those values and status = completed after the required work finishes. Configure TrackingField as json:$.trackingId on both sides. Keep topic retention and consumer offsets explicit so historical messages do not get mistaken for the current run.

• Set BootstrapServers and Topic for each Kafka endpoint; Consume mode also requires ConsumerGroupId.
• Configure SecurityProtocol and SASL settings to match the broker, and load credentials from the test environment.
• Decide whether StartFromEarliest and existing committed offsets fit the test; a new group may replay retained records.
• Record SDK version, broker version, topic partitions and payload shape before the first run.

Choose generated load or existing-traffic observation

For Kafka producer and consumer load testing, choose GenerateAndCorrelate. Set Source to a Kafka Produce endpoint on orders.in and Destination to a Kafka Consume endpoint on orders.completed. The Kafka protocol examples provide the language-specific endpoint objects. Wrap the scenario with the tracking configuration and apply a supported load simulation; the application supplies the processing between the topics.

For an API-to-Kafka workflow, keep GenerateAndCorrelate but use an HTTP Produce source and a Kafka Consume destination. Preserve the source tracking ID in the completion event. The run-mode documentation contains this endpoint pairing; an HTTP success response by itself does not certify the downstream operation.

To observe another service creating traffic, set Source and Destination to Consume mode, choose CorrelateExistingTraffic and set ForDuration. The linked C# example observes for ten minutes and uses a thirty-second CorrelationTimeout; these are sample configuration values to adapt to your completion contract. Do not add WithLoadSimulations to an observation scenario.

• Generation mode creates input traffic and matches its downstream results.
• Observation mode listens to both sides; it does not inject work or automatically generate missing tracking IDs.
• Configure a separate observer group on each observed topic so test observation does not share the application worker assignment.
• Use the SDK language tab for exact method names and duration units rather than copying another language verbatim.

Verify a smoke completion before increasing the workload

Run a small smoke workload and inspect an input and its corresponding result. Check that the observer actually receives the completion and that the report records the expected match. A started test process does not establish that a Kafka consumer has its partition assignment; verify the path through the observed smoke outcome.

After matching works, select an arrival-rate profile for a fixed offered rate or a concurrency profile for a fixed number of active invocations. Use the documented Inject, RampingInject or concurrency equivalents for your SDK. Start modestly, increase one load dimension at a time and keep payload size, partitions, group configuration and client settings fixed.

Compare offered and achieved scenario invocations before blaming the application for a missed target. One invocation can generate more than one transport operation, so record actual Kafka records and bytes separately. Collect application consumer lag and broker resource measurements through Kafka monitoring; the native endpoint does not automatically add all those diagnostics to the LoadStrike report.

• Check matched completion, timeouts, duplicate tracking IDs and unmatched destinations after the smoke run.
• Use GatherByField on the destination when tenant or event-type groups need separate completion evidence.
• Scope correlation latency to the endpoint timestamps. In C# Kafka, successful delivery time and the destination record timestamp can differ from a client stopwatch; align clocks and record the topic timestamp behavior.
• Agree stop conditions before stressing the environment and test only systems you own or are authorized to use.

Keep the reports and decide whether LoadStrike fits

Retain the run configuration, LoadStrike artifacts and Kafka monitoring export for the same period. State the start and completion boundaries, achieved load, match count, timeouts and timestamp basis. Read matched latency together with incomplete work; a fast completed subset is not proof that every operation met the completion window.

Repeat the same workload against the same application build before comparing a configuration change. If a run includes historical replay, mixed event shapes or a saturated generator, describe that limitation. A short smoke check proves that the setup can observe one path; it does not establish production capacity or long-duration stability.

Evaluate LoadStrike on the feature your team needs: source-to-destination matching across a Kafka or API workflow, with reportable completion, duplicate and timeout evidence in your chosen SDK language. Use the linked protocol examples for the setup, the endpoint docs for authentication and entitlements, and the report guide to inspect the first real run. Publish performance claims only when those artifacts support them.

• Record the environment, SDK and application versions, load phases and repeated-run count.
• Keep the business-ID denominator distinct from duplicate destination observations.
• Use Kafka monitoring to explain broker and consumer behavior beside the application completion evidence.
• Bring one measured run and its completion contract to the evaluation; no universal throughput improvement is claimed here.