Frequently Asked
Questions

async-fusion/data is a unified TypeScript library that brings together Apache Kafka streaming, Apache Spark processing, a fluent pipeline builder, React hooks for real-time UI, an AI-powered pipeline generator, and a live monitoring dashboard — all in one npm package.

It targets backend TypeScript/Node.js engineers who build real-time data pipelines, as well as full-stack developers who want live streaming data directly in their React components without wiring up raw WebSockets or Kafka clients by hand.

The core pipeline builder, Kafka, Spark, error handling, and dashboard features are stable as of v1.2.0. The React hooks (useKafkaTopic, useSparkQuery) are under active development and should not be used in production yet. Check the release notes for the latest stable surface.

kafkajs and native Spark APIs are excellent primitives, but they require a lot of boilerplate — separate retry logic, monitoring, checkpointing, and error classification for each. async-fusion/data layers a single fluent API on top of all of them, ships retries and circuit-breakers out of the box, and adds a live dashboard in one line.

Node.js ≥ 16.0.0 is required. We recommend Node.js 18 LTS or 20 LTS. The library is ESM and CJS dual-packaged so it works with both module systems.

Only if you use those features. You can use the Pipeline Builder, error-handling utilities, LLM Generator, and most of the library without any Kafka or Spark infrastructure. For Kafka features, start a local broker via Docker. For Spark features, point the client at any accessible Spark cluster.

Yes — kafkajs is a peer dependency to keep bundle sizes lean for users who only need Spark or the pipeline builder. Install it with: npm install kafkajs

Absolutely. The package uses standard package exports and works with any monorepo setup. Point your workspace dependency at @async-fusion/data and the types and dual CJS/ESM exports will resolve correctly.

Yes. Call .source() multiple times — events from all sources are merged (fan-in) before the first .transform().

Yes. Each .sink() call adds another output target. Processed records are forwarded to every sink in parallel. Common pattern: sink to both Kafka (for downstream consumers) and a database (for persistence).

By default the error bubbles up, the pipeline stops, and the offending message is sent to the dead letter queue if configured. With retryConfig set, the transform is retried with exponential back-off before going to the DLQ.

The current API runs pipelines until completion or an unrecoverable error. Pause/resume controls are planned for v1.3.0. For now, use checkpointing (checkpointLocation) so a restarted pipeline picks up where it left off.

Transactional producers (exactly-once semantics) are on the roadmap. Currently the Producer supports at-least-once delivery with optional idempotence settings passed through to kafkajs.

Yes. Pass the full kafkajs SSL/SASL configuration object through the brokers config. async-fusion/data passes these options transparently to kafkajs so all authentication mechanisms it supports are available.

Always review generated code before running it in production. The generator produces valid PipelineBuilder code, but it may not handle edge cases specific to your data or infrastructure. Treat it as a fast starting point, not a finished artefact.

provider: 'local' points to your local Ollama or LM Studio instance. No API key required.

.enableDashboard(port) starts an Express + WebSocket server inside your existing Node.js process. No separate deployment needed.

The dashboard uses a singleton pattern — only one server instance runs per process. Multiple pipelines appear as separate rows in the same dashboard UI.

Not out of the box. For production, put the dashboard behind a reverse proxy (nginx, Caddy) with basic auth or OAuth. Binding to localhost and using an SSH tunnel is also a common pattern.

dist/types/index.d.ts is bundled in the package. No @types/* package needed.

Yes. Both CJS (require) and ESM (import) builds are shipped. You lose type-checking but the runtime is identical. JSDoc type comments are available in some editors via the bundled .d.ts files.

Open an issue on GitHub. Include your Node.js version, library version, a minimal reproduction script, and the full error stack trace.

Open a feature request on GitHub Issues with the Feature Request template. Describe your use case, not just the solution.

MIT License — free to use in commercial projects, modify, and distribute. See the LICENSE file.