HomeDocsFAQ
GitHubGet Started
v1.2.0 — Now with LLM Pipeline Generator

One library.Infinite streams.

async-fusion/data unifies Kafka streaming, Spark processing, real-time React hooks, an LLM pipeline generator, and a live dashboard — all in one TypeScript-first package.

Read the DocsStar on GitHub
terminal
$ npm install @async-fusion/data
9
Core Features
5+
LLM Providers
1.2.0
Latest Version
MIT
License
Features

Everything you need,
nothing you don't

Stop gluing together five different libraries. async-fusion/data ships batteries-included.

Kafka Streaming

Full producer/consumer support with backpressure, windowing, joins, and stateful processing built-in.

ProducerConsumerWindowing

Spark Integration

Submit and monitor Spark jobs, run SQL queries, and execute PySpark scripts directly from Node.js.

Job SubmissionSpark SQL

Pipeline Builder

Fluent, chainable API for building complex multi-source, multi-sink data pipelines in minutes.

Fluent APIMulti-source

React Hooks

Drop-in hooks for real-time Kafka data, Spark queries, and combined streams right in your components.

useKafkaTopicuseSparkQuery

LLM Generator

Describe a pipeline in plain English — GPT, Gemini, or Claude generates the code for you.

OpenAIGeminiAnthropic

Live Dashboard

One-line WebSocket dashboard with real-time throughput, metrics, and error tracking.

WebSocketMetrics

Reliability

Exponential-backoff retries, circuit breaker, dead-letter queues, and checkpointing — built-in.

Circuit BreakerRetriesDLQ

TypeScript First

Full generics, strict types, and auto-complete everywhere. Zero runtime surprises.

GenericsStrict
Quick Start

Up and running
in minutes

1

Install

bash
# npm
npm install @async-fusion/data

# yarn / pnpm
yarn add @async-fusion/data
2

Build your first pipeline

pipeline.ts
typescript
import { PipelineBuilder } from '@async-fusion/data';

const pipeline = new PipelineBuilder({ name: 'my-pipeline' })
  .source('kafka', {
    topic: 'user-clicks',
    brokers: ['localhost:9092'],
  })
  .transform(data => ({
    ...data,
    processedAt: new Date().toISOString(),
  }))
  .transform(data => (data.value > 100 ? data : null))
  .sink('console', { format: 'pretty' });

await pipeline.run();
LLM Generator

Describe a pipeline,
get the code

Use GPT-4, Gemini, Anthropic Claude, Groq, or a local model. The generator returns production-ready TypeScript and an immediately runnable pipeline instance.

  • OpenAI
  • Google Gemini
  • Anthropic Claude
  • Groq
  • Local (Ollama / LM Studio)
generate.ts
typescript
import { LLMPipelineGenerator } from '@async-fusion/data';

const gen = new LLMPipelineGenerator({
  provider: 'openai',
  apiKey: process.env.OPENAI_KEY,
});

const result = await gen.generateFromDescription(
  'Read from Kafka topic orders, filter > $100, group by user every 5min'
);

console.log(result.code);
await result.pipeline?.run();
LiveFeed.tsx
tsx
import { useKafkaTopic } from '@async-fusion/data/react';

function LiveFeed() {
  const { messages, isConnected } = useKafkaTopic('user-activity', {
    brokers: ['localhost:9092'],
  });

  return (
    <div>
      <span className={isConnected ? 'live' : 'connecting'}>
        {isConnected ? 'Live' : 'Connecting…'}
      </span>
      {messages.map(msg => (
        <div key={msg.offset}>{JSON.stringify(msg.value)}</div>
      ))}
    </div>
  );
}
React Hooks

Real-time data,
in your components

Drop in useKafkaTopic, useSparkQuery, or useRealtimeData and your React components stay in sync with live streaming data automatically.

Start streaming today

Join developers building real-time data pipelines with async-fusion/data.

Read the Docs View FAQ