The general goal of the simulator is to provide a simple extensible model for plugging together systems with behavior, with a simple predictable interaction.

There are some basic concepts:

• Essentially communication is done with "messages" - even if the interaction is really a simple RPC call, or even function call. Messages are how systems interact with each other.
• Each system has state, which is loaded from the set of messages (if you like, the simulation is event-sourced in nature to make it easy to reason about)

The Actual Code

The in-flight messages are rendered with a particle system - something like this:

    const { messages } = useMessages()
:: ::
   <CustomGeometryParticles position={[0, 0, 0]} messages={messages} />

We need to get the messages from somewhere, actually from one system to the next one. Let's say we have a market data simulator...

    const api = useMessages()
    const n=0
    <Button onClick={()=> n+=100}/>
    <Simulator id="sim" sendMessage={api.sendMessages} maxSequence={n}/>
    <Kafka id="kafka" receiveMessages={api.receiveMessages}/>
    <CustomGeometryParticles position={[0, 0, 0]} messages={api.messages} />

This will result in 100 messages being sent from the simulator to Kafka when the button is clicked.

Partitions and State

The actual state of the Partitions and segments in Kafka would be maintained inside the hierarchy.

const Kafka : (sendMessage,...) => {
    const localSendMessage = (message)=> message.destination.id===kafka.id ? sendMessage: null)
    return topics.map( <Topic sendMessage={sendMessage}/>)
}

const Topic : (sendMessage,...) => {
    const localSendMessage =(message)=> message.topic==topic.id ? sendMessage:null)
    return partitions.map(<Partition sendMessage={sendMessage}>)
}

//finally
const Partition: (sendMessage,...) => {
    const localSendMessage = (message) => setState( newState of Partition))
}

Subscribe From Kafka

Subscribing the messages is basically treated like a factory. New messages are created based on the state of the component. These messages are added to the message state.

const Simulator = (receiveMessages,minSequence,maxSequence,...) => {
    useEffect( ()=> {
        if (minSequence>internalState) receiveMessages(new Message(...))
    },[maxSequence])

}

const Kafka = (receiveMessages,minOffset:{topic:[{partition:1, offset: 121}}) => {
    // delegate the offset down to the child components

}

Maintaining consumer State

The last two missing pieces of the puzzle is the subscription state in the middle.

• We need to be able to manually trigger the source of messages by setting the maxSequence and limit on the subscription state for the market data simulator
• We need to (automatically?) update the partition offset in the kafka consumer so that consuming messages from Kafka works like real-life Kafka
• There's an orchestator in the middle here doing the integration. That holds of the state of the subscription

The data model for the orchestrator is simply the source/destination pairs defining the connections between the nodes in the graph. The min/max below reflects the state of the subscription. The difference between max and min is the batch size that will be processed for one set of messages.

[{
    "from": {
        id:"simulator"
    },
    "min": 0,
    "max": 100,
    "to": {
        "id": "kafka",
        "topic": "topic-a"
    }
},
{
    "from": {
        "id":"kafka",
        "topic":"topic-a"
    },
    "min":121,
    "max":200,
    "to": {
        "id": "kafka",
        "topic", "topic-b"
    }
},{
    "from": {
        "id":"kafka",
        "topic":"topic-b"
    },
    "min":0,
    "max:":200,
    "to": {
        "id": "postgres",
        "db": "market",
        "table": "price-data
    }
}]

Adding Behavior

Systems will have behavior (for example, Kafka log compaction). But also processing steps will do more than nothing to messages.

function process( sourceMessage: InputMessage[]  ) : MessageType[] {
    return sourcesMessages.reduce<MessageType>( (p,c) => {...},[])
}

Decoupling Animation / Layout And Views

Would it be better to arrange the layout generically..

1. Hierarchy container pattern would use a flex layout
2. Vertical stacking would use a stacking hook
3. Structure of the hierarchy is in the data
4. Positions set in atoms, rendered later

Higher Order Component for positions

Injects the position of a component from the store

export function withPosition<T extends PositionSizeProps>(WrappedComponent: ComponentType<T>) {
    // eslint-disable-next-line react/display-name
    return ({ id, ...props }: Omit<T,"position"|"size"> & {id:string}>) => {
        const [{ size, position }, setPos] = useAtom(commonAtom(id));
        const newProps: T = { ...props, position, size } as unknown as T;
        return <WrappedComponent {...newProps} />;
    };
}

So how do we actually set the position and size?

export function SetPos({position,size}: PositionSizeProps) {
    useSetPos({position,size})
    return null;
}

export function withSetPosition( stack:Stack) {
    return () => {
        const position = stack.getPosition(index)
        return <SetPos position={position} size={size}/>

    }
}

// use components that return nothing, but run the hook to update the position
<ContainerPosition atom={region} children={x=>x.children} stack={stack}/>
<ContainerPosition atom={region} children={x=>x.children}/>
<ContainerPosition atom={region} children={x=>x.children}/>
<ContainerPosition atom={region} children={x=>x.children}/>

<WrappedPosRegion id="XX">
{
    region.zones.map( p => <WrappedPosZone id={p.id}/> )
}