Vel Development Update: Milestone 1 Complete
The foundational data pipeline is live and verified end-to-end. Here's what we built, what we learned, and what's coming in Milestone 2.
Milestone 1 Is Done
We shipped the foundational data pipeline for Vel and verified it end-to-end. This was a quieter milestone from the outside — no UI, no dashboards, nothing visible yet. But it's the most critical piece of the architecture, and getting it right mattered more than getting it fast.
What We Built
The M1 pipeline runs: Splunk connector → Kafka (raw events) → Normalization service → Kafka (normalized events) → Indexer → Elasticsearch.
Every component is independently deployable and independently testable. The normalization service converts raw source events into OCSF-compliant structures using a mapping layer that's entirely config-driven — adding a new source means adding a mapping file, not rewriting service logic.
The Kafka backbone gives us durability and decoupling. If the indexer falls behind, events queue and catch up. If normalization needs to be updated, it can be redeployed without touching the connectors.
What We Learned
A few things surprised us during M1 that are shaping M2:
Schema gaps are inevitable. OCSF covers most common event types well, but there are edge cases — particularly in identity and authentication events — where the mapping requires interpretation. We're building a mechanism to flag unmapped fields rather than silently dropping them.
Connector reliability needs more attention than throughput. The temptation is to optimize for how many events per second you can push. The more important thing is what happens when the source is temporarily unavailable. Reconnection logic and backpressure handling ended up consuming more engineering time than the happy path.
What's Coming in M2
Milestone 2 is the backend core — the services that actually power the Vel workbench:
- ◆User service with JWT authentication and RBAC (Hunter, Lead, Admin roles)
- ◆Hypothesis service with full lifecycle management and ATT&CK tagging
- ◆Evidence service with MinIO integration and chain-of-custody tracking
- ◆Hunt service for orchestration, assignment, and collaboration
- ◆Query Federation API for unified multi-source querying
- ◆API Gateway tying it all together
We're targeting Week 8 for M2 completion. If you want early access when we start opening beta slots in M3, now is the time to apply.
Ready to put this into practice?
Vel is the workbench that makes these workflows operational — hypothesis tracking, evidence management, query federation, and leadership visibility in one place.