We work in highly regulated and constrained environments. That means the operational space we work in also requires a tailored approach to data problems. Instead of relying on scalable platforms such as Databricks, connecting AWS OpenSearch to S3 storage, or using other top-tier cloud solutions, the architecture we provide often has to be designed from the ground up around the constraints of the target environment.
This means working within strict hardware budgets while still meeting requirements for sustainable data ingestion, query performance, latency, and concurrency. The territory is often unknown, and benchmarks are necessarily best estimates until the solution reaches production.
In this case, we had to design a deliberately constrained, disk-oriented indexing architecture for searching hundreds of terabytes of JSON without materializing the dataset into a conventional search engine or data warehouse. We could not afford the usual architecture given the hardware budget, so we aggressively exploited inexpensive NVMe storage, ordered KV storage, and immutable blobs.
The Problem #
For this project, we were working with JSON blobs. Each blob represented a small record, averaging 1–2 KB, with records batched into JSONL files. The total data volume was expected to be around 250 TB of raw JSON, amounting to more than hundred billion JSON objects distributed across more than one hundred thousand JSONL files.
The high-level requirements were:
- Preserve the original raw data during ETL as “cold” storage.
- Leave room to introduce additional data sources in the future.
- Make a subset of keys from the JSON objects searchable.
- Support the following search modes:
- Search by a specific key (typed search).
- Case-insensitive search (normal search mode).
- Trailing wildcard (prefix) search.
- Filtering by the record time present in the record, using from and to.
- Return 1,000 records from a single request with an average round-trip time of less than 5 seconds.
The real roadblock was the available hardware budget. The data had to be stored and made searchable within the same rack, and the budget constraints made essentially any memory-heavy solution infeasible.
In this environment, that meant throwing the standard stack out of the window. We instead shifted our focus toward fast disk solutions and began designing around a low-memory operating model.
Architecture #
MinIO and TiKV #
We decided to use MinIO for storage and TiKV for index.
MinIO was a natural choice because of its reliability, broad adoption, and existing clients such as boto3 and minio/minio-go. We knew it could scale to large volumes while maintaining good performance. More importantly, MinIO supports requesting a byte range from a specific offset within an object.
That capability was particularly attractive for our design. Combined with fast NVMe drives, it allowed us to keep the original JSONL files intact while retrieving individual records without loading entire files.
TiKV was much less familiar territory for us. TiKV is built around RocksDB, which is well suited to disk-heavy, relatively low-memory workloads. Its cluster architecture also provides a straightforward path to scaling across multiple nodes.
We could provision similar hardware with fast NVMe drives for the TiKV cluster, with the main difference being the additional memory required for RocksDB indexes and filter blocks on each node.
Our data flow, at a high level, was:
- Save the original JSON batches to the MinIO cluster as JSONL files.
- For each JSON record, index the searchable key values in TiKV. We refer to this as our index format.
- Store pointers in TiKV values that identify the corresponding JSON record within a MinIO JSONL file.
- During a search, use the TiKV keys to locate matching records and then fetch those records from MinIO using the stored pointers.
flowchart LR
Sources["Data Sources"]
Ingest["JSONL Ingestion"]
subgraph Storage["Storage & Search Layer"]
MinIO["MinIO Cluster
Raw JSONL / Cold Storage"]
TiKV["TiKV Cluster
Search Index"]
end
Search["Search API"]
Client["Client / User"]
Sources --> Ingest
Ingest --> MinIO
Ingest --> TiKV
Client --> Search
Search --> TiKV
TiKV -->|"Record path + byte offsets"| MinIO
MinIO -->|"JSON record"| Search
Search --> Client
The core idea was to turn the TiKV keyspace into a purpose-built secondary index, while using each value as a byte-range pointer into an immutable JSONL file.
This allowed the source of truth to remain in cheap, immutable blobs. The index only needed to materialize the information required to locate a matching record.
Our initial hardware plan was:
- A 4 node MinIO cluster.
- A 4 node TiKV cluster, with one node configured as the placement driver.
Extract-Transform-Load #
Saving the JSONL batches to MinIO was the easy part. Because new data was expected to arrive daily after the initial ingestion, we also built a pipeline around MinIO change monitoring, with watchers triggering indexing workflows whenever new data appeared.
The part that made the overall setup possible was the index architecture built around TiKV key-value pairs. To satisfy the search requirements while working with the expected data volume and very limited memory, the index had to be designed specifically for this project.
Here is an example of an index key-value pair:
- Key:
gbdeadbeef:20250101:572d5e5d-ea9e-4ecb-9b0b-7060c9f7fdea - Value:
/20260101/records_0052.jsonl:2560:4120
Both the key and the value contain encoded information.
flowchart LR
subgraph Key["TiKV Key"]
A["Source Code"]
B["Field Code"]
C["Hex-encoded Value"]
D["Record Date"]
E["Record UUID"]
A --> B --> C --> D --> E
end
Key --> Example["gbdeadbeef:20260101:572d5e5d-ea9e-4ecb-9b0b-7060c9f7fdea"]
subgraph Value["TiKV Value"]
F["MinIO Object Path"]
G["Start Offset"]
H["End Offset"]
F --> G --> H
end
Value --> Example2["/20260101/records_0052.jsonl:2560:4120"]
Example --> Lookup["Index lookup"]
Example2 --> Record["Exact JSON record
inside MinIO JSONL"]
Index Key #
The first byte of the key (g in this example) identifies the record source. This allows us to support multiple data sources in the future and filter by data source during searches.
The second byte (b) identifies the field being searched. Because we work with a known and limited set of searchable fields, we can assign a short code to each field.
From the third byte up to the first delimiter (:), we store the value of the searchable field as it appeared in the raw JSON, but encoded as hexadecimal (deadbeef).
We cannot store the value as plain text because it could contain our delimiter. We also cannot hash the value because we need to preserve the ability to perform wildcard searches. We therefore needed an encoding that retained the simplicity of range-based keys while remaining predictable, which led us to hexadecimal encoding.
The value between the first and second delimiters (:) is the record date (20260101). It is always stored in YYYYMMDD format, which keeps the date lexicographically sortable and allows it to participate in key-range filtering.
Finally, the value after the second delimiter is the record identifier (572d5e5d-ea9e-4ecb-9b0b-7060c9f7fdea) as it appears in the raw JSON record.
Including the record identifier ensures that the indexed key can be matched to the retrieved record and guarantees uniqueness. Without it, multiple records sharing the same source, field, value, and record date could overwrite one another.
Index Value #
The index value points to the MinIO object containing the record and stores the byte range for that record.
Because the original JSONL files are preserved, we track each JSON record using its byte offsets within the file. This is where MinIO’s support for byte-range requests becomes particularly useful.
The first value, from the beginning of the value up to the first delimiter (:), is the MinIO object identifier.
The second and third values are the start and end offsets of the JSON record within that object. The matching record is located between those two byte offsets.
ETL Pipeline #
The ETL pipeline therefore consists of the following steps:
- Save the original JSON batches to the MinIO cluster as JSONL files.
- Watchers monitor MinIO for changes and trigger indexing when an unindexed object appears or an existing object requires reindexing.
- Parse all records in the JSONL file.
- Extract record metadata and all searchable keys with their corresponding values.
- Build KV pairs for every searchable key, with each pair pointing to the corresponding record in MinIO.
- Write the KV pairs to TiKV.
- Mark the indexing job as complete once all records have been indexed.
A simplified version of the index key and value insertion looks like this:
searchTermHex := hex.EncodeToString([]byte(val))
tikvKey := fmt.Sprintf("%s%s%s:%s:%s", sourceCode, fieldCode, searchTermHex, recordTime, recordUUID) // key
tikvValue := fmt.Sprintf("%s:%d:%d", recordMinioObjectPath, startOffset, endOffset) // pointerHere, val is the actual value of the field represented by fieldCode in the original JSON record.
Search #
Once the index was in place, searching the data became a three-step process:
- Find the matching TiKV keys.
- Read the corresponding values, which act as pointers to the source records.
- Retrieve each source JSON record from MinIO using the object path and byte range stored in the TiKV value.
sequenceDiagram
participant U as User / Client
participant API as Search API
participant T as TiKV
participant M as MinIO
U->>API: Search request
API->>API: Build key range
API->>T: Scan(startKey, endKey, limit + 1)
T-->>API: Matching keys + pointers
loop For each matching record
API->>API: Parse MinIO path + offsets
API->>M: GetObject(byte range)
M-->>API: JSON record
end
API-->>U: Matching records
Because TiKV provides the tikv/client-go package, we implemented both the TiKV indexing and search components in Go. Key ranges were constructed according to the search mode: exact match, time range, or prefix search.
// exact match
func BuildExactRange(sourceCode, fieldCode, term string) ([]byte, []byte) {
prefix := fmt.Sprintf("%s%s%s:", sourceCode, fieldCode, term)
return []byte(prefix), append([]byte(prefix), 0xFF)
}
// time range filter (before/after record time)
func BuildTimeRange(fieldCode, term string, from, to string) ([]byte, []byte) {
start := []byte(fmt.Sprintf("%s%s%s:%s", sourceCode, fieldCode, term, from))
end := []byte(fmt.Sprintf("%s%s%s:%s", fieldCode, term, to))
return start, end
}
// prefix search (trailing wildcard)
func BuildPrefixRange(fieldCode, prefix string) ([]byte, []byte) {
start := []byte(fmt.Sprintf("%s%s%s", sourceCode, fieldCode, prefix))
end := append([]byte(fmt.Sprintf("%s%s%s", fieldCode, prefix)), 0xFF)
return start, end
}
//
var startKey, endKey []byte
switch mode {
case "prefix":
startKey, endKey = BuildPrefixRange("g", fieldCode, searchTerm)
case "time":
startKey, endKey = BuildTimeRange("g" fieldCode, searchTerm, from, to)
default:
startKey, endKey = BuildExactRange("g", fieldCode, searchTerm)
}As described earlier, all searchable fields were mapped to a fieldCode. Record time was always stored in YYYYMMDD format, allowing it to participate in range filtering.
There was one important limitation in this index design: trailing wildcard/prefix searches could not be combined with time filtering. Because time had to appear at the end of the key, the index could only prioritize matching the searchable value first.
The resulting key range was passed to Scan, which returned the matching keys:
keys, values, err := tikv.Scan(ctx, startKey, endKey, size+1) // +1 to detect if there is next pageAfter obtaining the matching keys, we parsed their corresponding values into MinIO object paths and byte offsets. Retrieving the actual JSON record then required a single range request to MinIO:
recordMinioObjectPath := valueParts[0]
start, _ := strconv.ParseInt(valueParts[1], 10, 64)
end, _ := strconv.ParseInt(valueParts[2], 10, 64)
// retrieve the json from jsonl in minio
opts := minio.GetObjectOptions{}
opts.SetRange(start, end-1)
obj, err := minioClient.GetObject(ctx, bucket, recordMinioObjectPath, opts)
if err != nil {
log.Printf("GetObject failed: %v", err)
continue
}
buf, _ := io.ReadAll(obj)
obj.Close()Lab #
For the lab setup, we replicated the topology intended for production. This allowed us to develop, benchmark, and build deployment pipelines against a virtual representation of the production environment. The easiest way for us to generate such heavily scaled down versions of the production cluster replicas was using Vagrant.
The clusters were hosted on a single machine with sufficient CPU, disk, and RAM to run all the virtual nodes. This also meant that the benchmark results would be skewed: the virtual environment relied on software RAID, QEMU disks, and internal networking rather than the native production hardware.
The Vagrantfile used to set up the cluster looked roughly like this:
Vagrant.configure("2") do |config|
config.vm.box = "bento/ubuntu-24.04"
config.vm.box_version = "202508.03.0"
config.ssh.insert_key = false
config.vm.provider :libvirt do |libvirt|
libvirt.driver = "kvm"
libvirt.cpu_mode = "host-passthrough"
end
def create_minio_node(config, index, data_disks)
name = "minio-#{index}"
config.vm.define name do |node|
node.vm.hostname = name
node.vm.provider :libvirt do |lv|
lv.cpus = 4
lv.memory = 6144
('c'..('c'.ord + data_disks - 1).chr).each do |letter|
lv.storage :file, :size => '30G', :type => 'qcow2', :device => "vd#{letter}"
end
end
node.vm.network "private_network",
ip: "192.168.100.#{10+index}",
libvirt__network_name: "br100",
libvirt__model_type: "virtio"
node.vm.network "private_network",
ip: "192.168.200.#{10+index}",
libvirt__network_name: "br25a",
libvirt__model_type: "virtio"
end
end
def create_index_node(config, index, data_disks)
name = "index-#{index}"
config.vm.define name do |node|
node.vm.hostname = name
node.vm.provider :libvirt do |lv|
lv.cpus = 4
lv.memory = 6144
('c'..('c'.ord + data_disks - 1).chr).each do |letter|
lv.storage :file, :size => '30G', :type => 'qcow2', :device => "vd#{letter}"
end
end
node.vm.network "private_network",
ip: "192.168.100.#{20+index}",
libvirt__network_name: "br100",
libvirt__model_type: "virtio"
node.vm.network "private_network",
ip: "192.168.201.#{20+index}",
libvirt__network_name: "br25b",
libvirt__model_type: "virtio"
end
end
(1..4).each { |i| create_minio_node(config, i, 24) }
(1..4).each { |i| create_index_node(config, i, 24) }
endWe used a dedicated 100 Gbps interface for inter-cluster communication. The setup of network interfaces required configuring the corresponding virtual bridges separately.
The amount and sizes of disks, core counts, and memory allocations were chosen to make the best possible use of the host machine and, consequently, allow us to ingest as much development data as possible, while keeping the topology intact.
Benchmarks #
Our benchmarks were run against internal and proprietary data, so we can only provide the key metrics from the lab environment.
The purpose of this article is not to compare our architecture with existing off-the-shelf solutions. Instead, the goal is to demonstrate that tailored approaches can still achieve respectable performance in highly constrained environments when the requirements and available resources do not fit a conventional architecture.
We indexed approximately 1.5 TB of records. Because the primary requirement was search round-trip time for 1,000 records rather than indexing speed, our benchmark goals were twofold:
- Validate that the round-trip time requirement could be met on a representative subset of the data.
- Validate that the TiKV index would not grow beyond 2x the source data or introduce excessive ingestion overhead.
Indexing the 1.5 TB of JSON records produced approximately 8 billion TiKV keys. This was satisfactory for us given that the dataset contained around 3 billion JSON records.
Raw disk usage tracked the JSONL storage at roughly a 1.5:1 ratio, suggesting that we were unlikely to exceed the 2:1 cap we had set for ourselves.
These numbers only demonstrate that the design works for our particular use case. Actual index growth remains highly dependent on the nature of the data and the number of searchable fields indexed in each JSON record.
We ran 100 searches, each returning 1,000 records, under various loads against the search endpoint and measured the complete round-trip time.
On average, we observed a full round trip of 4.059 seconds per query for 1,000 returned records. The average size of the 1,000 returned records was approximately 1.63 MB.
These results were encouraging. We did not expect response times to increase significantly as the overall data volume grew. Furthermore, because the lab used a virtualized environment, we expected it to perform worse than the native production setup.
The results therefore gave us confidence in the architectural choices we made under the hardware constraints and suggested that the approach is also scalable toward the intended production volumes.
