The past six months have brought a wave of BigData and Data Lake projects. We won’t touch the software side — everyone picks their own stack there. But on hardware, phasing, and sizing we keep seeing the same set of rakes people step on with impressive regularity. So we decided to break it down separately.
It usually starts with a specific request. A customer comes in: “we need 150 terabytes of storage for an analytics platform.” Where does 150 come from? We open the farm diagram, and there it is — someone added up the disks of every VM, and that’s how the round number appeared.
On its own, that number means nothing. Behind it hide three questions, and until they’re answered, any specification is guesswork. Expensive guesswork, too: the mistake doesn’t surface at procurement — it surfaces six months later, when the space has run out and the budget is already spent.
Nature of the data
Question one: what actually compresses
The array works with whatever reaches it. Anything compressed or encrypted inside the virtual machine arrives as an already-finished stream.
Let’s start from the end, because this is the most expensive mistake of the three.
The logic is trivially simple. The array works with whatever reaches it. If data is compressed or encrypted inside the virtual machine, what reaches the array is an already high-entropy stream — there’s nothing left to compress. Neither compression nor compaction will help, because the work has already been done a layer up.
Let’s break it down further, because there are a lot of nuances here.
What will never compress. Anything encrypted at the application or OS level — TDE in the DBMS, LUKS and BitLocker inside the VM, SSE in object storage. Exactly 1:1, no exceptions. A good cipher produces output that’s statistically indistinguishable from random.
An important caveat that gets confused constantly: SED drives and volume encryption done by the storage array itself have no effect on the ratio at all. They operate after compression. The problem is only encryption above the array level.
Next come already-compressed formats. Archives, media, deduplicated backups. Here it’s 1.0–1.05:1, and you simply can’t count on them when sizing.
The weighted ratio for such a platform comes out at 1.1–1.2 to 1. Yet the capacity calculation still assumes 2 to 1.
What won’t compress even though it looks innocent. This is where the real surprises live.
ClickHouse. MergeTree compresses data with its own codecs — LZ4 by default, often ZSTD. What lands on the array is already finished. Someone plugs in 2:1 “because it’s a database” — and misses by a factor of two.
Kafka. Depends on the producers’ compression.type. With snappy, lz4, or zstd, it’s already compressed on the client, and the array gets 1.05–1.15:1. With none, log segments compress beautifully, 3–5:1. This is the one heavy component where the answer might actually work in your favor. Go and ask.
Parquet and ORC with snappy or zstd. Data lake classics — and classically incompressible. If that’s what sits behind the term “object storage” in your case, forget deduplication as a sizing factor.
Databases with internal page compression — Oracle Advanced Compression, HCC, SQL Server page compression. A separate fork: sometimes it’s more cost-effective to turn compression off in the DBMS and hand it to the array, taking the load off server CPUs. But that’s a decision requiring a mandatory performance test, not a “let’s just try it in production.”
What compresses beautifully. Text logs, Airflow and Superset metadata databases, configuration files, source code — 3–10:1. Uncompressed CSV, JSON, XML, dumps. PostgreSQL without table compression. OS images of identical VMs.
Deduplication is a separate mechanism, and people forget about it. It works independently of compressibility. Two hundred identical VMs cloned from one template: every block is incompressible, but blocks repeat, and dedup delivers a great result. Conversely, a unique, incompressible dataset won’t yield to compression or dedup either. There it’s an honest 1:1, and no vendor guarantee will change that.
Now the arithmetic this was all for. Take a typical stack: ClickHouse nodes, object storage, Kafka brokers. In a normal project that’s around 90% of the total allocated volume. And all three compress data themselves. The nicely compressible part — metadata databases, logs, configs — is a few percentage points, and it has no effect on the overall ratio.
The weighted ratio for such a platform comes out at 1.1–1.2:1. Yet the capacity calculation still uses 2:1, because that’s what’s written in the standard virtualization sizer. The gap between these two numbers is exactly the delta someone will discover six months later. Usually not the person who signed off on the spec.
Placement architecture
Question two: where the data should physically live
Engines with their own replication belong on local disks. The array is needed where fast failover matters.
ClickHouse, Kafka, Greenplum, HDFS, and Elasticsearch were designed for local disks. They have their own replication, their own view of data placement, their own node-failure recovery mechanisms. This is shared-nothing architecture, and it’s not about the array.
Put something like that on shared storage and you get double protection: RAID on the array plus the engine’s own replicas. Capacity gets spent twice. And the bottleneck moves into the fabric, where every node in the cluster starts competing for the same ports. Instead of a distributed system, you end up with a distributed system that has one shared point of failure and one shared queue.
The reverse gets forgotten more often, and that’s a mistake too. Master nodes, catalogs, schema registries, metadata databases, VM images — they don’t need throughput. They need fast failover when a host dies, snapshots, and predictable recovery. That’s exactly what the array is bought for. Keeping them on local disks means voluntarily giving up HA in the one place where it comes almost for free.
A separate note on the object tier, which in our part of the world somehow always gets considered last. ClickHouse, Kafka, Greenplum, and Spark can all offload cold data to S3 today. ClickHouse — via S3 disk and storage policies. Kafka — via tiered storage. This is usually the cheapest way to handle volume growth, and it’s almost always left out of the initial discussion. Then it turns out 60% of the data hasn’t been touched in six months and could easily have been sitting on QLC capacity at a third of the price.
Virtualization
Question three: what to virtualize
The line isn’t drawn by product name, but by required throughput and acceptable latency jitter.
There’s rarely an honest yes-or-no answer here. The line isn’t drawn by product name, but by required throughput and by how critical latency jitter is.
Metadata databases, connectors, orchestration, registries, Karaf with its bundles, monitoring — these virtualize without a second thought. I/O load is low, and the payoff from live migration, snapshots, and fast recovery outweighs everything else. There’s nothing to argue about here.
Greenplum segments, DataNode, Kafka under a heavy write load, Elasticsearch data nodes — these usually stay on bare metal. Not because they “won’t work” — they’ll work fine. But the engine expects direct access to NVMe, and the hypervisor hands back predictability with caveats. You won’t see the difference in tests; you will see it in production under load.
Monitoring shows everything is fine, and the system is unusable.
Between these two extremes lies the largest and most interesting zone, where configuration decides everything. And here’s an observation from practice: problems in the middle zone almost never come from a shortage of resources. They come from vCPU overcommit and from the VM being smeared across sockets.
The symptom is painfully familiar: scheduler wait time climbs, CPU utilization looks modest, every graph is green, and users complain about stutter. Monitoring shows everything is fine, and the system is unusable. Ready time and co-stop are the first thing to check, not the last.
The second classic source is a shared datastore. An analytics VM eats up the queue while something latency-sensitive lives right next to it on the same LUN. Separate them at the design stage, not once it starts hurting.
Data pipelines
The glue: the last thing anyone remembers
Averaged, ward-wide sizing breaks exactly on the glue components: every one of them has its own requirements profile.
Everyone remembers ClickHouse and Kafka. Pipelines — almost never, and that’s a dozen components, each with its own character.
Kafka Connect and Debezium. A CDC stream out of transactional databases. The disk needs almost nothing, but memory for buffers and a stable network are essential. Latency suffers badly if this ends up on an overcommitted host.
NiFi. This is the one that’s systematically underestimated. It has three repositories — flowfile, content, provenance — and all three write actively. Under load, the content repository reliably generates more IOPS than people expect. Provenance grows quietly and one day fills the disk completely. Local NVMe, separate volumes per repository, fill-level monitoring — none of this is a luxury.
Apache Karaf and OSGi containers in general. Modest in resource needs, lives happily on a VM. What it actually needs is a fast restart and a snapshot before deploying bundles, because hot rollbacks happen more often than anyone would like. This is an argument for virtualization, not against it.
Airflow. Metadata database, task logs, DAGs. Light on resources, but logs grow linearly and endlessly if rotation isn’t configured. We’ve seen a 200-gigabyte Airflow metadata database bring down the scheduler simply because nobody cleaned up the run history.
Schema Registry, ZooKeeper, etcd. The volumes are trivial, but ZooKeeper and etcd are extremely sensitive to fsync latency. A slow disk under etcd, and the cluster starts re-electing its leader for no apparent reason. You can’t just put them “wherever there’s room.”
Trino and Presto. The coordinator virtualizes without issue. Workers want memory and network, and for spill — a local fast disk. If spill goes out over a network volume, heavy joins turn into a pumpkin.
Superset and Metabase. The BI layer, small metadata databases, but query caches grow. They go on a VM, live on the array, and raise no questions.
The point of this whole list is simple: averaged, ward-wide sizing breaks exactly on the glue components. You can’t take the farm’s total volume and divide it by the number of servers. One component needs memory, another needs IOPS, a third needs low fsync latency at a tiny volume. You have to size by profile.
Hardware map
Now the hardware: what fits what
A platform is almost always assembled from several types of hardware. Trying to get by with one is the source of most problems.
Let’s move on to specifics, because abstractions are fine right up until you sign the specification.
Let’s say this up front: a platform is almost always assembled from several types of hardware. Trying to get by with one type is exactly the source of most of the problems we’ve described above.
Lenovo ThinkAgile HX (Nutanix) — for the glue and the platform layer.
This is where HCI plays to its strengths. Metadata databases, Airflow, NiFi for small pipelines, Karaf, CDC connectors, registries, the BI layer, master nodes, dev and test environments. Everything where the value lies in manageability, snapshots, and fast recovery, not in squeezing out the last microsecond.
Nutanix Unified Storage deserves a separate look. It has an S3-compatible object layer that covers the cold tier for analytics right on the same cluster, with no separate hardware. Plus Files for shared datasets and Volumes for block access. For mid-sized platforms this often removes the need for a separate storage array entirely.
Currently: the ThinkAgile HX650 V4 on Intel Xeon 6 is the workhorse for this layer. The HX650a V4 — if that same cluster also needs GPUs for inference.
Lenovo ThinkSystem bare metal — for engine data.
Everything that’s shared-nothing and disk-hungry: ClickHouse nodes, Kafka brokers under load, Greenplum segments, DataNode, Spark and Trino workers, Elasticsearch data nodes, model-training nodes.
The SR650 V4 in 2U delivers up to 36 NVMe drives and up to 10 PCIe Gen5 slots — plenty for an analytics node. Xeon 6 with P-cores, up to 86 cores per socket, MRDIMM and CXL support. For ClickHouse and Greenplum, where everything comes down to memory and its bandwidth, that matters noticeably.
An important point that often gets missed: analytics doesn’t need the fastest processor, it needs the right balance of cores, memory, and NVMe lanes. A top-tier CPU starved of disk lanes will just sit waiting for data. This is exactly the case where skimping on disks makes overpaying for the processor pointless.
Lenovo DG Series (QLC, capacity) — for the cold tier, archive, and backups.
QLC flash on ONTAP. Cheap capacity that’s still flash. The ideal spot for the cold tier, shared datasets over NFS, and backup target storage. It has S3 right on the array, which removes the need for separate object storage for cold data.
One nuance from practice: usable capacity on these arrays runs roughly a third below raw — right-sizing, ADP partitioning, RAID-DP dual parity, WAFL reserve. That’s normal for enterprise-class all-flash, but it needs to be built into the calculation from the start, not discovered at commissioning. And you need to count in the same units as the requirement: sizers calculate in base-2 but the spec gets signed in base-10 — roughly a 10% gap.
Lenovo DM Series (NVMe, performance) — for whatever needs speed and HA.
Master nodes, catalogs, metadata databases, datastores for virtualization, shared datasets where latency matters. Unified — meaning both block and file. When you need fast NFS for shared training data, this is where it goes.
Lenovo DE and DS Series (block SAN) — for datastores and supporting DBMSs.
Classic block access. DE is midrange, available hybrid and all-flash; DS is block all-flash. Datastores for virtualization, supporting databases — wherever you don’t need file access or ONTAP features.
What not to do. Don’t put Greenplum segments and ClickHouse nodes on block storage, no matter how fast it is. Don’t keep metadata databases and master nodes on local disks with no HA. Don’t buy a single array “for everything” and then have to explain why analytics is slow and backup won’t fit in its window.
Procurement phasing
On phasing, since we’re here
A data platform is built iteratively, yet procurement somehow keeps getting planned as one big delivery three years out.
A separate headache — not about hardware, but about the same money.
A data platform is built iteratively. First a test bench and a pilot on a couple of nodes, then production, then expansion for the real workload. The problem is that procurement somehow keeps getting planned as one big delivery “three years out.”
The result is predictable: half the capacity sits idle heating the room in year one, and by year three it turns out the workload profile changed and something completely different was actually needed. On top of that, the hardware ages while the warranty clock runs from delivery, not from go-live.
It’s smarter to build in expandability and buy for the current phase plus a clear horizon. Fortunately, HX, ThinkSystem, and DG all expand cleanly with nodes and disks, with no service interruption.
That said, there’s a counterargument that’s louder than usual in 2026: AI-driven demand growth has made lead times unpredictable. Long deliberation has gotten expensive. So the balance between “don’t overpay now” and “don’t wait six months later” is something everyone has to find for themselves. But it has to be a conscious search, not the principle of “bought extra just in case.”
Checklist
Questions to ask before sizing the spec
Six questions that turn a round number into a calculation you can defend.
A short list that saves weeks of back-and-forth:
- Codecs. What’s configured in ClickHouse MergeTree, what compression.type the Kafka producers use, what parquet in the object tier is compressed with.
- Encryption. Is there TDE, LUKS, SSE — anything above the array level.
- Actual fill level. How much is written on the disks today, not how much is allocated. The gap is usually a multiple.
- Growth forecast. Over the planning horizon, broken down by component. Kafka with a 7-day retention and ClickHouse holding three years of history grow in completely different ways.
- Access profile. What share of the data is actually read in a month. This answers the question of whether a cold tier is needed.
- Recovery requirements. RPO and RTO for each layer. The Airflow metadata database and raw Kafka data have different ones, and protecting them identically is overspending.
Checking compressibility estimates isn’t hard: most vendors have utilities that scan a directory of real data and produce a forecast before you even migrate anything. Half a day of work against months of explaining why the space ran out.
Bottom line
Instead of a conclusion
A round number in a request isn’t a requirement — it’s a symptom. It means someone added up VM disks and didn’t ask a single one of the six questions above.
The good news: all six can be settled in a couple of meetings with the platform team. The bad news: without them, any specification is a bet, not a calculation. And it gets tested not on paper but in production, six months later, when changing anything is already expensive.
Planning a data platform? Get in touch — we’ll work through your specific configuration. And while we’re at it, we’ll calculate what will actually compress for you.