From Telegraf to Vector: Building Reliable Metrics Pipelines for ClickHouse®

Collecting system metrics is relatively straightforward.

Building a reliable metrics pipeline into ClickHouse® – one that handles transformations, schema alignment, and real-world debugging – is where complexity begins.

In many data engineering workflows, metrics are not just stored for analysis. They are also used to power dashboards, observability systems, and operational insights.

This article explores a practical approach to building a metrics ingestion pipeline into ClickHouse®, including lessons learned when moving from Telegraf to Vector.

The Problem: Metrics Ingestion into ClickHouse®

A typical requirement in observability systems involves:

• Collecting host metrics (CPU, memory, disk)
• Capturing GPU-level metrics
• Ingesting this data into ClickHouse®
• Making it queryable for analytics and dashboards

While this seems straightforward, challenges emerge when dealing with:

• Data format consistency
• Schema alignment
• Transformation requirements
• Debugging ingestion failures

Initial Approach: Telegraf-Based Metrics Pipeline

Telegraf is a widely used tool for collecting system and application metrics.

Its plugin-based architecture makes it a natural starting point for building a metrics pipeline.

However, in practice, certain limitations became apparent:

• No native output integration for ClickHouse®
• Limited flexibility in transforming data before ingestion
• Debugging issues without clear visibility into data flow
• Increasing rigidity as pipeline complexity grows

These challenges made it difficult to reliably shape and route data into ClickHouse®.

Rethinking the Pipeline Approach

One key realization was that the issue was not just tooling – it was the lack of control over how data moved through the system.

Instead of thinking in terms of configuration:

Write config → Run → Expect output

A more effective model is:

Data source → Transformation → Destination

This shift – from configuration-driven thinking to pipeline-driven thinking – is essential when building scalable data systems.

Moving to Vector for Better Control

To address these limitations, the pipeline was redesigned using Vector.

Vector introduces a clear and explicit pipeline model:

Sources → Transforms → Sinks

This approach provides:

• Clear visibility into data flow
• Fine-grained control over transformations
• Flexible routing of metrics
• Better alignment with ClickHouse® ingestion requirements

Understanding the Vector Pipeline Model

Each stage in a Vector pipeline plays a critical role:

• Sources – collect data from systems (host metrics, GPU metrics)
• Transforms – reshape and normalize data
• Sinks – deliver the processed data to destinations such as ClickHouse®

The pipeline only functions correctly when each stage is properly defined and connected.

Common Challenges in Metrics Pipelines

Even with a structured pipeline model, several challenges arise in practice:

• Incorrect source configurations leading to missing data
• Misconfigured sinks causing ingestion failures
• Data not flowing due to transformation issues
• Silent failures when pipeline stages are not correctly linked

In many cases, configurations appear correct, but small issues in one stage can prevent the entire pipeline from functioning.

Why Transformations Matter

One of the most critical aspects of a metrics pipeline is the transformation layer.

Raw system metrics often do not match the schema expected by ClickHouse®. This requires:

• Normalizing field structures
• Ensuring correct data types
• Handling null or missing values
• Converting timestamps into acceptable formats

Without proper transformations, even correctly collected data cannot be reliably ingested.

Key Takeaways

Building a reliable metrics pipeline into ClickHouse® requires more than selecting the right tools.

Some key lessons:

• Metrics ingestion pipelines require explicit control over data flow
• Transformation is a core part of the pipeline, not an optional step
• Debugging is an integral part of building data systems
• Small configuration issues can impact the entire pipeline

Most importantly:

Reliable pipelines are built by understanding how data flows – not just by configuring tools.

What’s Next

While the pipeline model introduced by Vector improves visibility and control, the real complexity lies in the transformation layer.

In the next part, we’ll explore how metrics need to be reshaped to match ClickHouse® expectations, including handling data types, timestamps, and schema alignment.

Final Thoughts

Modern data systems require pipelines that are both flexible and reliable.

While tools like Telegraf are effective for basic metrics collection, more complex use cases benefit from a pipeline-oriented approach using tools like Vector.

More importantly, treating pipelines as systems – not just configurations – leads to more predictable and maintainable data workflows.

References

ClickHouse® Data ingestion
Vector documentation
Telegraf Documentation