Open-Source Benchmarking Tools with Ad-Hoc Extension

Automation is essential for  performance benchmarking  because it ensures that results are reliable, repeatable, scalable, and comparable. Various open source benchmarking tools are used for Automation.  Tools are essential for benchmarking because they bring standardization, accuracy, efficiency, and repeatability to performance evaluation.

Open-Source Benchmarking Tools that support ad-hoc extensibility, meaning they can be customized or extended without rebuilding or heavily modifying the core codebase. These tools provide flexibility in creating custom test scenarios, simulating various workloads, and adapting to new APIs or environments. 

List of tools which we used for benchmarking:

• Phoronix Test Suite

• PerfKit Benchmarker

Phoronix Test Suite:

Phoronix Test Suite is the most comprehensive open-source benchmarking platform available for Linux, macOS, and windows systems. It is widely used for automated testing, performance analysis, and software comparisons.

What is PTS Extension?

A PTS Extension is a plugin or add-on for the Phoronix Test Suite (PTS) that extends its functionality. It allows users to add custom behaviors before, during, or after benchmark runs—ideal for automation, integration, or custom logging.

PTS extensions are used to:

• Add full socket runs

• Add open source docker tests

• Integrate with other systems.

• System & hardware benchmarking

Why Shift from PTS to PerfKit Benchmarker?

• Phoronix Test Suite (PTS) is primarily a single-node benchmarking tool, which runs on a single machine. To overcome this issue Perfkit Benchmarker tool is used. PKB is specifically built for cloud platforms.

• PKB handles provisioning, benchmarking, monitoring, and cleanup automatically. PTS requires manual test setup, especially for cloud VMs.

• PKB can push benchmark data to:

  • InfluxDB

  • Stackdriver

  • Grafana

  • JSON logs for CI/CD systems

         PTS does offer HTML/JSON/CSV output but lacks native telemetry integrations.

Perfkit Benchmarker(PKB):

PerfKit Benchmarker is an open-source tool developed by Google that automates the process of benchmarking cloud infrastructure across different cloud providers. 

Main Stages of a PerfKit Benchmarker Run:

What Is a PerfKit Benchmarker Extension?

 Extensions allow users to define:

• Custom benchmark

• Flags

• Providers

• Workloads

Top Benefits of PerfKit Benchmarker Extension:

• PerfKit Benchmarker can run distributed benchmarks involving multiple VMs across one or more cloud zones or providers.

• Automatically handles VM provisioning, software installation, test execution, teardown.

• Easily integrates with dashboards, analytics pipelines, or cost/performance reports.

• Useful in capacity planning, performance regression testing, or SLI validation.

• In addition PKB Extension supports Turbostat(useful for analyzing power and frequency behavior during benchmarks), Lm-Sensors(Linux utility used to monitor hardware sensors), and Sysstat(analyze CPU, memory, disk I/O, networking, and other system-level performance metrics.).

• PKB extension also support additional feature for Report generation, which is useful to generate report with all result and peripheral data. It supports various formats such as TXT, CSV and HTML.

Here’s a set of workload charts for PerfKit Benchmarker (PKB) organized by category. These charts summarize the common benchmark workloads available in PKB, helping you choose the right tests for CPU, memory, disk, network, and database performance analysis across cloud platforms.

Cloud Comparison Using PerfKit Benchmarker

Here's a comprehensive comparison of cloud providers (GCP, Azure, OCI) using PerfKit Benchmarker (PKB) as a common benchmarking framework:

---

Conclusion

• PTS is excellent for deep technical benchmarking of a single system.

• PKB is a robust choice for cloud performance comparisons, cost evaluation, and infrastructure benchmarking at scale.