Performance engineering is a vital area in software development that guarantees applications function efficiently and effectively. As modern software systems grow more complex, the need for skilled engineers who understand performance becomes increasingly important. This guide will cover five essential performance engineering skills every software engineer should develop to thrive in their careers. Grasping Performance Requirements To start, software engineers must excel at understanding performance requirements. This means knowing how the system behaves under different loads and the specific performance targets the application must meet. Involved discussions with stakeholders are crucial for defining clear performance metrics early in the development process. Important performance indicators (KPIs) include: Response Time : The time taken for a system to respond to a user request. According to a report, 47% of consumers expect a page to load in two seconds or less. Throughput : The amount of work completed in a given timeframe, often measured in transactions per second (TPS). Resource Utilization : Understanding how effectively system resources are being used, such as CPU, memory, and bandwidth. By setting these performance requirements early on, engineers can make better design decisions, leading to more efficient applications right from the beginning. Expertise in Performance Testing Tools A strong command of performance testing tools is essential. Knowledge of both open-source and proprietary tools enables engineers to simulate user traffic, evaluate system performance, and pinpoint potential problems. Some popular performance testing tools include Apache JMeter, LoadRunner, and Gatling. These tools help engineers create test scenarios that reflect real-world load conditions. For instance, a team using JMeter might simulate 10,000 concurrent users on their e-commerce site to ensure it can handle peak shopping times, like Black Friday. Effectively using performance testing tools helps reveal issues and provides actionable insights for optimization. In fact, organizations that conduct regular performance testing see a 30% improvement in application speed and responsiveness. Capacity Planning and Scalability A third essential skill is capacity planning and scalability. Software engineers must be able to forecast the resources needed to accommodate user growth without compromising performance. This involves analyzing historical usage data and anticipating future demands. For example, if a SaaS application reports a 20% monthly increase in active users, engineers must plan to scale infrastructure accordingly. This scaling can happen in two ways: Vertical Scaling : Adding more resources (like CPU or memory) to a single server. Horizontal Scaling : Adding more servers to distribute the load when the user demand increases. Team members should consistently monitor performance against these plans to refine forecasts and implement necessary adjustments. Mastering this skill enables teams to prevent performance issues and support seamless scaling as user needs change. Appreciating System Architecture A solid understanding of system architecture is crucial for performance engineering. Engineers need to be familiar with various architectural patterns such as microservices, serverless, and monolithic designs. Each architecture has its implications for performance. For example, a microservices architecture can enhance scalability but may lead to communication delays between services. In contrast, a monolithic architecture is easier to manage but might struggle under high loads due to its rigid structure. Understanding how different architectures influence performance helps engineers make informed design choices. For instance, a recent study showed that companies implementing microservices correctly reduced deployment times by 75%. Ongoing Performance Monitoring Lastly, ongoing performance monitoring is a critical skill that software engineers should cultivate. After an application is live, continuous monitoring allows teams to spot performance issues that may arise in real-world settings. Using tools like New Relic, Dynatrace, or Grafana helps engineers monitor application performance consistently. For instance, real-time monitoring can quickly alert teams when server response times exceed predefined limits, preventing user dissatisfaction. By integrating ongoing monitoring into their workflow, engineers foster a culture of performance awareness. Companies that prioritize performance monitoring often see conversion rates improve by up to 20% due to enhanced user experiences. Time to Enhance Performance Engineering Skills Mastering performance engineering skills is a necessity for software engineers, not just an option. With the increasing complexity of software systems, it is essential for engineers to possess the knowledge and tools required to ensure that applications meet crucial performance metrics. Focusing on understanding performance requirements, mastering performance testing tools, capacity planning and scalability, system architecture knowledge, and continuous performance monitoring can significantly boost engineers' effectiveness in this important field. As the demand for high-performance applications continues to rise, developing these skills will enhance individual careers while contributing to the success of software projects. Now is the time for aspiring engineers to invest in their own development and polish these performance engineering skills. Success is found in mastering these elements and effectively applying them to real-world challenges.

In the fast-paced world of technology, chip benchmarking is vital. It helps engineers and developers measure the performance of semiconductor devices to keep up with advancements. This post dives into the top five tools for chip benchmarking, highlighting their features, benefits, and real-world applications. 1. Geekbench Geekbench stands out as a cross-platform benchmarking tool for assessing CPU and GPU performance. Its versatility allows it to work seamlessly across different operating systems, making it a favorite among developers. With a massive database of devices, Geekbench offers detailed scores that let users compare their hardware easily. It measures both single-core and multi-core performance, crucial for modern chips that handle multiple tasks simultaneously. For instance, Geekbench allows you to see how a chip like the Apple M1 stacks up against Intel's latest processors. Setting up Geekbench is quick and user-friendly. It provides insights into memory and compute performance, making it an essential tool for hardware professionals. In fact, many developers report improvements of up to 30% in their designs after optimizing based on Geekbench results. 2. SPEC CPU Benchmark The SPEC CPU Benchmark suite is trusted in the industry for evaluating CPU performance. Created by the Standard Performance Evaluation Corporation, it includes a set of diverse workloads assessing integer and floating-point calculations. SPEC offers reliable reports that reveal both efficiency and speed, enabling engineers to make data-driven decisions. For example, analysis from SPEC has helped companies like AMD refine their latest Ryzen processors, enhancing performance by approximately 25%. SPEC's rigorous validation ensures that results are credible for manufacturers and users alike. Its broad application makes it perfect for systems that demand high performance, such as servers running complex applications. 3. 3DMark 3DMark is essential for gamers and graphics professionals. This graphical benchmarking tool primarily evaluates GPU performance in rendering graphics but can also provide key insights into chip performance concerning integrated graphics. 3DMark includes various tests reflecting real-world gaming scenarios. Users can examine frame rates and rendering speeds, helping them understand how their hardware performs under strain. For instance, the “Fire Strike” test can assess how well a system handles intensive gaming tasks, highlighting up to a 15% difference in performance between competing GPUs. Additionally, the "Time Spy" test evaluates DirectX 12 performance. These visual benchmarks not only present performance data engagingly but also help users spot design flaws in their chips. 4. LLaMA Benchmarks LLaMA (Large Language Model Meta AI) benchmarks are designed to evaluate the performance of various language models across multiple tasks. These benchmarks provide a standardized way to measure the capabilities of models in understanding and generating human-like text. The benchmarks include a wide range of tasks, such as text completion, question answering, and summarization, allowing researchers to assess the models' effectiveness in real-world applications. For instance, recent evaluations have shown that LLaMA models outperform previous iterations in generating coherent and contextually relevant text. One of the key features of LLaMA benchmarks is their focus on zero-shot and few-shot learning capabilities. This aspect enables models to perform well on tasks they have not been explicitly trained for, showcasing their adaptability and generalization abilities. 5. GPT-3 Benchmarks GPT-3 benchmarks provide a comprehensive framework for assessing the performance of the GPT-3 language model across various linguistic tasks. These benchmarks measure aspects such as fluency, coherence, and relevance in generated text. The evaluation includes a variety of tasks, including language translation, text generation, and creative writing, allowing for a holistic view of the model's capabilities. For example, companies utilizing GPT-3 for content creation have reported significant improvements in engagement and quality due to the insights gained from these benchmarks. The user-friendly interface of the benchmarking tools associated with GPT-3 ensures that both novice and experienced users can easily interpret the results. This accessibility has led to its widespread adoption in industries seeking to leverage advanced natural language processing technologies. Making the Right Choice Selecting the appropriate tool for chip benchmarking is crucial. Whether it's the adaptable Geekbench, the trusted SPEC CPU Benchmark, the detailed 3DMark, the versatile SiSoftware Sandra, or the all-encompassing PassMark PerformanceTest, each tool provides unique insights that can foster innovation in chip development. By effectively utilizing these tools, professionals can enhance the performance of semiconductor devices, keeping pace with rapid technological changes. Investing in the right benchmarking tools is not just beneficial; it is vital for success in chip development.

Benchmarks are vital for evaluating High-Performance Computing (HPC) system performance, guiding hardware choices, and optimizing software. This whitepaper focuses on understanding and overcoming bottlenecks in HPC benchmarks for CPU environments, specifically considering ARM/AARCH64 architectures, using miniFE and GROMACS as examples. 1. Introduction to miniFE and GROMACS Benchmarks 1.1. miniFE: A Finite Element Mini-Application miniFE, part of the Mantevo suite, simulates implicit finite element applications. It solves sparse linear systems, with its core kernels focused on element-operator computation, assembly, sparse matrix-vector products (SpMV), and basic vector operations. It's excellent for benchmarking systems handling sparse linear algebra and iterative solvers. To run miniFE, you typically compile it with an MPI-enabled compiler. Execution involves specifying problem dimensions and MPI processes. # Example for a single node with 16 MPI tasks srun -N 1 -n 16 miniFE.x -nx 264 -ny 256 -nz 256 # Example for a multi-node run (adjust N and n) srun -N 4 -n 64 miniFE.x -nx 528 -ny 512 -nz 512 Note: srun is for SLURM; mpirun or similar for other systems. 1.2. GROMACS: Molecular Dynamics Simulation Software GROMACS (GROningen MAchine for Chemical Simulations) is a highly optimized open-source software for molecular dynamics (MD) simulations. It models atomic and molecular movements, particularly for biochemical systems, and is efficient in calculating non-bonded interactions. A typical GROMACS workflow prepares input files, then runs the simulation. # Step 1: Prepare the run input file (.tpr) gmx grompp -f pme.mdp -c conf.gro -p topol.top -o topol.tpr # Step 2: Run the molecular dynamics simulation mpirun -np 4 gmx_mpi mdrun -s topol.tpr -ntomp 4 # To run a specific benchmark system (e.g., 'benchPEP-h') mpirun -np gmx_mpi mdrun -s benchPEP-h.tpr -ntomp Note: Tune MPI processes (-np) and OpenMP threads (-ntomp) to your hardware. 1.2. GROMACS: Molecular Dynamics Simulation Software GROMACS  (GROningen MAchine for Chemical Simulations) is a highly optimized open-source software for molecular dynamics (MD) simulations. It models atomic and molecular movements, particularly for biochemical systems, and is efficient in calculating non-bonded interactions. A typical GROMACS  workflow prepares input files, then runs the simulation. # Step 1: Prepare the run input file (.tpr) gmx grompp -f pme.mdp -c conf.gro -p topol.top -o topol.tpr # Step 2: Run the molecular dynamics simulation  mpirun -np 4 gmx_mpi mdrun -s topol.tpr -ntomp 4 # To run a specific benchmark system (e.g., 'benchPEP-h') mpirun -np gmx_mpi mdrun -s benchPEP-h.tpr -ntomp Note: Tune MPI processes (-np) and OpenMP threads (-ntomp) to your hardware. 2. Interpreting Performance Output (Benchmarking POV) Understanding benchmark output is crucial for evaluating HPC system throughput and efficiency. 2.1. miniFE Performance Metrics miniFE outputs performance data, primarily focused on: Total CG Mflops (Mega-Floating Point Operations per Second for Conjugate Gradient solve): The main Figure of Merit (FOM). Higher values indicate better performance, reflecting the system's efficiency in sparse linear algebra, often limited by memory bandwidth and FPU throughput. 2.2. GROMACS Performance Metrics GROMACS provides detailed output, with the key metric being: ns/day (nanoseconds per day): The standard performance metric for GROMACS. It shows how many nanoseconds of simulated time can be computed per real-world day. A higher ns/day means faster simulation. This metric is ideal for comparing different CPU architectures or configurations. Other useful outputs include Total Wall Time and a breakdown of time spent in different force calculations, which helps pinpoint specific bottlenecks. 3. Bottlenecks in Running HPC Benchmarks Achieving peak HPC performance requires identifying and mitigating bottlenecks that limit system throughput. 3.1. miniFE Specific Bottlenecks miniFE is particularly sensitive to: Memory Bandwidth: The Sparse Matrix-Vector Product (SpMV) is highly memory-bandwidth bound due to irregular memory access patterns. Cache Misses: Irregular accesses lead to frequent cache misses, increasing data retrieval latency. Inter-node Communication (for large problems): For distributed problems, communication during assembly and the Conjugate Gradient solver can be limited by network latency and bandwidth. 3.2. GROMACS Specific Bottlenecks For GROMACS, key bottlenecks include: CPU Core Performance & Threading: The number of cores and their individual performance (Instructions Per Cycle (IPC), clock speed) directly impact ns/day. Optimal balance between MPI ranks and OpenMP threads per rank is crucial. Memory Bandwidth: The CPU needs to access large datasets frequently for force calculations. SIMD Vectorization: GROMACS heavily relies on CPU SIMD instructions (e.g., NEON). If the CPU architecture or compiler doesn't fully exploit these, performance will suffer. Cache Utilization: Efficient cache usage is critical for the main simulation loop. Inter-node Communication: For large systems simulated across multiple nodes, MPI communication for domain decomposition and force summation can be a significant bottleneck, even with fast interconnects. NUMA Effects: Proper process and memory binding is crucial on multi-socket systems to minimize cross-socket memory access latency. Load Imbalance: Uneven workload distribution across PP and PME leads to idle compute units. 3.3. Dynamic Monitoring for Bottleneck Analysis (Frequency, Power, Temperature) Beyond static analysis, dynamic monitoring of CPU frequency, power consumption, and temperature during benchmark execution provides invaluable insights for root-causing performance bottlenecks. This data, when mapped over the run duration, can reveal transient issues that logs alone might miss. Application-Specific Context: For miniFE, if memory bandwidth is the primary bottleneck, the CPU might not be fully utilized, leading to lower-than-expected power consumption and temperatures, even if the frequency remains high. Conversely, if the SpMV operations push the CPU's compute capabilities, sustained high power and temperature could be observed. Any sudden dips in Mflops alongside frequency drops would directly point to thermal or power throttling. For GROMACS, which can be highly compute-intensive, sustained high power consumption and temperatures are common. Analyzing frequency, power, and temperature trends can reveal if the ns/day performance is being limited by the CPU's ability to maintain its turbo frequencies due to thermal constraints or if it's hitting a configured power envelope. Discrepancies between expected maximum performance and observed ns/day can often be explained by these dynamic system responses.  Tools for Monitoring:  Various tools can collect this data, including vendor-specific utilities (e.g., Intel's pcm, AMD's uProf), Linux tools (perf, turbostat, sensors), or IPMI/BMC interfaces for server-level metrics. Correlating these dynamic metrics with the benchmark's reported performance can significantly aid in precise bottleneck identification and system optimization. Conclusion Effective HPC benchmarking goes beyond simply running an application and reporting a single performance number. As demonstrated with miniFE and GROMACS in a CPU-centric environment, a deep understanding of the benchmark's computational characteristics is essential. Identifying whether a workload is memory-bound, compute-bound, or communication-bound is the first step toward optimizing performance. Furthermore, leveraging dynamic monitoring of CPU frequency, power consumption, and temperature provides invaluable diagnostic data. By integrating performance metrics with detailed system telemetry, HPC administrators and researchers can precisely pinpoint bottlenecks, fine-tune system configurations, and ultimately extract the highest possible performance.

In this case study, we describe our systematic approach to benchmarking and validating workloads on FPGA platforms using HAPS (High-performance ASIC Prototyping System) models. The workflow involves compiling and cross-compiling a diverse set of workloads using both native QEMU and the open source toolchain, executing them on FPGA hardware, and capturing detailed performance metrics such as instructions executed and cycle counts.  1. Benchmark Preparation and Build Process We classify our benchmarks into the following categories: High-Performance Computing (HPC) Benchmarks: Includes matrix multiplication, FFT, and other numerical kernels. Synthetic Benchmarks: Includes whetstone, dhrystone, and other CPU stress tests. Algorithmic Benchmarks: Includes sorting algorithms, graph traversal, and numerical integration. Cryptography and Security Benchmarks: AES, RSA, SHA-based microbenchmarks (in future pipeline). Memory and I/O Benchmarks: Includes stream, memcpy stressors, and file read/write tests. Industry-standard Benchmarks: SPEC CPU2017 for INT and FP tracks. All benchmarks are first built or cross-compiled depending on their compatibility: Native Build: Performed on QEMU-based emulation environment where toolchain compatibility allows. Cross Compilation: Done using toolchain targeting the architecture for cases where native build fails or is time-prohibitive. Application Categories Distribution 2. Deployment and Execution on FPGA The compiled binaries are deployed to the FPGA via HAPS models configured with a soft-core. Execution is controlled using a lightweight shell interface or boot script. We utilize a custom performance monitoring utility (akc_counter_capture) to gather the following metrics: Total instruction count Cycle count These values are stored for each benchmark run and are used in performance comparisons. 3. Workload Example  1: DGEMM (Double-Precision General Matrix Multiply) DGEMM is a key linear algebra kernel from the BLAS library. We compiled and executed the DGEMM kernel using double-precision arithmetic with matrix size NxN, where N=256. Performance was evaluated using instruction count, cycle count, and IPC (Instructions Per Cycle). 2: N-Queens Problem The N-Queens benchmark is a classic example of combinatorial search used to evaluate control-flow-heavy algorithm performance. It computes all valid arrangements of N queens on an N×N chessboard such that no two queens attack each other. We verified correctness by comparing the total number of valid solutions for standard board sizes (e.g., N=12 and N=14), which matched precisely across architectures. The benchmark’s output was deterministic, and no deviations were observed across multiple FPGA runs. 3: Red-Black Tree (RBTree) Manipulation Red-Black Tree (RBTree) manipulation represents a memory-bound and pointer-intensive workload that tests dynamic memory access patterns and data structure balancing algorithms. This benchmark was compiled using both the Embedded toolchain and natively on QEMU for consistency. Validation involved verifying the in-order traversal of the tree after bulk insertions and deletions. RBTree serves as a robust test of both instruction scheduling and memory subsystem behavior.  Conclusion Our approach demonstrates that workloads can be effectively compiled, executed, and validated on FPGA platforms using HAPS models.

Automation is essential for  performance benchmarking  because it ensures that results are reliable, repeatable, scalable, and comparable. Various open source benchmarking tools are used for Automation.  Tools are essential for benchmarking because they bring standardization, accuracy, efficiency, and repeatability to performance evaluation. Open-Source Benchmarking Tools that support ad-hoc extensibility, meaning they can be customized or extended without rebuilding or heavily modifying the core codebase. These tools provide flexibility in creating custom test scenarios, simulating various workloads, and adapting to new APIs or environments.  List of tools which we used for benchmarking: Phoronix Test Suite PerfKit Benchmarker Phoronix Test Suite: Phoronix Test Suite is the most comprehensive open-source benchmarking platform available for Linux, macOS, and windows systems. It is widely used for automated testing, performance analysis, and software comparisons. What is PTS Extension? A PTS Extension is a plugin or add-on for the Phoronix Test Suite (PTS) that extends its functionality. It allows users to add custom behaviors before, during, or after benchmark runs—ideal for automation, integration, or custom logging. PTS extensions are used to: Add full socket runs Add open source docker tests Integrate with other systems. System & hardware benchmarking Why Shift from PTS to PerfKit Benchmarker? Phoronix Test Suite (PTS) is primarily a single-node benchmarking tool, which runs on a single machine. To overcome this issue Perfkit Benchmarker tool is used. PKB is specifically built for cloud platforms. PKB handles provisioning, benchmarking, monitoring, and cleanup automatically. PTS requires manual test setup, especially for cloud VMs. PKB can push benchmark data to: InfluxDB Stackdriver Grafana JSON logs for CI/CD systems          PTS does offer HTML/JSON/CSV output but lacks native telemetry integrations. Perfkit Benchmarker(PKB): PerfKit Benchmarker is an open-source tool developed by Google that automates the process of benchmarking cloud infrastructure across different cloud providers.  Main Stages of a PerfKit Benchmarker Run: What Is a PerfKit Benchmarker Extension?  Extensions allow users to define: Custom benchmark Flags Providers Workloads Top Benefits of PerfKit Benchmarker Extension: PerfKit Benchmarker can run distributed benchmarks involving multiple VMs across one or more cloud zones or providers. Automatically handles VM provisioning, software installation, test execution, teardown. Easily integrates with dashboards, analytics pipelines, or cost/performance reports. Useful in capacity planning, performance regression testing, or SLI validation. In addition PKB Extension supports Turbostat (useful for analyzing power and frequency behavior during benchmarks), Lm-Senso rs(Linux utility used to monitor hardware sensors),   and Sysstat (analyze CPU, memory, disk I/O, networking, and other system-level performance metrics.). PKB extension also support additional feature for Report generation, which is useful to generate report with all result and peripheral data. It supports various formats such as TXT, CSV and HTML. Here’s a set of workload charts for PerfKit Benchmarker (PKB) organized by category. These charts summarize the common benchmark workloads available in PKB, helping you choose the right tests for CPU, memory, disk, network, and database performance analysis across cloud platforms. Cloud Comparison Using PerfKit Benchmarker Here's a comprehensive comparison of cloud providers (GCP, Azure, OCI) using PerfKit Benchmarker (PKB) as a common benchmarking framework: Conclusion PTS is excellent for deep technical benchmarking of a single system. PKB is a robust choice for cloud performance comparisons, cost evaluation, and infrastructure benchmarking at scale.

DLRM stands for Deep Learning Recommendation Model. It is a neural network architecture developed by Facebook AI (Meta) for large-scale personalized recommendation systems. DLRM is widely used in real-world applications  where personalized recommendations or ranking predictions are needed. DLRM designed for click-through rate (CTR) prediction and ranking task.  Examples: Online Advertising, E-commerce Recommendations, Social Media Feed Ranking, Streaming Services, Online Marketplace and Classifieds etc.  DLRM features: DLRM Installation Options: Install Original Facebook DLRM(PyTorch) using git and python. Install DLRM using TorchRec Install NVIDIA DLRM Install DLRM in Docker (CPU-only or GPU) What Is the Relationship Between DLRM and PyTorch? DLRM is built using PyTorch.  PyTorch serves as the foundational deep-learning framework that powers every component inside DLRM. PyTorch Is the Framework; DLRM Is the Model DLRM is not  a framework, it is a specific neural-network architecture  designed by Meta (Facebook) for large-scale recommendation systems. PyTorch provides: DLRM uses these tools to construct its dense MLPs, embedding tables, and feature-interaction layers. Pytorch Installation Options: PyTorch can be installed in several ways depending on your environment, hardware, and workflow. Install via pip (Most Common & Easiest) Install via Conda (Best for GPU Environments) Install via Docker (Isolated & Production-Friendly) Install from Source (For Developers and Custom Builds) Cloud-Based PyTorch Installation Install via Package Managers (Limited OS Support) Pytorch Installation via Docker: Installing PyTorch through Docker is one of the most reliable and hassle-free ways to set up a deep learning environment. Instead of manually managing Python versions, CUDA toolkits, cuDNN libraries, and system dependencies, Docker provides a pre-configured container where everything already works out of the box. By pulling an official PyTorch image—either CPU-only or with CUDA support—you get an isolated and reproducible environment that runs identically on any machine. Quick steps 1. Pull an image CPU-only: docker pull pytorch/pytorch:latest GPU (CUDA 11.8 example): docker pull pytorch/pytorch:latest-cuda11.8-cudnn8-runtime 2. Run the container CPU: docker run -it pytorch/pytorch:latest bash GPU (with NVIDIA container toolkit): docker run -it --gpus all pytorch/pytorch:latest-cuda11.8-cudnn8-runtime bash 3. Verify inside the container python3 -c "import torch; print(torch.__version__); print('cuda:', torch.cuda.is _available())" How to Run DLRM Inside a PyTorch Docker Container? Pull a PyTorch Docker Image Start the Container Install Dependencies (Inside the Container) Clone DLRM Repository Run DLRM DLRM Command: Running DLRM effectively requires understanding the key command-line options that control data loading, model architecture, training configuration, and performance tuning.  DLRM accepts a rich set of flags that allow you to configure everything from batch sizes to embedding dimensions. These options fall into four major categories: Data Options Training Options Model Architecture Options System / Performance Options Frequently Used DLRM Command: python dlrm_s_pytorch.py \     --data-generation=synthetic \     --mini-batch-size=2048 \     --learning-rate=0.01 \     --arch-sparse-feature-size=16 \     --arch-mlp-bot="13-512-256-64-16" \     --arch-mlp-top="512-256-1" \     --print-freq=10 Conclusion Using PyTorch Docker containers  to run DLRM (Deep Learning Recommendation Model)  provides a streamlined, consistent, and reproducible environment across different hardware platforms. Docker eliminates dependency conflicts, simplifies setup, and ensures that the exact software stack—PyTorch version, libraries, and optimizations—can be deployed seamlessly. In short, PyTorch Docker + DLRM  offers a reliable, flexible, and efficient path to train, evaluate, and deploy recommendation models with minimal friction.

Benchmarking SPECint on FPGA: Introductio n With the growing interest in AI hardware for high-performance and power-efficient computing, understanding how industry-standard benchmarks perform on such platforms is critical. In this paper, we focus on SPECrate®2017 Integer workloads, a widely-used CPU benchmark suite, and share a case study comparing various runs on an FPGA target: a base run and a tuned run that achieved better performance. This paper describes how the tuning and benchmarking procedure was executed, the challenges faced, and what we learned from this hands-on analysis. Why SPECint on FPGA? SPECrate®2017 integer evaluates the integer processing capabilities of a CPU through a set of compute-intensive, single-threaded programs. Running these on an FPGA (with soft or hardened CPU cores) helps evaluate and tune how custom logic performs in realistic software scenarios, especially in workloads like compilers, compression, and AI preprocessing. Benchmarking Setup Platform: FPGA Emulation: Run on QEMU for pre-validation, native execution on FPGA target Benchmark Suite: SPEC CPU2017 Cross-compilation: All benchmarks built using a target toolchain with specmake, applying -static, and a base set of flags Base Run: No tuning; baseline compiler flags, minimal memory tuning Optimized Run: Enhanced compiler flags, better memory layout, cache tuning Here’s how the benchmarking was carried out: Cross-Compilation of SPECrate®2017 Integer Benchmarks Ensured static linking for portability Verified ELF binaries using file and readelf Execution with runspec Invoked with runspec --config=target.cfg --tune=base --size=test,train.ref for initial testing Data Collection Captured runtime, SPEC score, and individual benchmark outputs Track CPU MHz, instruction counts using perf or counters Tricks Use math models to reduce run times of Spec Workloads Get a sense for Test, Train and Ref workloads and find a relation so there is no need to runref everytime.

1. Introduction In today’s fast-paced digital world, system performance is critical to the success of applications ranging from cloud computing platforms to high-performance computing (HPC) workloads. Performance modelling is a powerful technique used to predict, analyze, and optimize the efficiency of computing systems. By simulating and understanding system behavior, developers, engineers, and IT managers can make informed decisions about design, scaling, and optimization strategies. 2. What is Performance Modelling? Performance modelling is the process of creating abstract representations (models) of a system's behavior under various workloads and configurations. These models help predict how systems respond to changes in usage, hardware, software, or architecture. Performance models can be analytical, simulation-based, or empirical, each offering unique insights into system behavior. 3. Objectives of Performance Modelling Prediction:  Estimate system behavior before deployment. Bottleneck Identification:  Locate components that limit performance. Optimization:  Inform design choices to improve efficiency. Capacity Planning:  Guide resource allocation for current and future needs. Cost Efficiency:  Avoid over-provisioning and reduce operational expenses. 4. Key Techniques in Performance Modelling Analytical Models:  Use mathematical formulas to describe system performance. Simulation Models:  Create detailed simulations to mimic system behavior over time. This could be as simple as equations with simple assumptions or using models available online. Empirical Models:  Rely on real-world data and benchmarks to build predictive models. This is more involved since this requires in-depth knowledge of system architecture. 5. Steps in Developing a Performance Model Define Goals:  Determine what you want to achieve (e.g., optimize response time, throughput). Collect Data:  Gather metrics from logs, monitoring tools, or benchmarks. Choose Modelling Technique:  Decide between analytical, simulation, or empirical models. Build the Model:  Construct the performance model using appropriate tools or software. Validate the Model:  Compare predictions with actual performance to ensure accuracy. Analyze & Optimize:  Use the model to explore different configurations and identify optimal settings. 6. Tools for Performance Modelling Queuing Models for analyzing response times Simulators for detailed, event-based modeling Benchmarking Suites for real-world performance data Profiling Tools for low-level performance metrics 7. Applications of Performance Modelling High-Performance Computing (HPC):  Optimize cluster performance and parallel job scheduling. Cloud Computing:  Predict performance under varying loads and optimize resource allocation. Software Engineering:  Improve application architecture and identify inefficient code paths. Enterprise IT:  Plan for infrastructure upgrades and disaster recovery. 8. Challenges and Best Practices Challenges: Model accuracy vs. complexity trade-off Data collection overhead Environmental variability Best Practices: Keep models as simple as possible while maintaining accuracy Continuously validate models against real performance Use a combination of modelling techniques when necessary 9. Conclusion Performance modelling is an indispensable approach for understanding, predicting, and optimizing system efficiency. Whether you're designing a new application, upgrading infrastructure, or managing a complex cloud environment, performance models can help you make better, data-driven decisions. By embracing the right modelling techniques and tools, organizations can improve performance, reduce costs, and deliver superior user experiences.

Performance analysis can often feel like searching for a needle in a haystack. When your application isn't running as fast as you'd like, where do you even begin to look? Is it a memory bottleneck? Are you stalling in the CPU's front-end? Answering these questions is critical, but traditional tools can be complex and overwhelming. This is where Intel's PerfSpect comes in. And now, thanks to some recent contributions, this powerful tool is no longer just for x86 systems. I'm happy to share - how I've been able to natively compile PerfSpect on ARM architecture- enabling deep performance analysis on platforms like Neoverse series for processors like Ampere, AWS Graviton, Google Axion, NVIDIA Grace and Microsoft Cobalt series of Processors supporting. Why PerfSpect? A Simpler Path to Performance Insights PerfSpect is a lightweight, command-line performance analysis tool. Its primary strength lies in its use of the Top-Down Microarchitecture Analysis (TMA)  methodology. Instead of drowning you in hundreds of raw performance counters, TMA provides a structured, hierarchical way to identify the primary bottleneck in your system. It breaks down CPU cycles into a few high-level categories: Front-End Bound:  The CPU isn't getting instructions fast enough. Back-End Bound:  Instructions are available, but the execution units are stalled. This is further broken down into: Core Bound:  The computation units are the bottleneck. Memory Bound:  The CPU is waiting on data from memory or caches. Retiring:  The CPU is successfully executing instructions. This is the "good" category. Bad Speculation / Miss:  The CPU wasted work on instructions that were ultimately discarded (e.g., due to branch misprediction). By presenting performance data through this lens, PerfSpect makes it incredibly easy to pinpoint the character  of your bottleneck and tells you exactly where to focus your optimization efforts. The Competitive Landscape: How Does PerfSpect Compare? PerfSpect doesn't exist in a vacuum. The Linux ecosystem is rich with powerful profiling tools like perf, Intel VTune Profiler, AMD uProf. PerfSpect's unique value is its combination of simplicity, structured TMA methodology, and now, cross-architecture support.  It provides actionable insights without the steep learning curve of raw perf or the complexity of a full-blown GUI profiler. My Contribution: Native Support for ARM My primary contribution was to port PerfSpect, enabling it to build and run natively on ARMv8/ARMv9 architectures. This involved mapping the ARM Performance Monitoring Unit (PMU) events to the TMA categories, allowing the same intuitive reporting to work seamlessly on platforms from Ampere, Amazon, and Microsoft. Now, developers can use a single, familiar tool to analyze workloads across different server fleets. Get Started: Build and Run PerfSpect on ARM Ready to try it on your ARM machine? Here’s how you can get it up and running. Prerequisites Ensure you have Python, pip, and the standard Linux performance tools installed. # For Debian/Ubuntu-based systems $ sudo apt-get update $ sudo apt-get install -y python3 python3-pip linux-tools-common linux-tools-generic Step 1: Clone the Repository $ git clone -b Neoverse-native-support https://github.com/Whileone-Techsoft/PerfSpect.git $ cd PerfSpect Step 2: Build Tools Docker for aarch64 $ ./builder/build.sh Step 3: Build Perfspect natively on aarch64 $ make -j Sample TMA image for Graviton4

Complex computing environments, spanning cloud, HPC, AI, and edge workloads; observability is no longer optional. With multiple layers of hardware and software working together, traditional monitoring alone cannot surface the insights needed for optimizing performance or preventing downtime. At Whileone Techsoft Pvt. Ltd. , we help companies go beyond monitoring by building deep observability frameworks that connect performance benchmarking , system analytics , telemetry , and profiling . This integrated approach helps engineering teams gain complete visibility into their systems, enabling faster debugging, reduced operational costs, and enhanced end-user experiences. Why Observability Matters As infrastructures scale and workloads diversify, blind spots emerge. This can lead to: Observability addresses these challenges by providing end-to-end visibility into the state and behavior of your systems. This means you can detect issues earlier, understand their root causes, and fix them before they impact users. Observability vs. Traditional Monitoring Traditional monitoring answers the “what”, for example, CPU utilisation or error counts. Observability goes deeper and answers the “why” behind performance issues. It focuses on three core pillars: Metrics  – Quantifiable measurements (e.g., latency, throughput) Logs  – Detailed event records for context Traces  – Understanding requests as they travel across distributed systems At Whileone Techsoft, we layer these pillars with analytics, telemetry, and profiling  to deliver actionable insights. Performance Benchmarking: The Foundation Our Performance Benchmarking Services form the cornerstone of observability. We help companies: This data-driven approach uncovers bottlenecks early before they become costly in production. System Analytics for Deeper Understanding Benchmarking generates performance data, but analytics transforms that data into insights. System analytics  helps teams understand: How workloads utilize CPU, memory, I/O, and network resources Correlation between resource consumption and performance outcomes Trends and anomalies in system behavior over time Our analytics frameworks leverage advanced models to identify optimization opportunities, ensuring your workloads perform consistently and reliably. Telemetry for Real-Time Visibility Telemetry extends observability by collecting live data from hardware, firmware, middleware, and applications. It captures fine-grained performance metrics continuously Enables proactive alerts for deviations from benchmarks Allows visualization of live system health through unified dashboards Whileone’s use of open standards like OpenTelemetry  makes this telemetry layer scalable and interoperable with your existing tools. Profiling for Root Cause Analysis Even the best benchmarking and telemetry setups cannot replace profiling when you need detailed root cause analysis. System-level profiling : Identifies hotspots in the kernel, drivers, or hardware interfaces Code-level profiling : Finds inefficient functions, loops, or algorithms in the application stack By correlating profiling data with benchmark and telemetry insights, we help engineering teams quickly diagnose and resolve performance regressions. An Integrated Observability Framework At Whileone Techsoft, we integrate benchmarking, analytics, telemetry, and profiling into a single observability framework: Unified dashboards to correlate data across layers Automated workflows for continuous testing and monitoring Cross-silo visibility that spans hardware, system software, and applications This holistic approach ensures reliable, high-performance outcomes for pre-silicon validation, cloud workload optimization, and edge deployments. Benefits of Observability-Driven Benchmarking Real-World Example One of our semiconductor customers was struggling with inconsistent performance in their post-silicon validation phase. By deploying Whileone’s observability-driven benchmarking framework , they were able to: Pinpoint compiler-level inefficiencies using code profiling: https://www.whileone.in/post/tuning-compiler-flags-for-custom-hardware Correlate memory bandwidth metrics from telemetry data with workload performance: https://www.whileone.in/post/investigating-performance-discrepancy-in-hpl-test-on-arm64-machines Best Practices for Building Observability Observability is no longer a “nice-to-have”, it’s essential for ensuring reliable, high-performance systems. Whileone Techsoft Pvt. Ltd.  brings together performance benchmarking, system analytics, telemetry, and profiling  to build observability frameworks tailored for semiconductor companies, cloud providers, and software enterprises. Ready to take your performance engineering efforts to the next level? Reach out to us  to learn how our observability-driven services can help you reduce costs, accelerate time-to-market, and achieve industry-leading performance.

1. Introduction High-Performance Computing (HPC) is at the core of solving complex computational problems in scientific research, engineering, and large-scale data analysis. Benchmarking plays a critical role in evaluating and optimizing HPC system performance. The Standard Performance Evaluation Corporation (SPEC) provides widely recognized benchmarking suites tailored for different computing environments, helping researchers, businesses, and hardware vendors assess system capabilities. 2. What is SPEC HPC Benchmarking? SPEC HPC benchmarks are designed to measure the performance of high-performance computing systems under real-world workloads. Unlike general performance testing, SPEC HPC benchmarks focus on evaluating scalability, efficiency, and computational power across various hardware and software configurations. Key metrics include execution time, scalability efficiency, and energy consumption. 3. Why SPEC HPC Benchmarks Matter? Evaluating Scalability & Efficiency:  SPEC benchmarks measure how well HPC systems scale with increasing workloads. Benchmarking Real-World Applications:  Unlike synthetic benchmarks, SPEC HPC benchmarks reflect real-world HPC workloads used in scientific and industrial applications. Standardization & Comparability:  They enable fair performance comparisons between different architectures, compilers, and system configurations. 4. Key SPEC HPC Benchmark Suites SPEC MPI:  Measures parallel computing performance using MPI-based workloads. SPEC OMP:  Evaluates OpenMP-based applications for multi-threaded workloads. SPEC ACCEL:  Assesses performance on GPUs and other accelerators. SPEC CPU:  Focuses on single-thread and multi-thread performance in computational workloads. 5. How SPEC HPC Benchmarks Work Benchmark execution process: Benchmarks are executed under controlled conditions to ensure reproducibility. Setting up the testing environment: Includes configuring system parameters, compilers, and libraries. Running SPEC benchmarks on various HPC hardware: Executing the benchmark suite on HPC hardware to collect performance data. Collecting and analyzing results CPU, GPU, and memory performance Compiler optimizations and software configurations Networking and storage bottlenecks Factors that impact benchmarking results: 6. Understanding Benchmark Results Interpreting SPEC Scores:  Higher scores indicate better performance. Comparing Results:  Performance ratios help compare different architectures and software configurations. Case Studies:  SPEC benchmarks are widely used in industries like climate modeling, genomics, and engineering simulations to evaluate and improve HPC systems. 7. Best Practices for Running SPEC HPC Benchmarks Preparing an Optimized Benchmarking Environment:  Ensure system settings and compiler options align with best practices. Choosing the Right SPEC Benchmark:  Select the benchmark that aligns with the intended workload. Avoiding Common Mistakes:  Properly setting up software and avoiding misinterpretations of results ensures accurate assessments. 8. Future Trends in SPEC HPC Benchmarking AI, ML, and Cloud Computing:  Emerging workloads in artificial intelligence and machine learning are shaping future benchmarks. Heterogeneous Computing:  SPEC is evolving to benchmark performance across GPUs, FPGAs, and new architectures like RISC-V. Upcoming Developments:  Continuous updates in benchmarking methodologies are expected to keep pace with next-generation HPC innovations. 9. Conclusion SPEC HPC benchmarks provide a standardized way to evaluate and compare HPC system performance. Businesses, researchers, and hardware vendors can leverage these benchmarks to optimize their computing infrastructure. For further exploration, SPEC’s official website and research publications offer in-depth insights into benchmarking methodologies.

Goal of this project: This project aims to determine RISC-V's readiness for running YOLOX for the latest edge requirements. Target Application: Running YOLOX on RISC-V QEMU involves setting up a RISC-V virtual machine and then configuring the necessary environment to compile and run YOLOX. Please note that this is a complex process, and it's essential to have prior experience with virtualization and RISC-V development. From the RISCV website, this is a blog ( https://riscv.org/blog/2023/07/yolox-for-object-detection/ ) which describes the steps to build and run YOLOX for a development board. These steps did not work as is when running on QEMU. This blog assumes the readers of this blog are comfortable with a Linux-based host system (this guide is based on Ubuntu 22.04). Step 1: Install QEMU and Set Up a RISC-V Virtual Machine First, you need to install QEMU and the RISC-V toolchain. You can do this by running: ​sudo apt-get install qemu-system-riscv In this step, you'll create a RISC-V virtual machine using QEMU. You'll need a RISC-V disk image for this. You can find pre-built RISC-V images for various Linux distributions online. You can also build your own RISC-V image if you prefer. wget https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img.xz tar xf ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img.xz #Rename the qemu_image mv ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img riscv-ubuntu2204.img qemu-img resize ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img +16G Launch the Qemu VM as follows: qemu-system-riscv64 -nographic -machine virt -m 16G -append "root=/dev/vda rw" -drive file=riscv-ubuntu2204.img,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -device virtio-net-device,netdev=net0 -netdev user,id=net0 This will boot the RISC-V VM with 16GB of RAM. Step 2: Configure the Python Environment Once the VM is up and running, log in, and set up your RISC-V development environment. You may need to install the necessary dependencies, which may vary depending on the distribution and the version. Most of the software packages that Python program software depends on can be installed by pip. You can run the following command to install pip. apt install python3-pip Before installing other Python packages, install the venv package that can be used to create a Python virtual environment. apt install python3.11-venv Create a Python virtual environment and activate it. cd /root python3 -m venv yolox source /root/yolox/bin/activate Step 3: Install necessary whl packages The Python ecology of the RISC-V architecture is still lacking. We have created build packages to be able to install directly on python3.11.  Step 4: Build and Run YOLOX Next, clone the YOLOX repository into your RISC-V qemu git clone https://github.com/Megvii-BaseDetection/YOLOX Navigate to the YOLOX directory and build the YOLOX code. This step may involve installing additional dependencies and configuring the build for RISC-V architecture. cd YOLOX make With YOLOX successfully built, you can now run it on your RISC-V system. You'll need to adapt the YOLOX commands to work with your specific use case and input data. ​Standard models https://github.com/Megvii-BaseDetection/YOLOX#standard-models In this example, yolox_s is downloaded. wget wttps://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.pth -P /home/ubuntu/ python3 tools/demo.py image -n yolox-s -c /home/ubuntu/yolox_s.pth --path assets/demo.png --conf 0.25 --nms 0.45 --tsize 640 --save_result --device cpu #Output Logs 2023-09-15 17:05:49.803 | INFO | __main__:main:269 - Model Summary: Params: 8.97M, Gflops: 26.93 2023-09-15 17:05:49.860 | INFO | __main__:main:282 - loading checkpoint 2023-09-15 17:05:53.884 | INFO | __main__:main:286 - loaded checkpoint done. 2023-09-15 17:06:24.598 | INFO | __main__:inference:165 - Infer time: 30.0775s 2023-09-15 17:06:24.708 | INFO | __main__:image_demo:202 - Saving detection result in ./YOLOX_outputs/yolox_s/vis_res/2023_09_15_17_05_53/demo.png We would like to hear from you if this blog was useful to you. Please contact us at info@whileone.in. We would be happy to understand and discuss your requirements and showcase our expertise in a variety of cloud and edge technologies.