Framework for Analysis & Comparison of Tests
Benchmarking, Visualisation and Analysis Tool
-
Real-Time System Performance Tracking
-
Run Comparison Dashboards
-
Detailed Metric Drill-downs
-
Auto-Diff Reports Between Runs
-
Integrated Visual Report Generation
-
Multi-Workload and Multi-Metric Support
-
Anomaly Detection Capabilities
-
User-Friendly Filtering and Navigation
-
Run History with Timestamp and Tagging
-
Exportable PDF/CSV Reports
-
Customizable Report Pages​​
Feature Walkthroughs
​
​1. Reports Page: Run History & Overview
The Reports page serves as the central hub for accessing all previously executed runs across different workloads. It displays a consolidated, scrollable list of test executions along with metadata such as run name, machine name, workload type, timestamps, and execution tags.
​Key Capabilities:​
​
-
Workload Grouping: All runs are grouped under the workload (e.g., SpecJbb2015-Multi) for easy traceability.
-
Chronological Sorting: Latest runs appear at the top, making it easy to access recent test results.
-
Run Metadata Display: Tags such as version info, machine ID, runtime parameters, and OS details are displayed.
-
Interactive Navigation: Clicking on a run ID instantly navigates the user to a detailed view with graphs and logs.
This page is essential for QA teams, developers, and performance engineers who need an at-a-glance view of test history, execution details, and trends across time.
​​
​2. Individual Run Drill-Down
​
This functionality provides a comprehensive breakdown of metrics collected during a specific run. Once a run is selected from the reports page, users are taken to a dedicated drill-down interface.
​
​​​​​Key Capabilities:
​
-
Graphical Timeline Charts: Metrics such as CPU usage, memory consumption, garbage collection frequency, throughput, and latency are plotted over time.
-
Interactive Hover Details: Hovering over any graph reveals timestamp-specific values, useful for pinpointing spikes or dips.
-
Metric Tabs: All metrics are organized into categorized tabs like “CPU Metrics,” “Memory Metrics,” “Application Metrics,” and “GC Metrics.”
-
Color Highlights: Anomalous values or thresholds breached are shown using red/orange hues for visual prioritization.
This detailed page helps engineers isolate issues to exact time windows, correlate system behavior to workload intensity, and diagnose bottlenecks during test execution.
3. Diff View Between Two Runs
​
The diff functionality allows users to select two runs and compare their metric values side-by-side. This is especially valuable for validating software changes, hardware upgrades, or performance regression checks.
​
Key Capabilities:​
​
-
Metric-by-Metric Comparison: Each monitored metric (e.g., max CPU, avg throughput) is compared and highlighted with percentage difference.
-
Color-Coded Deltas:
-
Green: Improvement in metric (e.g., lower response time).
-
Red: Regression (e.g., higher CPU usage).
-
Grey: Insignificant/no change.
-
-
Interactive Drill-Down: Users can click on each metric to explore graphs and data.
-
Summary Header: A top summary shows the net improvement/regression across the run.
​
Ideal for release engineering and benchmark validation, this view enables quick decision-making regarding the performance impact of changes.
​
4. Intelligent Metric View
​
The intelligent metric view showcases important metrics with derived intelligence, such as visual indicators for spikes, drop-offs, or stable periods.​​
​
​​​Key Capabilities:
​
-
Metric Prioritization: Important metrics are ranked and shown on top (e.g., CPU Saturation, GC Pause).
-
Dynamic Coloring: Smart coloring is used to indicate metric health based on predefined thresholds.
-
Annotations Support: Users or system auto-tags events of interest like “GC Storm” or “Memory Saturation.”
​​
This view is perfect for both beginners and expert users who want to quickly get a sense of “what’s going wrong or right” without inspecting every graph.
5. Metric Overlay for Comparative Insight
​
This feature allows multiple metrics to be overlaid on a single graph to identify correlations.
​
Key Capabilities:
​
-
Time-Synchronized Graphing: All overlaid metrics share the same time axis.
-
Cross-Correlation: Example: CPU spike can be compared against memory load or thread count.
-
Interactive Filtering: Users can toggle visibility of metrics to declutter the graph.
This helps in tracing root causes—whether high CPU was caused by a memory leak, high disk IO, or thread contention.
6. Tabular Data Export and View
​
Besides graphs, all data points are stored and viewable in a tabular format. Each row corresponds to a timestamp; each column is a metric.
​
Key Capabilities:
​
-
High Precision: Data includes real decimal values (not rounded).
-
Exportability: CSV/Excel export is available directly from the UI.
-
Sorting and Filtering: Built-in options to sort by time or metric value.
This view is valuable for teams who want to analyze data in external tools like Excel, Tableau, or Python.
​​​
​
7. Raw Log Snapshot and Timeline Sync
​
Logs collected during a test run are aligned with the metric timeline to allow root cause analysis.
​
Key Capabilities:
​
-
Timestamp Synchronization: Clicking on a spike in the metric chart auto-highlights corresponding log entries.
-
Search Functionality: Logs can be filtered using search terms like “ERROR” or “OOM.”
-
Severity Highlighting: Logs are colored based on severity – info, warning, error, fatal.
Use case: An engineer sees a CPU spike at 14:31 and checks if there was an OutOfMemory error or GC pause logged at that time.
​
​​
8. Metric Configuration & Threshold Tuning
​
Admins or advanced users can configure which metrics to monitor and set threshold levels for alerting or flagging.
​
Key Capabilities:
​
-
Custom Thresholds: Set your own upper/lower limits for any metric.
-
Alert Integration: Hook alerts into email, Slack, or incident tools.
-
Workload-Specific Profiles: Different workloads may have different thresholds (e.g., CPU load for ML vs. DB workload).
This allows automated validation or red-flagging for performance failures.
​​​
9. Backend Tagging and Metadata Panel
​
Each run is tagged with metadata pulled from the system, git commit, test parameters, and user inputs.
​
Key Capabilities:
​
-
Traceability: Know exactly what version/config ran when.
-
Tag Search: Filter runs by tags like “release-1.1” or “baseline”.
-
Auto-Generated Tags: Git commit hashes, kernel version, CPU model, and more are pulled automatically.
Crucial for compliance, reproducibility, and post-mortem analysis.
