Troubleshooting
Find answers to common issues encountered during setup, test execution, and CI/CD pipeline integration.
Common API Errors
Our API gateway intercepts unauthorized or invalid requests before they run.
Why this happens: The request lacked the REGRESSIONBOT_API_KEY environment variable or passed an invalid string.
Solution: Retrieve your API key from the Dashboard and set the standard environment variable REGRESSIONBOT_API_KEY before running the CLI or SDK.
Why this happens: You queried status for a Job ID that does not exist or has expired. Visual regression jobs are archived after 30 days.
Solution: Double-check the Job ID UUID string returned from your start request. Verify you are using the correct target org space.
Why this happens: You triggered a job for an existing project, but the parameters in your request body (such as testOrigin, devices, paths, etc.) did not match the configuration stored in the project's record.
Solution: Either omit these parameters from your request to use the saved configuration automatically (a parameter-less run), or run via POST /project/{name}/run which always loads the saved config regardless of what you pass. To intentionally change the config, use the MCP update_project tool — note that changing config will invalidate existing baselines.
Handling Failed Jobs
If a visual regression job transits into the FAILED state, this indicates that the worker could not capture one or more pages. Common reasons include:
- DNS Failures: The host could not be resolved from inside the cloud network (e.g. localhost, local company dev URL).
- SSL Errors: Self-signed certificates on staging machines. Use a publicly trusted certificate for environments you want the cloud workers to capture.
- Authentication Walls: Basic auth, OAuth redirect screens, or captcha verification pages preventing headless browser crawls.
Timeouts on Large Pages
Headless visual workers allocate a standard 30-second budget for screenshot capture and lazy-load scroll execution.
Tips to fix timeouts:
- Exclude large media or infinite-scrolling pages (e.g., product lists, social feeds) from your sitemaps.
- Provide dynamic element masking using
data-vr-maskto bypass dynamic elements. - Limit comparison depths when invoking sitemap crawler endpoints.
SSIM & Visual Match Metrics
To eliminate false positives caused by subpixel antialiasing or rendering shifts on differing OS/browsers, RegressionBot computes two separate difference indices:
Diff Percentage (% Offset)
The ratio of exact mismatched pixels divided by total viewport pixels. For example, a minor font weight change across a whole document can mark 15% of pixels as red differences, even though the layout remains structurally correct.
Visual Match Score (SSIM)
Structural Similarity Index (SSIM) measures human-perceived luminance, contrast, and structure. If SSIM stays above 98.5%, the visual variation may represent structural noise rather than a meaningful product regression.
Pro Tip on Noise Filtering:
Fetch the job summary after completion and inspect visualMatchScore alongside diffPercentage to identify likely rendering noise:
await job.waitForCompletion();
const summary = await job.getSummary();
const likelyNoise = summary.regressions.filter((regression) =>
(regression.visualMatchScore ?? 0) >= 99 &&
(regression.diffPercentage ?? 0) < 0.5
);First Run (Empty Baseline)
When running tests on a brand new project, there are no prior baselines stored in cloud storage. RegressionBot will treat this as a successful execution but mark each page result status as isNewBaseline: true. Subsequent test runs will automatically compare against this newly established baseline.
Short Screenshot URLs
Troubleshoot common issues when using shortened, clickable screenshot redirect URLs.
Why this happens: The short screenshot URL token has expired. Redirection tokens are only valid for one hour.
Solution: Request the job summary again using the API. This generates fresh short URLs and new tokens in the database.
Why this happens: You requested the job summary before the job finished processing (i.e. the status is still PROCESSING).
Solution: Poll the job status endpoint (/job/{jobId}) and wait for the status to become COMPLETED, APPROVED, or FAILED before fetching the summary.
Why this happens: The backend fell back to standard pre-signed URLs, likely due to a database write error when saving short-lived tokens.
Solution: Check the RegressionBot system status or contact support. The client received the response successfully, so your network connection is fine.