mirror of
https://github.com/NawfalMotii79/PLFM_RADAR.git
synced 2026-06-08 22:47:16 +00:00
4989c33aa6
Wire a per-frame MCU→FPGA "beam pattern ready" handshake so the chirp scheduler can stall between 48-chirp frames until the MCU finishes writing the next ADAR1000 pattern. The legacy unused stm32_new_chirp input on PD8 is repurposed as stm32_beam_ready; chirp_scheduler.v gets a new S_BEAM_WAIT state entered after each frame_pulse and an 80 ms watchdog so a missed MCU toggle degrades to wall-clock cadence with a sticky telemetry bit rather than stalling the radar. Cold-reset defaults the handshake off (host_handshake_enable=0, new opcode 0x1A); the GUI opts in once the MCU PD8 wiring is verified on the bench. Both the FT601 and FT2232H status word 4 paths get the new beam_handshake_watchdog_fired sticky at bit [1] (reclaimed from the range_mode retirement in commit 1). RTL: - chirp_scheduler.v: 2-FF ASYNC_REG sync on beam_ready_async; 1-cycle edge detect (any transition, MCU side uses HAL_GPIO_TogglePin); new S_BEAM_WAIT state entered at frame_pulse when host_handshake_enable=1; 23-bit beam_watchdog counter with BEAM_WATCHDOG_MAX = 8_000_000 (~80 ms at 100 MHz, ~10 nominal frames); beam_handshake_watchdog_fired output sticky across mixers_enable cycles, cleared only by reset_n; mid-wait disable releases the FSM so dropping the opcode never strands the radar between frames. - radar_receiver_final.v: thread stm32_beam_ready_async + host_handshake_enable + beam_handshake_watchdog_fired through the scheduler instance. - radar_system_top.v: rename input port stm32_new_chirp → stm32_beam_ready; add host_handshake_enable register (cold-reset = 1'b0); opcode 0x1A dispatch (value[0]); add rx_beam_handshake_watchdog wire; pack into status_words[4][1] in both USB paths. - radar_system_top_50t.v: rename wrapper port + sub-instance wiring. - usb_data_interface.v + usb_data_interface_ft2232h.v: add status_beam_handshake_watchdog input + 2-FF level CDC (same convention as F-6.4 / F-1.2 stickies); refresh word-4 layout doc comment; pack beam_handshake_wd_sync_1 into status_words[4][1]. XDC: - xc7a50t_ftg256.xdc + xc7a200t_fbg484.xdc: rename stm32_new_chirp port references to stm32_beam_ready (same PD8 pin, F13 on 50T / L18 on 200T). MCU: - main.h: add FPGA_BEAM_READY_Pin = GPIO_PIN_8 + FPGA_BEAM_READY_GPIO_Port = GPIOD alongside the existing FPGA_FRAME_PULSE alias. - main.cpp:runRadarPulseSequence: insert HAL_GPIO_TogglePin(GPIOD, GPIO_PIN_8) after each setCustomBeamPattern16(RX) — once after the per-azimuth broadside (vector_0), once after matrix1, once after matrix2 — between the SPI burst completion and waitForFramePulse. GUI: - radar_protocol.py: Opcode.HANDSHAKE_ENABLE = 0x1A; StatusResponse.beam_handshake_watchdog = 0 default; parse word 4 bit [1] in parse_status_packet; update word-4 layout comment. - test_GUI_V65_Tk.py: add beam_handshake_watchdog kwarg to _make_status_packet (sets bit [1] of word 4); refresh test_parse_status_word4_layout_co_spec to cover the new bit (used+9=32); add test_parse_status_beam_handshake_watchdog round-trip; test_handshake_enable_opcode pins 0x1A; defaults / chirps_mismatch / agc-coexist tests gain a watchdog==0 assertion; bump test_all_rtl_opcodes_present expected set to include 0x17/0x18/0x19/0x1A. TB: - new tb_chirp_scheduler_handshake.v (16 checks): legacy open-loop, edge exit (rising + falling), 200-cycle idle hold, watchdog auto-advance via force on dut.beam_watchdog, sticky-survives-mixers_disable, mid-wait disable release, reset_n clears sticky. - run_regression.sh: register the new TB in PHASE 1. - tb_radar_receiver_final.v: tie the 3 new receiver ports off (beam_ready_async=0, handshake_enable=0, watchdog unconnected). - tb_system_mechanics.v / tb_system_opcodes.v: explicit .stm32_beam_ready(1'b0) connection (the cold-reset host_handshake_enable=0 keeps the FSM out of S_BEAM_WAIT). - tb_usb_data_interface.v / tb_usb_protocol_v2.v / tb_e2e_dsp_to_host.v / tb_ft2232h_frame_drop.v: tie .status_beam_handshake_watchdog(1'b0). Ride-along ruff sweep (14 → 0 across the repo): - tb/cosim/compare_independent.py: RUF003 — '5×' → 'at least 5x'. - tb/cosim/gen_e2e_expected.py: noqa: E402 on the post-sys.path import; drop unused EXPECTED_RANGE_BIN + EXPECTED_DOPPLER_BIN_PER_SF imports; fold the detect-class slot if/else into a ternary (SIM108). - tb/cosim/gen_e2e_stimulus.py: drop int() wrapping round() at four call sites (RUF046 — round() already returns int in Python 3); rewrite the range-bin derivation comment block from code-like `# range_bin = ...` to prose (ERA001); strip stray f from placeholder-free error string (F541). - tb/cosim/tb_e2e_dsp_to_host_parse.py: open(path, 'r') → open(path) (UP015). - v7/dashboard.py: '3×' → '3x' (RUF003); drop quotes from 'StatusResponse | None' annotation (UP037, file already has `from __future__ import annotations`). CI summary (all suites green pre-commit): - ruff: All checks passed! - FPGA regression (iverilog): 43 / 0 / 0 (incl. new handshake TB 16/16). - MCU tests: 51 / 0 + 34 / 0 + 13 / 13 ADAR1000_AGC. - GUI Tk (test_GUI_V65_Tk): 120 / 0. - GUI v7 (test_v7): 152 / 0. Production rollout note: bitstream cold-resets with host_handshake_enable=0 so existing flashes keep their open-loop cadence until the GUI sends opcode 0x1A=1. Once enabled, the per-pattern dwell tracks both the chirp ladder (PD14 frame_pulse from commit-3 work) and the MCU pattern-write completion (PD8 toggle from this commit), eliminating drift from the SPI burst timing.
854 lines
31 KiB
Bash
Executable File
854 lines
31 KiB
Bash
Executable File
#!/bin/bash
|
|
# ===========================================================================
|
|
# FPGA Regression Test Runner for AERIS-10 Radar
|
|
# Phase 0: Vivado-style lint (catches issues iverilog silently accepts)
|
|
# Phase 1+: Compile and run all verified iverilog testbenches
|
|
#
|
|
# Usage: ./run_regression.sh [--quick] [--skip-lint]
|
|
# --quick Skip long-running integration tests (receiver golden, system TB)
|
|
# --skip-lint Skip Phase 0 lint checks (not recommended)
|
|
#
|
|
# Exit code: 0 if all tests pass, 1 if any fail
|
|
# ===========================================================================
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
cd "$SCRIPT_DIR"
|
|
|
|
QUICK=0
|
|
SKIP_LINT=0
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--quick) QUICK=1 ;;
|
|
--skip-lint) SKIP_LINT=1 ;;
|
|
esac
|
|
done
|
|
|
|
PASS=0
|
|
FAIL=0
|
|
SKIP=0
|
|
LINT_WARN=0
|
|
LINT_ERR=0
|
|
ERRORS=""
|
|
|
|
# Colors (if terminal supports it)
|
|
RED='\033[0;31m'
|
|
GREEN='\033[0;32m'
|
|
YELLOW='\033[0;33m'
|
|
CYAN='\033[0;36m'
|
|
NC='\033[0m' # No Color
|
|
|
|
# ===========================================================================
|
|
# PHASE 0: VIVADO-STYLE LINT
|
|
# Two layers:
|
|
# (A) iverilog -Wall full-design compile — parse for serious warnings
|
|
# (B) Custom regex checks for patterns Vivado treats as errors
|
|
# ===========================================================================
|
|
|
|
# Production RTL file list (same as system TB minus testbench files)
|
|
# Uses ADC stub for IBUFDS/BUFIO primitives that iverilog can't parse
|
|
PROD_RTL=(
|
|
radar_system_top.v
|
|
radar_transmitter.v
|
|
dac_interface_single.v
|
|
plfm_chirp_controller_v2.v
|
|
radar_receiver_final.v
|
|
tb/ad9484_interface_400m_stub.v
|
|
ddc_400m.v
|
|
nco_400m_enhanced.v
|
|
cic_decimator_4x_enhanced.v
|
|
cdc_modules.v
|
|
cdc_async_fifo.v
|
|
fir_lowpass.v
|
|
ddc_input_interface.v
|
|
chirp_reference_rom.v
|
|
matched_filter_multi_segment.v
|
|
matched_filter_processing_chain.v
|
|
range_bin_decimator.v
|
|
doppler_processor.v
|
|
xfft_16.v
|
|
fft_engine.v
|
|
xfft_2048.v
|
|
fft_engine_axi_bridge.v
|
|
frequency_matched_filter.v
|
|
usb_data_interface.v
|
|
usb_data_interface_ft2232h.v
|
|
edge_detector.v
|
|
chirp_scheduler.v
|
|
rx_gain_control.v
|
|
cfar_ca.v
|
|
mti_canceller.v
|
|
fpga_self_test.v
|
|
)
|
|
|
|
# Source-only RTL (not instantiated at top level, but should still be lint-clean)
|
|
# Note: ad9484_interface_400m.v is excluded — it uses Xilinx primitives
|
|
# (IBUFDS, BUFIO, BUFG, IDDR) that iverilog cannot compile. The production
|
|
# design uses tb/ad9484_interface_400m_stub.v for simulation instead.
|
|
EXTRA_RTL=(
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Shared RTL file lists for integration / system tests
|
|
# Centralised here so a new module only needs adding once.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# Receiver chain (used by golden generate/compare tests)
|
|
RECEIVER_RTL=(
|
|
radar_receiver_final.v
|
|
chirp_scheduler.v
|
|
tb/ad9484_interface_400m_stub.v
|
|
ddc_400m.v nco_400m_enhanced.v cic_decimator_4x_enhanced.v
|
|
cdc_modules.v cdc_async_fifo.v fir_lowpass.v ddc_input_interface.v
|
|
chirp_reference_rom.v
|
|
matched_filter_multi_segment.v matched_filter_processing_chain.v
|
|
range_bin_decimator.v doppler_processor.v xfft_16.v fft_engine.v
|
|
xfft_2048.v fft_engine_axi_bridge.v
|
|
frequency_matched_filter.v
|
|
rx_gain_control.v mti_canceller.v
|
|
)
|
|
|
|
# Full system top (receiver chain + TX + USB + detection + self-test)
|
|
SYSTEM_RTL=(
|
|
radar_system_top.v
|
|
radar_transmitter.v dac_interface_single.v plfm_chirp_controller_v2.v
|
|
"${RECEIVER_RTL[@]}"
|
|
usb_data_interface.v usb_data_interface_ft2232h.v edge_detector.v
|
|
cfar_ca.v fpga_self_test.v
|
|
)
|
|
|
|
# ---- Layer A: iverilog -Wall compilation ----
|
|
run_lint_iverilog() {
|
|
local label="$1"
|
|
shift
|
|
local files=("$@")
|
|
local warn_file="/tmp/iverilog_lint_$$_${label}.log"
|
|
|
|
printf " %-45s " "iverilog -Wall ($label)"
|
|
|
|
if ! iverilog -g2001 -DSIMULATION -Wall -o /dev/null "${files[@]}" 2>"$warn_file"; then
|
|
# Hard compile error — always fatal
|
|
echo -e "${RED}COMPILE ERROR${NC}"
|
|
while IFS= read -r line; do
|
|
echo " $line"
|
|
done < "$warn_file"
|
|
LINT_ERR=$((LINT_ERR + 1))
|
|
rm -f "$warn_file"
|
|
return 1
|
|
fi
|
|
|
|
# Parse warnings — classify as error-level or info-level
|
|
local err_count=0
|
|
local info_count=0
|
|
local err_lines=""
|
|
|
|
while IFS= read -r line; do
|
|
# Part-select out of range — Vivado Synth 8-524 (ERROR in Vivado)
|
|
if echo "$line" | grep -q 'Part select.*is selecting after the vector\|out of bound bits'; then
|
|
err_count=$((err_count + 1))
|
|
err_lines="$err_lines\n ${RED}[VIVADO-ERR]${NC} $line"
|
|
# Port width mismatch / connection mismatch
|
|
elif echo "$line" | grep -q 'port.*does not match\|Port.*mismatch'; then
|
|
err_count=$((err_count + 1))
|
|
err_lines="$err_lines\n ${RED}[VIVADO-ERR]${NC} $line"
|
|
# Informational warnings (timescale, dangling ports, array sensitivity)
|
|
elif echo "$line" | grep -q 'timescale\|dangling\|sensitive to all'; then
|
|
info_count=$((info_count + 1))
|
|
# Unknown warning — report but don't fail
|
|
elif [[ -n "$line" ]]; then
|
|
info_count=$((info_count + 1))
|
|
fi
|
|
done < "$warn_file"
|
|
|
|
if [[ "$err_count" -gt 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} ($err_count Vivado-class errors, $info_count info)"
|
|
echo -e "$err_lines"
|
|
LINT_ERR=$((LINT_ERR + err_count))
|
|
else
|
|
echo -e "${GREEN}PASS${NC} ($info_count info warnings)"
|
|
fi
|
|
|
|
rm -f "$warn_file"
|
|
}
|
|
|
|
# ---- Layer B: Custom regex static checks ----
|
|
# Catches patterns that Vivado treats as errors/warnings but iverilog ignores
|
|
run_lint_static() {
|
|
printf " %-45s " "Static RTL checks"
|
|
|
|
local err_count=0
|
|
local warn_count=0
|
|
local err_lines=""
|
|
local warn_lines=""
|
|
|
|
for f in "$@"; do
|
|
[[ -f "$f" ]] || continue
|
|
# Skip testbench files (tb/ directory) — only lint production RTL
|
|
case "$f" in tb/*) continue ;; esac
|
|
|
|
local linenum=0
|
|
while IFS= read -r line; do
|
|
linenum=$((linenum + 1))
|
|
|
|
# --- CHECK 1: Part-select with literal range on reg ---
|
|
# Pattern: identifier[N:M] where N exceeds declared width
|
|
# (iverilog catches this, but belt-and-suspenders)
|
|
|
|
# --- CHECK 2: case/casex/casez without default (non-full case) ---
|
|
# Vivado SYNTH-6 / inferred latch warning
|
|
# Heuristic: look for case/casex/casez, then check if 'default' appears
|
|
# before the matching 'endcase'. This is approximate — full parsing
|
|
# would need a real parser. We flag 'case' lines so the developer
|
|
# can manually verify.
|
|
# (Handled below as a multi-line check)
|
|
|
|
# --- CHECK 3: Blocking assignment (=) inside always @(posedge ...) ---
|
|
# Vivado SYNTH-5 warning for inferred latches / race conditions
|
|
# Only flag if the always block is clocked (posedge/negedge)
|
|
# This is a heuristic — we check for '= ' that isn't '<=', '==', '!='
|
|
# inside an always block header containing 'posedge' or 'negedge'.
|
|
# (Too complex for line-by-line — skip for now, handled by testbenches)
|
|
|
|
# --- CHECK 4: Multi-driven register (assign + always on same signal) ---
|
|
# (Would need cross-file analysis — skip for v1)
|
|
|
|
done < "$f"
|
|
done
|
|
|
|
# --- Multi-line check: case without default ---
|
|
for f in "$@"; do
|
|
[[ -f "$f" ]] || continue
|
|
case "$f" in tb/*) continue ;; esac
|
|
|
|
# Find case blocks and check for default
|
|
# Use awk to find case..endcase blocks missing 'default'
|
|
local missing_defaults
|
|
missing_defaults=$(awk '
|
|
/^[[:space:]]*(case|casex|casez)[[:space:]]*\(/ {
|
|
case_line = NR
|
|
case_file = FILENAME
|
|
has_default = 0
|
|
in_case = 1
|
|
next
|
|
}
|
|
in_case && /default[[:space:]]*:/ {
|
|
has_default = 1
|
|
}
|
|
in_case && /endcase/ {
|
|
if (!has_default) {
|
|
printf "%s:%d: case statement without default\n", FILENAME, case_line
|
|
}
|
|
in_case = 0
|
|
}
|
|
' "$f" 2>/dev/null)
|
|
|
|
if [[ -n "$missing_defaults" ]]; then
|
|
while IFS= read -r hit; do
|
|
warn_count=$((warn_count + 1))
|
|
warn_lines="$warn_lines\n ${YELLOW}[SYNTH-6]${NC} $hit"
|
|
done <<< "$missing_defaults"
|
|
fi
|
|
done
|
|
|
|
# CHECK 5 ($readmemh in synth code) and CHECK 6 (unused includes)
|
|
# require multi-line ifdef tracking / cross-file analysis. Not feasible
|
|
# with line-by-line regex. Omitted — use Vivado lint instead.
|
|
|
|
if [[ "$err_count" -gt 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} ($err_count errors, $warn_count warnings)"
|
|
echo -e "$err_lines"
|
|
LINT_ERR=$((LINT_ERR + err_count))
|
|
elif [[ "$warn_count" -gt 0 ]]; then
|
|
echo -e "${YELLOW}WARN${NC} ($warn_count warnings)"
|
|
echo -e "$warn_lines"
|
|
LINT_WARN=$((LINT_WARN + warn_count))
|
|
else
|
|
echo -e "${GREEN}PASS${NC}"
|
|
fi
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helper: compile, run, and compare a matched-filter co-sim scenario
|
|
# run_mf_cosim <scenario_name> <define_flag>
|
|
# ---------------------------------------------------------------------------
|
|
run_mf_cosim() {
|
|
local name="$1"
|
|
local define="$2"
|
|
local vvp="tb/tb_mf_cosim_${name}.vvp"
|
|
local scenario_lower="$name"
|
|
|
|
printf " %-45s " "MF Co-Sim ($name)"
|
|
|
|
# Compile — build command as string to handle optional define
|
|
local cmd="iverilog -g2001 -DSIMULATION"
|
|
if [[ -n "$define" ]]; then
|
|
cmd="$cmd $define"
|
|
fi
|
|
cmd="$cmd -o $vvp tb/tb_mf_cosim.v matched_filter_processing_chain.v fft_engine.v xfft_2048.v fft_engine_axi_bridge.v frequency_matched_filter.v chirp_reference_rom.v"
|
|
|
|
if ! eval "$cmd" 2>/tmp/iverilog_err_$$; then
|
|
echo -e "${RED}COMPILE FAIL${NC}"
|
|
ERRORS="$ERRORS\n MF Co-Sim ($name): compile error ($(head -1 /tmp/iverilog_err_$$))"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
|
|
# Run TB
|
|
local output
|
|
output=$(timeout 120 vvp "$vvp" 2>&1) || true
|
|
rm -f "$vvp"
|
|
|
|
# Check TB internal pass/fail (allow leading whitespace; see run_test note)
|
|
local tb_fail
|
|
tb_fail=$(echo "$output" | grep -Ec '^[[:space:]]*\[FAIL' || true)
|
|
if [[ "$tb_fail" -gt 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} (TB internal failure)"
|
|
ERRORS="$ERRORS\n MF Co-Sim ($name): TB internal failure"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
|
|
# Run Python compare
|
|
if command -v python3 >/dev/null 2>&1; then
|
|
local compare_out
|
|
local compare_rc=0
|
|
compare_out=$(python3 tb/cosim/compare_mf.py "$scenario_lower" 2>&1) || compare_rc=$?
|
|
if [[ "$compare_rc" -ne 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} (compare_mf.py mismatch)"
|
|
ERRORS="$ERRORS\n MF Co-Sim ($name): Python compare failed"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
else
|
|
echo -e "${YELLOW}SKIP${NC} (RTL passed, python3 not found — compare skipped)"
|
|
SKIP=$((SKIP + 1))
|
|
return
|
|
fi
|
|
|
|
echo -e "${GREEN}PASS${NC} (RTL + Python compare)"
|
|
PASS=$((PASS + 1))
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helper: compile, run, and compare a Doppler co-sim scenario
|
|
# run_doppler_cosim <scenario_name> <define_flag>
|
|
# ---------------------------------------------------------------------------
|
|
run_doppler_cosim() {
|
|
local name="$1"
|
|
local define="$2"
|
|
local vvp="tb/tb_doppler_cosim_${name}.vvp"
|
|
|
|
printf " %-45s " "Doppler Co-Sim ($name)"
|
|
|
|
# Compile — build command as string to handle optional define
|
|
local cmd="iverilog -g2001 -DSIMULATION"
|
|
if [[ -n "$define" ]]; then
|
|
cmd="$cmd $define"
|
|
fi
|
|
cmd="$cmd -o $vvp tb/tb_doppler_cosim.v doppler_processor.v xfft_16.v fft_engine.v"
|
|
|
|
if ! eval "$cmd" 2>/tmp/iverilog_err_$$; then
|
|
echo -e "${RED}COMPILE FAIL${NC}"
|
|
ERRORS="$ERRORS\n Doppler Co-Sim ($name): compile error ($(head -1 /tmp/iverilog_err_$$))"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
|
|
# Run TB
|
|
local output
|
|
output=$(timeout 120 vvp "$vvp" 2>&1) || true
|
|
rm -f "$vvp"
|
|
|
|
# Check TB internal pass/fail (allow leading whitespace; see run_test note)
|
|
local tb_fail
|
|
tb_fail=$(echo "$output" | grep -Ec '^[[:space:]]*\[FAIL' || true)
|
|
if [[ "$tb_fail" -gt 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} (TB internal failure)"
|
|
ERRORS="$ERRORS\n Doppler Co-Sim ($name): TB internal failure"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
|
|
# Run Python compare
|
|
if command -v python3 >/dev/null 2>&1; then
|
|
local compare_out
|
|
local compare_rc=0
|
|
compare_out=$(python3 tb/cosim/compare_doppler.py "$name" 2>&1) || compare_rc=$?
|
|
if [[ "$compare_rc" -ne 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} (compare_doppler.py mismatch)"
|
|
ERRORS="$ERRORS\n Doppler Co-Sim ($name): Python compare failed"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
else
|
|
echo -e "${YELLOW}SKIP${NC} (RTL passed, python3 not found — compare skipped)"
|
|
SKIP=$((SKIP + 1))
|
|
return
|
|
fi
|
|
|
|
echo -e "${GREEN}PASS${NC} (RTL + Python compare)"
|
|
PASS=$((PASS + 1))
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helper: compile and run a single testbench
|
|
# run_test <name> <vvp_path> <iverilog_args...>
|
|
# ---------------------------------------------------------------------------
|
|
run_test() {
|
|
# Optional: --timeout=N as first arg overrides default 120s
|
|
local timeout_secs=120
|
|
if [[ "$1" == --timeout=* ]]; then
|
|
timeout_secs="${1#--timeout=}"
|
|
shift
|
|
fi
|
|
|
|
local name="$1"
|
|
local vvp="$2"
|
|
shift 2
|
|
local args=("$@")
|
|
|
|
printf " %-45s " "$name"
|
|
|
|
# Compile
|
|
if ! iverilog -g2001 -DSIMULATION -o "$vvp" "${args[@]}" 2>/tmp/iverilog_err_$$; then
|
|
echo -e "${RED}COMPILE FAIL${NC}"
|
|
ERRORS="$ERRORS\n $name: compile error ($(head -1 /tmp/iverilog_err_$$))"
|
|
FAIL=$((FAIL + 1))
|
|
return
|
|
fi
|
|
|
|
# Run
|
|
local output
|
|
output=$(timeout "$timeout_secs" vvp "$vvp" 2>&1) || true
|
|
|
|
# Count PASS/FAIL in output (testbenches use explicit [PASS]/[FAIL] markers)
|
|
# Match `[PASS]` and `[PASS <digits>]` (and same for FAIL). Excludes
|
|
# informational tags like `[FAIL-INFO]` (used for known unrelated bugs,
|
|
# e.g. RX-NEW-1 fft_engine bin-shift in tb_matched_filter_processing_chain.v)
|
|
# which would otherwise false-fire as real failures.
|
|
# Allow leading whitespace — many TBs emit " [PASS]" with indentation
|
|
# (the historical anchor `^[PASS]` silently missed those, hiding 22+
|
|
# failures across tb_system_e2e / tb_fft_engine / tb_fullchain_realdata).
|
|
local test_pass test_fail
|
|
test_pass=$(echo "$output" | grep -Ec '^[[:space:]]*\[PASS( [0-9]+)?\]' || true)
|
|
test_fail=$(echo "$output" | grep -Ec '^[[:space:]]*\[FAIL( [0-9]+)?\]' || true)
|
|
|
|
if [[ "$test_fail" -gt 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} (pass=$test_pass, fail=$test_fail)"
|
|
ERRORS="$ERRORS\n $name: $test_fail failure(s)"
|
|
FAIL=$((FAIL + 1))
|
|
elif [[ "$test_pass" -gt 0 ]]; then
|
|
echo -e "${GREEN}PASS${NC} ($test_pass checks)"
|
|
PASS=$((PASS + 1))
|
|
else
|
|
# No PASS/FAIL markers — check for clean completion
|
|
if echo "$output" | grep -qi 'finish\|complete\|done'; then
|
|
echo -e "${GREEN}PASS${NC} (completed)"
|
|
PASS=$((PASS + 1))
|
|
else
|
|
echo -e "${YELLOW}UNKNOWN${NC} (no PASS/FAIL markers)"
|
|
ERRORS="$ERRORS\n $name: no pass/fail markers in output"
|
|
FAIL=$((FAIL + 1))
|
|
fi
|
|
fi
|
|
|
|
rm -f "$vvp"
|
|
}
|
|
|
|
# ===========================================================================
|
|
echo "============================================"
|
|
echo " AERIS-10 FPGA Regression Test Suite"
|
|
echo "============================================"
|
|
echo ""
|
|
echo "Date: $(date)"
|
|
echo "iverilog: $(iverilog -V 2>&1 | head -1)"
|
|
echo ""
|
|
|
|
# ===========================================================================
|
|
# PHASE 0: LINT (Vivado-class error detection)
|
|
# ===========================================================================
|
|
if [[ "$SKIP_LINT" -eq 0 ]]; then
|
|
echo "--- PHASE 0: LINT (Vivado-class checks) ---"
|
|
|
|
# Layer A: iverilog -Wall on full production design
|
|
run_lint_iverilog "production" "${PROD_RTL[@]}"
|
|
|
|
# Layer A: standalone modules not in top-level hierarchy
|
|
# (use ${EXTRA_RTL[@]+...} guard so empty array doesn't trip set -u)
|
|
if [[ ${#EXTRA_RTL[@]} -gt 0 ]]; then
|
|
for extra in "${EXTRA_RTL[@]}"; do
|
|
if [[ -f "$extra" ]]; then
|
|
run_lint_iverilog "$(basename "$extra" .v)" "$extra"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
# Layer B: custom static regex checks
|
|
if [[ ${#EXTRA_RTL[@]} -gt 0 ]]; then
|
|
ALL_RTL=("${PROD_RTL[@]}" "${EXTRA_RTL[@]}")
|
|
else
|
|
ALL_RTL=("${PROD_RTL[@]}")
|
|
fi
|
|
run_lint_static "${ALL_RTL[@]}"
|
|
|
|
echo ""
|
|
if [[ "$LINT_ERR" -gt 0 ]]; then
|
|
echo -e "${RED} LINT FAILED: $LINT_ERR Vivado-class error(s) detected.${NC}"
|
|
echo " Fix lint errors before pushing to Vivado. Aborting regression."
|
|
echo ""
|
|
exit 1
|
|
elif [[ "$LINT_WARN" -gt 0 ]]; then
|
|
echo -e "${YELLOW} LINT: $LINT_WARN advisory warning(s) (non-blocking)${NC}"
|
|
else
|
|
echo -e "${GREEN} LINT: All checks passed${NC}"
|
|
fi
|
|
echo ""
|
|
else
|
|
echo "--- PHASE 0: LINT (skipped via --skip-lint) ---"
|
|
echo ""
|
|
fi
|
|
|
|
# ===========================================================================
|
|
# PHASE 1: UNIT TESTS — Changed Modules (HIGH PRIORITY)
|
|
# ===========================================================================
|
|
echo "--- PHASE 1: Changed Modules ---"
|
|
|
|
run_test "CIC Decimator" \
|
|
tb/tb_cic_reg.vvp \
|
|
tb/tb_cic_decimator.v cic_decimator_4x_enhanced.v
|
|
|
|
run_test "Chirp Controller (BRAM)" \
|
|
tb/tb_chirp_reg.vvp \
|
|
tb/tb_chirp_controller.v plfm_chirp_controller_v2.v
|
|
|
|
run_test "Chirp Contract" \
|
|
tb/tb_chirp_ctr_reg.vvp \
|
|
tb/tb_chirp_contract.v plfm_chirp_controller_v2.v
|
|
|
|
run_doppler_cosim "stationary" ""
|
|
run_doppler_cosim "moving" "-DSCENARIO_MOVING"
|
|
run_doppler_cosim "two_targets" "-DSCENARIO_TWO"
|
|
|
|
run_test "Threshold Detector (detection bugs)" \
|
|
tb/tb_threshold_detector.vvp \
|
|
tb/tb_threshold_detector.v
|
|
|
|
run_test "RX Gain Control (digital gain)" \
|
|
tb/tb_rx_gain_control.vvp \
|
|
tb/tb_rx_gain_control.v rx_gain_control.v
|
|
|
|
run_test "MTI Canceller (ground clutter)" \
|
|
tb/tb_mti_canceller.vvp \
|
|
tb/tb_mti_canceller.v mti_canceller.v
|
|
|
|
run_test "CFAR CA Detector" \
|
|
tb/tb_cfar_ca.vvp \
|
|
tb/tb_cfar_ca.v cfar_ca.v
|
|
|
|
run_test "FPGA Self-Test" \
|
|
tb/tb_fpga_self_test.vvp \
|
|
tb/tb_fpga_self_test.v fpga_self_test.v
|
|
|
|
run_test "FFT AXI Bridge tready handshake (AUDIT-C10)" \
|
|
tb/tb_fft_engine_axi_bridge.vvp \
|
|
tb/tb_fft_engine_axi_bridge.v fft_engine_axi_bridge.v
|
|
|
|
run_test "FT2232H Frame Drop Counter (AUDIT-C12)" \
|
|
tb/tb_ft2232h_frame_drop.vvp \
|
|
tb/tb_ft2232h_frame_drop.v usb_data_interface_ft2232h.v
|
|
|
|
run_test "Doppler Frame-Start Gate (AUDIT-S3)" \
|
|
tb/tb_doppler_frame_start_gate.vvp \
|
|
tb/tb_doppler_frame_start_gate.v doppler_processor.v xfft_16.v fft_engine.v
|
|
|
|
run_test "ADC PWDN opcode 0x32 (AUDIT-S25)" \
|
|
tb/tb_adc_pwdn_opcode.vvp \
|
|
tb/tb_adc_pwdn_opcode.v
|
|
|
|
run_test "Status-word stickies CDC + DIG7 fault-OR (AUDIT-S10 + PR-AB.b Step 1)" \
|
|
tb/tb_status_words_stickies.vvp \
|
|
tb/tb_status_words_stickies.v
|
|
|
|
run_test "DIG6 frame-pulse stretcher (PR-AB.b)" \
|
|
tb/tb_dig6_frame_pulse.vvp \
|
|
tb/tb_dig6_frame_pulse.v
|
|
|
|
run_test "Chirp Scheduler beam-ready handshake (PR-AB.b expanded c5)" \
|
|
tb/tb_chirp_scheduler_handshake.vvp \
|
|
tb/tb_chirp_scheduler_handshake.v chirp_scheduler.v
|
|
|
|
run_test "NUM_CELLS sizing 50T (AUDIT-C16)" \
|
|
tb/tb_audit_c16_num_cells_50t.vvp \
|
|
tb/tb_audit_c16_num_cells.v
|
|
|
|
run_test --timeout=120 "NUM_CELLS sizing 200T (AUDIT-C16)" \
|
|
tb/tb_audit_c16_num_cells_200t.vvp \
|
|
-DSUPPORT_LONG_RANGE \
|
|
tb/tb_audit_c16_num_cells.v
|
|
|
|
echo ""
|
|
|
|
# ===========================================================================
|
|
# PHASE 2: INTEGRATION TESTS
|
|
# ===========================================================================
|
|
echo "--- PHASE 2: Integration Tests ---"
|
|
|
|
run_test "DDC Chain (NCO→CIC→FIR)" \
|
|
tb/tb_ddc_reg.vvp \
|
|
tb/tb_ddc_cosim.v ddc_400m.v nco_400m_enhanced.v \
|
|
cic_decimator_4x_enhanced.v fir_lowpass.v cdc_modules.v cdc_async_fifo.v
|
|
|
|
# Real-data co-simulation: committed golden hex vs RTL (exact match required).
|
|
# These catch architecture mismatches (e.g. 32-pt → dual 16-pt Doppler FFT)
|
|
# that self-blessing golden-generate/compare tests cannot detect.
|
|
run_test --timeout=300 "Doppler Real-Data (synthetic, exact match)" \
|
|
tb/tb_doppler_realdata.vvp \
|
|
tb/tb_doppler_realdata.v doppler_processor.v xfft_16.v fft_engine.v
|
|
|
|
run_test --timeout=600 "Full-Chain Real-Data (decim→Doppler, exact match)" \
|
|
tb/tb_fullchain_realdata.vvp \
|
|
tb/tb_fullchain_realdata.v range_bin_decimator.v \
|
|
doppler_processor.v xfft_16.v fft_engine.v
|
|
|
|
if [[ "$QUICK" -eq 0 ]]; then
|
|
# Receiver integration (structural + bounds + pulse assertions).
|
|
# Replaces the earlier "Receiver golden generate/compare" pair, which was
|
|
# self-blessing (both passes ran identical RTL on identical stimulus, so
|
|
# it passed regardless of bugs). Real co-sim coverage is now provided by
|
|
# tb_doppler_realdata.v and tb_fullchain_realdata.v (Python goldens,
|
|
# exact match); this integration test exercises the full RX pipeline
|
|
# (ADC stub → DDC → MF → Decim → Doppler) and verifies that
|
|
# doppler_frame_done is a single-cycle pulse at module boundaries.
|
|
run_test --timeout=600 "Receiver Integration (tb_radar_receiver_final)" \
|
|
tb/tb_rx_final_reg.vvp \
|
|
tb/tb_radar_receiver_final.v "${RECEIVER_RTL[@]}"
|
|
|
|
# A6 end-to-end DSP -> host test (PR-Z). Replaces the two zero-assertion
|
|
# `radar_system_tb` smoke runs (USB_MODE=0 + USB_MODE=1) that this PR
|
|
# supersedes. Three stages:
|
|
# 1. gen_e2e_stimulus.py - deterministic single-target stimulus
|
|
# 2. gen_e2e_expected.py - bit-exact Python golden (fpga_model)
|
|
# 3. tb_e2e_dsp_to_host.v - production-faithful chain
|
|
# (range_decim -> mti -> doppler -> dc_notch
|
|
# -> cfar -> sync -> usb_data_interface_ft2232h)
|
|
# 4. tb_e2e_dsp_to_host_parse.py - radar_protocol round-trip + section asserts
|
|
printf " %-45s " "E2E DSP-to-Host (PR-Z A6)"
|
|
set +e
|
|
a6_log=/tmp/a6_e2e_$$.log
|
|
{
|
|
python3 tb/cosim/gen_e2e_stimulus.py && \
|
|
python3 tb/cosim/gen_e2e_expected.py && \
|
|
iverilog -g2001 -DSIMULATION -o tb/tb_e2e_dsp_to_host.vvp \
|
|
tb/tb_e2e_dsp_to_host.v mti_canceller.v doppler_processor.v \
|
|
xfft_16.v fft_engine.v cfar_ca.v usb_data_interface_ft2232h.v \
|
|
edge_detector.v && \
|
|
timeout 300 vvp tb/tb_e2e_dsp_to_host.vvp && \
|
|
python3 tb/cosim/tb_e2e_dsp_to_host_parse.py
|
|
} > "$a6_log" 2>&1
|
|
a6_rc=$?
|
|
set -e
|
|
rm -f tb/tb_e2e_dsp_to_host.vvp
|
|
a6_tb_pass=$(grep -Ec '^[[:space:]]*\[PASS( [0-9]+)?\]' "$a6_log" || true)
|
|
a6_tb_fail=$(grep -Ec '^[[:space:]]*\[FAIL( [0-9]+)?\]' "$a6_log" || true)
|
|
a6_parse_overall_pass=$(grep -Ec '^\[OVERALL PASS\]' "$a6_log" || true)
|
|
if [[ "$a6_rc" -eq 0 && "$a6_tb_fail" -eq 0 && "$a6_parse_overall_pass" -ge 1 ]]; then
|
|
echo -e "${GREEN}PASS${NC} (TB pass=$a6_tb_pass + parse OVERALL PASS)"
|
|
PASS=$((PASS + 1))
|
|
else
|
|
echo -e "${RED}FAIL${NC} (rc=$a6_rc, TB pass=$a6_tb_pass fail=$a6_tb_fail, parse=$a6_parse_overall_pass)"
|
|
ERRORS="$ERRORS\n E2E DSP-to-Host: rc=$a6_rc"
|
|
echo " ---- A6 last 30 lines of log ----"
|
|
tail -30 "$a6_log" | sed 's/^/ /'
|
|
FAIL=$((FAIL + 1))
|
|
fi
|
|
rm -f "$a6_log"
|
|
|
|
# PR-I subsuites (replace tb_system_e2e). Each TB instantiates
|
|
# radar_system_top with USB_MODE=1 (production FT2232H path) and
|
|
# carves a focused slice of what the legacy tb_system_e2e tried to
|
|
# cover all at once:
|
|
# tb_system_opcodes - opcode dispatch via FT2232H send_cmd (fast)
|
|
# tb_system_mechanics - reset/RF/safety/CDC mechanics (fast)
|
|
# Note: tb_system_dataflow was retired in PR-Z — its 3 liveness-only
|
|
# asserts (chirp_frames>0, range_valid>0, range_valid>=100) are now
|
|
# dominated by A6's stronger in-TB checks (egress-byte exact, doppler
|
|
# bit-exact vs Python golden, cfar class). ~7 min wall reclaimed.
|
|
run_test "System Opcodes (tb_system_opcodes)" \
|
|
tb/tb_system_opcodes_reg.vvp \
|
|
tb/tb_system_opcodes.v "${SYSTEM_RTL[@]}"
|
|
|
|
run_test "System Mechanics (tb_system_mechanics)" \
|
|
tb/tb_system_mechanics_reg.vvp \
|
|
tb/tb_system_mechanics.v "${SYSTEM_RTL[@]}"
|
|
else
|
|
echo " (skipped receiver integration + e2e dsp-to-host + opcodes/mechanics — use without --quick)"
|
|
SKIP=$((SKIP + 4))
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# ===========================================================================
|
|
# PHASE 2b: MATCHED FILTER CO-SIMULATION (RTL vs Python golden reference)
|
|
# Runs tb_mf_cosim.v for 4 scenarios, then compare_mf.py validates output
|
|
# against committed Python golden CSV files. In SIMULATION mode, thresholds
|
|
# are generous (behavioral vs fixed-point twiddles differ) — validates
|
|
# state machine mechanics, output count, and energy sanity.
|
|
# ===========================================================================
|
|
echo "--- PHASE 2b: Matched Filter Co-Sim ---"
|
|
|
|
run_mf_cosim "chirp" ""
|
|
run_mf_cosim "dc" "-DSCENARIO_DC"
|
|
run_mf_cosim "impulse" "-DSCENARIO_IMPULSE"
|
|
run_mf_cosim "tone5" "-DSCENARIO_TONE5"
|
|
|
|
echo ""
|
|
|
|
# ===========================================================================
|
|
# PHASE 3: UNIT TESTS — Signal Processing
|
|
# ===========================================================================
|
|
echo "--- PHASE 3: Signal Processing ---"
|
|
|
|
run_test "FFT Engine" \
|
|
tb/tb_fft_reg.vvp \
|
|
tb/tb_fft_engine.v fft_engine.v
|
|
|
|
run_test "NCO 400MHz" \
|
|
tb/tb_nco_reg.vvp \
|
|
tb/tb_nco_400m.v nco_400m_enhanced.v
|
|
|
|
run_test "FIR Lowpass" \
|
|
tb/tb_fir_reg.vvp \
|
|
tb/tb_fir_lowpass.v fir_lowpass.v
|
|
|
|
run_test --timeout=600 "Matched Filter Chain" \
|
|
tb/tb_mf_reg.vvp \
|
|
tb/tb_matched_filter_processing_chain.v matched_filter_processing_chain.v \
|
|
fft_engine.v xfft_2048.v fft_engine_axi_bridge.v \
|
|
chirp_reference_rom.v frequency_matched_filter.v
|
|
|
|
# RX-B regression coverage: chain pipeline depth + full-chain
|
|
# autocorrelation peak position. Both run the production fft_engine
|
|
# (no SIMULATION-only behavioural FFT exists). Long-running because
|
|
# the production FFT is BRAM-pipelined (~153k cycles per chain pass).
|
|
run_test --timeout=120 "RX-B Chain Pipeline Latency (tb_rxb_latency_measure)" \
|
|
tb/tb_rxb_lat_reg.vvp \
|
|
tb/tb_rxb_latency_measure.v matched_filter_processing_chain.v \
|
|
fft_engine.v xfft_2048.v fft_engine_axi_bridge.v frequency_matched_filter.v
|
|
|
|
run_test --timeout=600 "RX-B Full-Chain Autocorrelation (tb_rxb_fullchain_latency)" \
|
|
tb/tb_rxb_fc_reg.vvp \
|
|
tb/tb_rxb_fullchain_latency.v matched_filter_multi_segment.v \
|
|
matched_filter_processing_chain.v fft_engine.v xfft_2048.v \
|
|
fft_engine_axi_bridge.v frequency_matched_filter.v \
|
|
chirp_reference_rom.v
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# T-6 independent reference drift cosim (PR-M).
|
|
# Bytewise spot-checks of NCO_SINE_LUT, fft_twiddle_{16,2048}.mem, and
|
|
# DOPPLER_WINDOW_COEFF against analytical Q15 values, plus end-to-end peak
|
|
# and roundtrip invariants for NCO / FFT / MF / Doppler. Catches the bug
|
|
# class where a transcription error exists identically in both fpga_model.py
|
|
# (RTL-mirroring twin) and the RTL — which the bit-exact cosim cannot detect
|
|
# because both sides of that comparison are computing the same wrong values.
|
|
# Pure Python (numpy + scipy), ~1 s wall, no iverilog compile.
|
|
#
|
|
# Required deps: numpy, scipy (declared in pyproject.toml dev group).
|
|
# Install with: uv sync --group dev (from repo root)
|
|
# CI handles this in the fpga-regression job; locally activate the
|
|
# resulting .venv (or use `uv run bash run_regression.sh`).
|
|
# If a dep is missing the script emits a [SKIP] marker and exits with
|
|
# code 2; the regression treats it as SKIP rather than FAIL so the
|
|
# missing-dep state is visible without breaking the gate.
|
|
# ---------------------------------------------------------------------------
|
|
printf " %-46s" "Independent Reference Drift (T-6)"
|
|
set +e
|
|
drift_output=$(python3 tb/cosim/compare_independent.py 2>&1)
|
|
drift_rc=$?
|
|
set -e
|
|
drift_pass=$(echo "$drift_output" | grep -Ec '^[[:space:]]*\[PASS\]' || true)
|
|
drift_fail=$(echo "$drift_output" | grep -Ec '^[[:space:]]*\[FAIL\]' || true)
|
|
if [[ "$drift_rc" -eq 2 ]]; then
|
|
# Script signalled missing-dep skip. Show its message body so the
|
|
# operator knows which package to install.
|
|
echo -e "${YELLOW}SKIP${NC} (missing python dep — see below)"
|
|
echo "$drift_output" | sed 's/^/ /'
|
|
SKIP=$((SKIP + 1))
|
|
elif [[ "$drift_fail" -gt 0 ]]; then
|
|
echo -e "${RED}FAIL${NC} (pass=$drift_pass, fail=$drift_fail)"
|
|
ERRORS="$ERRORS\n Independent Reference Drift: $drift_fail failure(s)"
|
|
echo "$drift_output" | sed 's/^/ /'
|
|
FAIL=$((FAIL + 1))
|
|
elif [[ "$drift_pass" -gt 0 && "$drift_rc" -eq 0 ]]; then
|
|
echo -e "${GREEN}PASS${NC} ($drift_pass checks)"
|
|
PASS=$((PASS + 1))
|
|
else
|
|
echo -e "${RED}FAIL${NC} (rc=$drift_rc, no clean PASS/FAIL markers)"
|
|
ERRORS="$ERRORS\n Independent Reference Drift: rc=$drift_rc"
|
|
echo "$drift_output" | sed 's/^/ /'
|
|
FAIL=$((FAIL + 1))
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# ===========================================================================
|
|
# PHASE 4: UNIT TESTS — Infrastructure
|
|
# ===========================================================================
|
|
echo "--- PHASE 4: Infrastructure ---"
|
|
|
|
run_test "CDC Modules (3 variants)" \
|
|
tb/tb_cdc_reg.vvp \
|
|
tb/tb_cdc_modules.v cdc_modules.v
|
|
|
|
run_test "CDC Async FIFO (AUDIT-C11)" \
|
|
tb/tb_cdc_async_fifo_reg.vvp \
|
|
tb/tb_cdc_async_fifo.v cdc_async_fifo.v
|
|
|
|
run_test "Edge Detector" \
|
|
tb/tb_edge_reg.vvp \
|
|
tb/tb_edge_detector.v edge_detector.v
|
|
|
|
run_test "USB Data Interface" \
|
|
tb/tb_usb_reg.vvp \
|
|
tb/tb_usb_data_interface.v usb_data_interface.v
|
|
|
|
run_test "Range Bin Decimator" \
|
|
tb/tb_rbd_reg.vvp \
|
|
tb/tb_range_bin_decimator.v range_bin_decimator.v
|
|
|
|
echo ""
|
|
|
|
# ===========================================================================
|
|
# SUMMARY
|
|
# ===========================================================================
|
|
TOTAL=$((PASS + FAIL + SKIP))
|
|
echo "============================================"
|
|
echo " RESULTS"
|
|
echo "============================================"
|
|
if [[ "$SKIP_LINT" -eq 0 ]]; then
|
|
if [[ "$LINT_ERR" -gt 0 ]]; then
|
|
echo -e " Lint: ${RED}$LINT_ERR error(s)${NC}, $LINT_WARN warning(s)"
|
|
elif [[ "$LINT_WARN" -gt 0 ]]; then
|
|
echo -e " Lint: ${GREEN}0 errors${NC}, ${YELLOW}$LINT_WARN warning(s)${NC}"
|
|
else
|
|
echo -e " Lint: ${GREEN}clean${NC}"
|
|
fi
|
|
fi
|
|
echo " Tests: $PASS passed, $FAIL failed, $SKIP skipped / $TOTAL total"
|
|
echo "============================================"
|
|
|
|
if [[ -n "$ERRORS" ]]; then
|
|
echo ""
|
|
echo "Failures:"
|
|
echo -e "$ERRORS"
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# Exit with error if any failures
|
|
if [[ "$FAIL" -gt 0 ]]; then
|
|
exit 1
|
|
fi
|
|
|
|
exit 0
|