Live watchdog

Once mode="live" runs real money, boring failure modes matter more than interesting ones. A feed that stops ticking. A broker rejecting every order because credentials expired overnight. An unexpected intraday drawdown. An account slipping below the PDT threshold. Without a watchdog, orders fly into a blind market until someone notices.

horizon.ops.LiveWatchdog is a pure state-machine safety layer that halts submissions when any of those conditions trip. Pass it to hz.run(mode="live", watchdog=...) and it runs alongside the rest of the tier. Complements the core Kill switch.

Quickest start

from horizon.ops import LiveWatchdogConfig

result = hz.run(
    mode="live",
    feed=my_feed,
    strategies=[MyStrategy],
    accounts=registry,
    audit_log=audit_log,
    watchdog=LiveWatchdogConfig(
        feed_stale_seconds=30,
        max_consecutive_rejects=5,
        max_intraday_loss_pct=0.03,
        min_equity_usd=25_000,
        auto_flatten_on_stop=True,
        grace_period_s=10,
    ),
    max_duration_s=3600 * 6,
)

A tripped condition emits WatchdogHalt and blocks new submissions for the rest of the run. Existing in-flight orders and fills continue draining so the ledger converges before exit.

Policies

Opt-in. Set a threshold to 0 or 0.0 to disable.

feed_stale_seconds

Halt if no tick has arrived for more than N seconds. Default 30s is aggressive. Tune by asset class.

max_consecutive_rejects

Halt after N back-to-back broker rejections. Common causes: expired credentials, wrong account id, broker circuit breaker, exchange halt. Default 5.

max_intraday_loss_pct

Halt if equity drops more than N% below the session peak. Computed on every tick. Session peak resets at on_start; the first observation becomes the initial peak.

min_equity_usd

Absolute floor. Typical values: $25,000 (FINRA PDT) for margin accounts, or the IPS max_drawdown_pct times initial funding. Halts when equity dips below.

grace_period_s

Skip all checks for the first N seconds after on_start. Lets the feed warm up. Default 5s.

auto_flatten_on_stop

When the run loop exits (stop event, max duration, max ticks, watchdog halt, feed disconnect), iterate open ledger positions and submit market flatten orders. Best-effort; failures do not raise. Emits AutoFlatten with the order count. Default False.

halt_severity

Severity for WatchdogHalt: "info", "warning", "critical" (default). Drives alerting subscribers.

session_calendar and halt_outside_session

When a calendar is attached, the watchdog halts submissions outside the session.

from horizon.calendars import NYSECalendar, AlwaysOpenCalendar, ResolutionCalendar

# US equities / ETFs / options
watchdog=LiveWatchdogConfig(
    session_calendar=NYSECalendar(),
    halt_outside_session=True,
)

# Crypto
watchdog=LiveWatchdogConfig(
    session_calendar=AlwaysOpenCalendar(),
    halt_outside_session=True,    # no-op for crypto
)

# Prediction markets
rcal = ResolutionCalendar()
rcal.register("nyc_mayor_2026", datetime(2026, 11, 4, tzinfo=UTC))
watchdog=LiveWatchdogConfig(
    session_calendar=rcal,
    halt_outside_session=True,
)

Without a calendar, the flag is a no-op. With a calendar and the flag off, the calendar only feeds flatten_before_close_seconds.

flatten_before_close_seconds

Submit market flatten orders for every open position when the session approaches close. One-shot per session. Fires when calendar.next_close(now) - now drops inside the threshold; stays silent until the next session.

watchdog=LiveWatchdogConfig(
    session_calendar=NYSECalendar(),
    flatten_before_close_seconds=120,   # 2 minutes before close
    halt_outside_session=True,
)

End-of-day-flat pattern: flatten 2 minutes before close so orders have time to fill, then halt once the session is over. _auto_flatten_positions fires and emits AutoFlatten.

Half-days are respected. NYSECalendar.next_close() returns 13:00 ET on the shortened session, so the threshold measures against the early close.

Lifecycle and audit

Events chain-link into the audit log. A later auditor ties the halt reason to the preceding state (last tick, last fill, last risk decision). See Audit trail.

Threading model

Pure Python state machine. No threads, no locks, no I/O. The run loop calls:

• on_start(now) at startup.
• observe_equity(equity) on every _process_tick.
• on_reject() / on_success() after submit attempts.
• check(now, last_tick_ts, operator_stop) at the top of every poll cycle.

Deterministic (same inputs, same verdict), which makes post-mortems straightforward.

Recommended configs

Paper / research

LiveWatchdogConfig(
    feed_stale_seconds=300,      # 5 min
    max_consecutive_rejects=50,
    max_intraday_loss_pct=0.20,
    grace_period_s=30,
    auto_flatten_on_stop=False,  # leave positions for inspection
)

Live prod

LiveWatchdogConfig(
    feed_stale_seconds=10,
    max_consecutive_rejects=3,
    max_intraday_loss_pct=0.03,
    min_equity_usd=25_000,
    grace_period_s=5,
    auto_flatten_on_stop=True,
    halt_severity="critical",
)

Session-scoped, calendar-driven

For an open-and-close-flat strategy:

from horizon.calendars import NYSECalendar

LiveWatchdogConfig(
    feed_stale_seconds=15,
    max_consecutive_rejects=5,
    session_calendar=NYSECalendar(),
    halt_outside_session=True,
    flatten_before_close_seconds=120,
    auto_flatten_on_stop=True,
)

• Starts flattening 2 minutes before close.
• Halts new submissions once the market closes (half-days and holidays respected via pandas_market_calendars).
• Resumes normally when the next session opens if the loop is still running.

No max_duration_s needed. The calendar is the source of truth.

Graceful operator stop

import signal
import threading

stop_event = threading.Event()

def _on_sigint(signum, frame):
    stop_event.set()

signal.signal(signal.SIGINT, _on_sigint)

hz.run(
    mode="live",
    feed=my_feed,
    watchdog=LiveWatchdogConfig(
        grace_period_s=0,
        auto_flatten_on_stop=True,
    ),
    stop_event=stop_event,
    ...,
)
# Ctrl-C -> stop_event -> watchdog OperatorStop halt -> loop exits ->
# auto_flatten fires -> run returns normally with a clean audit trail.

The watchdog records the operator-stop reason so a later audit shows the exit was intentional.

Not in scope

• Auto-recovery. Once halted, the run stays halted. Resume means a new run.
• Broker-state inspection. The watchdog does not call venue.balance(). Deep broker checks live in the venue.
• Flatten-then-halt ordering. On mid-session trip, the loop halts submissions but does not auto-flatten until exit. For flat-on-halt, wire an observer on WatchdogHalt that sets stop_event.
• Per-market thresholds. One threshold per account. Per-symbol / per-strategy granularity: L2.
• Remote kill switch. stop_event from another process requires multiprocessing.Event or a file watcher. The watchdog itself is thread-local.