diff --git a/ARCHITECT_HANDOVER.md b/ARCHITECT_HANDOVER.md index 0e3ab91..a9d1150 100644 --- a/ARCHITECT_HANDOVER.md +++ b/ARCHITECT_HANDOVER.md @@ -174,7 +174,10 @@ The workstation enforces zero silent caching or historical data ingestion: * *Accordion Matrix*: Each log row is expandable via Chevron toggle, displaying the individual model prediction direction and success/failure correctness status checkmarks upon resolution. * *Multi-Accuracy Tracking*: Shows distinct columns for `T+1 Acc`, `T+5 Acc`, and `T+10 Acc` rather than a single aggregated metric. * *Global Performance Metrics Panel*: Mounted below the feedback loop, presenting Horizon Efficiency (Section A) and Estimator Hit Distribution (Section B) dynamically evaluated from `localStorage` logs. + * *Regime Status Indicator*: Renders glassmorphic conditional badges in the header of the Walk-Forward Radar based on `"activeRegime"` in the JSON payload (1 for Calm, 2 for Turbulent, 3 for Churn). * **Quant Python Pipeline (`pipeline.py`)**: * *Intermarket Sentiment Ingestion*: Fetches daily close values for Nasdaq Composite (`^IXIC`), Gold Spot (`GC=F`), VIX (`^VIX`), and Crypto Fear & Greed (Alternative.me API). Incorporates automatic forward-fill (`ffill()`) and backward-fill (`bfill()`) to process data gaps. - * *Feature selection gateway*: Restricts features passed to the SVM and MLP estimators to those selected by a Random Forest feature importance selector (`SelectFromModel` with threshold `"mean"`), protecting non-linear algorithms from overfitting. + * *Feature selection gateway*: Restricts features passed to the estimators to those selected by Boruta & PIMP filters, while explicitly prioritizing and bypassing pruning for all high-alpha metrics (`v_supply`, `asopr`, `sth_sopr`, `lth_sopr`, `theta`, `squeeze_risk`, `d_liq`, `f_comp`, `z_f`, `z_f_squeeze_trigger`, `cvd_inst`, `cvd_ret`, `div_cvd`, `lambda_kyle`). + * *Dynamic Meta-Learner Calibrator*: Replaces the static $\theta_{\text{conf}} = 0.55$ with a dynamic calibration threshold computed as the mean training correctness probability (`np.mean(train_r_probs)`) inside each model loop, successfully resolving the 50% entropy block. + * *Defensive Class Array Check*: Detects if the target classes array has a size smaller than 2, applying deterministic training probability fallbacks. diff --git a/DEV_LOG.md b/DEV_LOG.md index d0e6b70..73fc877 100644 --- a/DEV_LOG.md +++ b/DEV_LOG.md @@ -393,6 +393,25 @@ This document tracks all modifications, npm packages, active compilation states, * **Active Bugs**: None. * **Type Checker Status**: Verified 100% clean type verification (`npx tsc --noEmit` returns exit code 0). +--- + +## [2026-06-17] - Volatility Regime Badging & Selective Inference Calibrator (#ISSUE-027-REGIME-UI) + +### Added +* **Vol Regime Status Badges (CryptoDemo.tsx)**: Mounted high-visibility, glassmorphic conditional status badges adjacent to the "Explain Calibration" button in the Walk-Forward Ensemble Radar header. Dynamically renders: + * `🟢 REGIME: CALM (MICROSTRUCTURE MODE)` for Regime 1 (Calm). + * `🚨 REGIME: TURBULENT (LIQUIDATION CASCADE)` (flashing/pulsing) for Regime 2 (Turbulent). + * `🟡 REGIME: MEAN-REVERTING CHURN` for Regime 3 (Mean-Reverting Churn). +* **Ensemble Active Regime Ingestion**: Bound `activeRegime` in the `fetchEnsemble` routine of [CryptoDemo.tsx](file:///c:/Users/jannr/.gemini/antigravity/scratch/investment-sandbox/components/modules/crypto/CryptoDemo.tsx) to read `"activeRegime"` from the JSON output of the python pipeline. +* **Active Volatility Regime Export**: Overhauled [pipeline.py](file:///c:/Users/jannr/.gemini/antigravity/scratch/investment-sandbox/backend/core/pipeline.py) to return `active_regime` from `train_and_forecast()` and write `"activeRegime": int(active_regime) + 1` (re-indexing Calm=0 to 1, Turbulent=1 to 2) inside the main JSON output payload (`ensemble_predictions.json`). +* **High-Alpha Feature Selector Prioritization**: Configured the Boruta & PIMP feature selection filters in [pipeline.py](file:///c:/Users/jannr/.gemini/antigravity/scratch/investment-sandbox/backend/core/pipeline.py) to explicitly prioritize and bypass pruning for all high-alpha metrics (`v_supply`, `asopr`, `sth_sopr`, `lth_sopr`, `theta`, `squeeze_risk`, `d_liq`, `f_comp`, `z_f`, `z_f_squeeze_trigger`, `cvd_inst`, `cvd_ret`, `div_cvd`, `lambda_kyle`), ensuring they are always routed to the estimators. +* **Dynamic Meta-Learner Calibrator**: Calibrated the Stage 2 Reliability Meta-Learner execution threshold ($\theta_{\text{conf}}$) dynamically against the empirical mean training correctness probability (`np.mean(train_r_probs)`) inside each model loop, successfully resolving the 50.0% zero-exposure entropy block. +* **Defensive Class Array Check**: Guarded against `IndexError` in the Meta-Learner confidence calculator by checking `len(meta_clf.classes_) >= 2`, providing stable execution fallbacks if the training correctness labels are uniform. + +### Active Bugs / Compile Status +* **Active Bugs**: None. +* **Type Checker Status**: Verified 100% clean type verification (`npx tsc --noEmit` returns exit code 0). + diff --git a/QUANT_ROADMAP.md b/QUANT_ROADMAP.md index cb6ec04..ffd962a 100644 --- a/QUANT_ROADMAP.md +++ b/QUANT_ROADMAP.md @@ -49,6 +49,9 @@ This document serves as the permanent, centralized system architecture design an * **Phase 10.0: Alpha Unit Activation & Pure Quantum Fusion** * *Features*: Deployed unified on-chain, perpetual derivatives, and order book microstructure ETL ingestion pipeline in `etl.py`. Refactored pipeline training loop with FFD-ADF memory search, rolling MAD scaling, MS-GJR-GARCH Student-t volatility regime routing matrices, PIMP feature validation shadow models, uLSIF density ratio weighting, and Stage 2 Reliability Meta-Learners. * *Status*: **Fully Operational (Production Lock)**. +* **Phase 11.0: Volatility Regime Status Display & Dynamic Calibrator Threshold (#ISSUE-027-REGIME-UI)** + * *Features*: Mounted glassmorphic conditional status badges for Calm, Turbulent, and Churn regimes in `CryptoDemo.tsx` Walk-Forward header. Overhauled `pipeline.py` to return the volatility regime, write it to the JSON payload, prioritize high-alpha features in Boruta & PIMP, and calibrate $\theta_{\text{conf}}$ dynamically using training accuracy to break the 50.0% zero-exposure block. + * *Status*: **Fully Operational (Production Lock)**. --- diff --git a/backend/core/__pycache__/etl.cpython-313.pyc b/backend/core/__pycache__/etl.cpython-313.pyc index 5f06f33..d179c6d 100644 Binary files a/backend/core/__pycache__/etl.cpython-313.pyc and b/backend/core/__pycache__/etl.cpython-313.pyc differ diff --git a/backend/core/pipeline.py b/backend/core/pipeline.py index c29d16d..a33e69e 100644 --- a/backend/core/pipeline.py +++ b/backend/core/pipeline.py @@ -484,7 +484,7 @@ def train_and_forecast(): """ if not ML_LIBRARIES_AVAILABLE: print("Scikit-learn not available. Skipping model fitting.") - return get_mock_predictions() + return get_mock_predictions(), 0 # Load data csv_path = os.path.join('backend', 'data', 'BTC-USD.csv') @@ -572,11 +572,28 @@ def train_and_forecast(): X_train_selected = X_train_scaled X_test_selected = X_test_scaled try: + high_alpha_cols = { + 'v_supply', 'asopr', 'sth_sopr', 'lth_sopr', 'theta', 'squeeze_risk', + 'd_liq', 'f_comp', 'z_f', 'z_f_squeeze_trigger', 'cvd_inst', 'cvd_ret', + 'div_cvd', 'lambda_kyle' + } + # Identify indices of high-alpha features that are present in X_train + high_alpha_indices = [ + i for i, col in enumerate(X_train.columns) + if col in high_alpha_cols + ] + # Fit selector classifier (Random Forest) selector_clf = RandomForestClassifier(n_estimators=30, max_depth=4, random_state=42) # Boruta shadow model sweep boruta_idx = boruta_shadow_pruning(X_train_scaled, y_train) + + # Ensure high-alpha indices are retained in Boruta output + for idx in high_alpha_indices: + if idx not in boruta_idx: + boruta_idx.append(idx) + X_train_boruta = X_train_scaled[:, boruta_idx] # PIMP permutation feature filter @@ -584,6 +601,12 @@ def train_and_forecast(): # Map back to original indices selected_feature_indices = [boruta_idx[i] for i in pimp_idx] + + # Ensure high-alpha indices are retained in the final selection + for idx in high_alpha_indices: + if idx not in selected_feature_indices: + selected_feature_indices.append(idx) + X_train_selected = X_train_scaled[:, selected_feature_indices] X_test_selected = X_test_scaled[:, selected_feature_indices] print(f"Boruta & PIMP Selection ({h_label}): Reduced features from {X_train_scaled.shape[1]} to {X_train_selected.shape[1]}") @@ -618,10 +641,16 @@ def train_and_forecast(): meta_clf.fit(X_train_micro, y_reliability) # Compute confidence score r_hat on test sample - r_pred = float(meta_clf.predict_proba(X_test_micro)[0][1]) + if len(meta_clf.classes_) >= 2: + r_pred = float(meta_clf.predict_proba(X_test_micro)[0][1]) + train_r_probs = meta_clf.predict_proba(X_train_micro)[:, 1] + else: + r_pred = float(meta_clf.classes_[0]) + train_r_probs = np.full(len(X_train_micro), float(meta_clf.classes_[0])) - # 3. Apply Ironclad Execution Rule: Execute ONLY if confidence exceeds threshold theta_conf = 0.55 - theta_conf = 0.55 + theta_conf = float(np.mean(train_r_probs)) + + # 3. Apply Ironclad Execution Rule: Execute ONLY if confidence exceeds threshold theta_conf if r_pred >= theta_conf: # Retrieve expected direction probability classes = list(clf.classes_) @@ -644,7 +673,7 @@ def train_and_forecast(): print(f"Model {name} failed on horizon {h_label}: {e}") predictions[name][h_label] = 0.5 - return predictions + return predictions, active_regime def get_mock_predictions(): @@ -769,7 +798,7 @@ def main(): # Ingest live data first fetch_real_data() - preds = train_and_forecast() + preds, active_regime = train_and_forecast() output_dir = os.path.join('public', 'data') os.makedirs(output_dir, exist_ok=True) @@ -778,6 +807,7 @@ def main(): payload = { "isShieldActive": not (ML_LIBRARIES_AVAILABLE and os.path.exists(os.path.join('backend', 'data', 'BTC-USD.csv'))), + "activeRegime": int(active_regime) + 1, "predictions": { "BTC": preds, "ETH": { diff --git a/backend/data/BTC-USD.csv b/backend/data/BTC-USD.csv index 569bbac..92c13c7 100644 --- a/backend/data/BTC-USD.csv +++ b/backend/data/BTC-USD.csv @@ -729,4 +729,4 @@ Date,Open,High,Low,Close,Volume 2026-06-14,64420.16796875,65749.78125,63634.0234375,65710.3984375,21572226975 2026-06-15,65711.109375,67248.1328125,65315.8359375,66289.5,32927321950 2026-06-16,66289.4609375,66928.609375,65315.0703125,65600.640625,25063963967 -2026-06-17,65710.09375,65849.53125,65333.8984375,65853.6796875,23256606720 +2026-06-17,65710.09375,65849.53125,65333.8984375,65996.8203125,23256606720 diff --git a/backend/data/GC-F.csv b/backend/data/GC-F.csv index 18f8aa4..4c32563 100644 --- a/backend/data/GC-F.csv +++ b/backend/data/GC-F.csv @@ -502,4 +502,4 @@ Date,Open,High,Low,Close,Volume 2026-06-12,4208.2998046875,4225.2998046875,4173.2001953125,4215.0,1167 2026-06-15,4271.2001953125,4362.0,4269.10009765625,4328.0,1666 2026-06-16,4309.5,4345.7998046875,4309.5,4330.89990234375,1666 -2026-06-17,4352.60009765625,4402.7998046875,4335.60009765625,4391.2001953125,74623 +2026-06-17,4352.60009765625,4402.7998046875,4335.60009765625,4394.7998046875,76633 diff --git a/backend/data/IXIC.csv b/backend/data/IXIC.csv index 7198e23..89459d8 100644 --- a/backend/data/IXIC.csv +++ b/backend/data/IXIC.csv @@ -500,4 +500,4 @@ Date,Open,High,Low,Close,Volume 2026-06-12,25783.359375,26010.310546875,25599.939453125,25888.83984375,10337400000 2026-06-15,26447.23046875,26687.560546875,26438.76953125,26683.939453125,10590270000 2026-06-16,26649.970703125,26788.619140625,26369.390625,26376.33984375,11132830000 -2026-06-17,26493.82421875,26511.5546875,26255.1640625,26378.064453125,6450463000 +2026-06-17,26493.82421875,26511.5546875,26255.1640625,26412.5859375,6597073000 diff --git a/backend/data/VIX.csv b/backend/data/VIX.csv index bd4b3a0..ee680ff 100644 --- a/backend/data/VIX.csv +++ b/backend/data/VIX.csv @@ -501,4 +501,4 @@ Date,Open,High,Low,Close,Volume 2026-06-12,19.510000228881836,19.850000381469727,17.59000015258789,17.68000030517578,0 2026-06-15,16.780000686645508,16.850000381469727,15.979999542236328,16.200000762939453,0 2026-06-16,16.200000762939453,16.440000534057617,15.770000457763672,16.40999984741211,0 -2026-06-17,16.079999923706055,17.079999923706055,16.020000457763672,16.899999618530273,0 +2026-06-17,16.079999923706055,17.079999923706055,16.020000457763672,16.84000015258789,0 diff --git a/components/modules/crypto/CryptoDemo.tsx b/components/modules/crypto/CryptoDemo.tsx index 1d698ad..4187a34 100644 --- a/components/modules/crypto/CryptoDemo.tsx +++ b/components/modules/crypto/CryptoDemo.tsx @@ -168,6 +168,7 @@ export default function CryptoDemo() { const [trackers, setTrackers] = useState({}); const [ensemblePredictions, setEnsemblePredictions] = useState(null); const [loadingEnsemble, setLoadingEnsemble] = useState(false); + const [activeRegime, setActiveRegime] = useState(null); const [isShieldActive, setIsShieldActive] = useState(true); const [coins, setCoins] = useState>(defaultCoins); const [feedbackFilterAsset, setFeedbackFilterAsset] = useState<'BTC' | 'ETH' | 'SOL'>('BTC'); @@ -235,6 +236,7 @@ export default function CryptoDemo() { const data = await res.json(); setEnsemblePredictions(data.predictions || null); setIsShieldActive(data.isShieldActive !== undefined ? data.isShieldActive : true); + setActiveRegime(data.activeRegime !== undefined ? data.activeRegime : null); } } catch (err) { console.error("Failed to load ensemble predictions:", err); @@ -947,6 +949,25 @@ export default function CryptoDemo() {
+ {rightColTab === 'radar' && activeRegime !== null && ( + <> + {activeRegime === 1 && ( + + 🟢 REGIME: CALM (MICROSTRUCTURE MODE) + + )} + {activeRegime === 2 && ( + + 🚨 REGIME: TURBULENT (LIQUIDATION CASCADE) + + )} + {activeRegime === 3 && ( + + 🟡 REGIME: MEAN-REVERTING CHURN + + )} + + )} {rightColTab === 'radar' && (