Synthetic Data Enhanced Prediction of the neuropathological landscape of dementia before death using multimodal AI

Pathology-Aware Representation Learning, Prediction, and Synthetic Data in NACC-Style CohortsThis repository contains the full analysis pipeline and supporting code for pathology-aware representation learning, neuropathology prediction, and DDPM-based synthetic patient generation in NACC-style datasets, together with harmonized external cohorts (ROSMAP and Munich Neurobiobank).The deposit is organized into three main components:Main Clean Code.ipynb – final integrated TabPFN/SEM/DDPM pathology pipeline and prediction workflows.1Rosmap&NbmDtata2NaccStyle.ipynb – harmonization of ROSMAP and Munich Neurobiobank into NACC-compatible tables and joint latent-factor export.1RawCode.zip – archive of original scripts used during development, figure generation, and SHAP-based interpretation.1Additional model result bundles (e.g. DDPMModel&Result.zip, CoPathologyModel&Result.zip) provide trained models and evaluation outputs corresponding to the pipelines in the main notebook.11. Main Clean Code.ipynbPurposeMain Clean Code.ipynb is the primary research pipeline. It composes several interlocking workflows for:TabPFN-based embedding learning from clinical features with explicit NACCID traceability.Construction of five normalized pathology latent variables (0–1) via TabPFN imputation, correlation filtering, and SEM.DDPM-based conditional synthetic patient generation over the 5D pathology latent space (healthy → maximal pathology).Downstream prediction pipelines for ordinal and continuous neuropathology targets, including co-pathology and cost-group analyses, ROC/PR evaluation, and DDPM-augmented oversampling variants.11.1 TabPFN Embedding WorkflowThis section loads a NACC-style training CSV, performs standardized cleaning, and trains a TabPFN-based embedding model:Cleans custom missing codes (e.g. 8, 9, 88, 99, 888, 999, 88.8, 99.9) to NaN in numeric columns.Derives an lvSUM target as the sum of latent variables lv1–lv5 and keeps only rows with non-missing lvSUM.Extracts numeric clinical features while excluding pathology targets, NACCID, and lv columns, then fits a TabPFNEmbedding model on $X$ with target lvSUM.Saves:Embeddings as .npy (train_tabpfn_lvsum_embeddings.npy) and .csv with NACCID, lvSUM, and a sample_index column.Feature-name list and a small summary CSV with number of samples, features, device, and checkpoint paths.Throughout, the notebook enforces strict alignment between rows and NACCID and aborts if counts diverge.1This produces TabPFN embeddings that can be reused across downstream models while preserving sample-level traceability.1.2 Pathology Latent-Variable Module (SEM 0–1 Factors)The pathology latent module builds a 5D latent pathology representation scaled to:1Pathology feature extraction and imputationLoads neuropathology tables (e.g. Thal phase, CERAD, Braak stage, NIA–Reagan, TDP-43, Lewy body staging, infarcts, hemorrhages, vascular markers).Casts all selected pathology columns to numeric and imputes missing values column-wise using a TabPFN classifier trained on the other pathology features.1Correlation filtering / multicollinearity handlingUses absolute pairwise correlations to identify highly collinear measure groups (e.g. Braak stage vs BRAAKBScore) and keeps a single representative per group, logging which variables were kept and removed.1SEM latent-factor modelBuilds a SEM specification where each latent pathology factor (e.g. amyloid, tau/Braak/ADNC, TDP-43, Lewy, cerebrovascular) loads onto its corresponding manifest pathology indicators.Standardizes inputs, fits the model with semopy, and extracts factor scores, padding to 5 latent dimensions if needed.1Scaling to 0–1 and savingApplies a MinMax scaler to map factor scores to, where 0 ≈ low/no pathology and 1 ≈ maximal observed pathology.1Saves:LV1–LV5 columns plus NACCID as a dataframe.The scaling object (latent0to1_scaler.pkl) for later reuse.1These five normalized latent pathology factors are then merged with clinical tables and used as conditioning variables for DDPM sampling and for downstream modeling.1.3 DDPM-Based Synthetic Patient Generation (Latent 0–1)The DDPM section trains a conditional denoising diffusion model on TabPFN embeddings and latent pathology labels:TabPFN embedding with missingness masksConcatenates TabPFN probability embeddings with standardized missing-value masks for all clinical features.Stores the quantile transformer, mask scaler, and feature metadata in embedding_transformers.pkl to invert embeddings back to feature space.1DDPM architectureImplements a fully connected residual DDPM with:Time embedding (nn.Embedding+MLP), latent pathology embedding, and residual blocks with SiLU activations and LayerNorm.Diffusion schedule with 1000 steps and linear betas.1Training and generationTrains the DDPM on real training embeddings plus 5D latent pathology labels with cosine annealing LR for 300 epochs, saving model weights.Samples from the model for three canonical pathology settings:Healthy: all 0s in latent space (synthetichealthy0.csv).Moderate: all 0.5s (syntheticmoderate05.csv).Max pathology: all 1s (syntheticmaxpatho1.csv).Uses embedding_to_real_data to invert TabPFN probabilities and masks back to reconstructed feature space, assigning random values to masked entries as an approximate reconstruction step.1These synthetic cohorts are intended for distribution-shift analysis, DDPM-based oversampling, and qualitative exploration, not as direct replacement for real clinical data.1.4 Pathology Prediction PipelinesSeveral prediction workflows are implemented:Classical ML regression/classification per pathology groupLoads pathology targets and multiple clinical feature tables.For each pathology group (e.g. amyloid, tau/Braak/ADNC, TDP-43, Lewy, CVD), iterates over tables and targets, applying:Train/test splits (random or stratified for classification).Table-specific preprocessing with ColumnTransformer (median imputation for numeric features and one-hot encoding for categorical).Model fitting for Decision Trees, Random Forests, Gradient Boosting, XGBoost, LightGBM, CatBoost with optional grid search.Saves per-target models, metrics, and diagnostic plots into a nested output directory structure grouped by pathology, table, and model name.21Ordinal TabPFN + Random Forest ensemblesFor strictly ordinal targets, uses TabPFN-based classifiers combined with Random Forests in a VotingClassifier ensemble, with optional SMOTE or DDPM-based oversampling of rare classes.Computes comprehensive ordinal metrics: exact accuracy, ordinal MAE, weighted kappa, class-wise recall, and AUC for binary tasks.1Independent, leakage-robust pipelinesImplements stricter pipelines where each target is predicted from features only, with no other targets used as inputs, explicit train-only preprocessing, stratified splits, and explicit memory/CUDA cleanup after each target.21Co-pathology and cost-group analysesDefines cost groups over subsets of clinical tables (e.g. “low-cost clinical”, “imaging + biomarkers”) and evaluates pathology prediction across these groups.Provides post-processing scripts that aggregate per-target ROC/PR curves into micro/macro averaged ROC/PR summaries across experiments, and generates multi-panel plots comparing cost groups.21Outputs from these pipelines – including per-target metrics, feature-importance plots, and cost-group ROC/PR figures – are stored under the various pathology*results folders and bundled in CoPathologyModel&Result.zip.212. Rosmap&NbmDtata2NaccStyle.ipynbPurposeRosmap&NbmDtata2NaccStyle.ipynb (referred to in the Zenodo UI as Rosmap&NbmDtata2NaccStyle.ipynb) handles harmonization of external cohorts into a strict NACC-style schema. It standardizes ROSMAP and Munich Neurobiobank datasets to align column names, coding, and structure with the NACC clinical and pathology templates, enabling seamless reuse of the main TabPFN/SEM/DDPM pipeline on external cohorts.12.1 ROSMAP → NACC-style conversionThe ROSMAP section:Maps ROSMAP variables onto NACC-style columns for:Demographics, APOE and genetics, cognitive testing (e.g. MMSE), CSF biomarkers, imaging, and clinical diagnoses.Autopsy and neuropathology variables (amyloid, tau, Lewy body, TDP-43, vascular lesions).Derives diagnosis indicators by parsing text descriptions and structured fields, using rule-based patterns to infer NACC-like diagnosis flags when only textual pathology summaries are available.Encodes yes/no or ordinal fields into NACC-style categorical codes (e.g. 0/1, 1–3).Saves:A NACC-shaped ROSMAP clinical+pathology CSV with NACC-conformant columns.A mapping/review file documenting each ROSMAP source column, its NACC target, and any transformation logic applied.12.2 Munich Neurobiobank → NACC-style conversionThe Munich Neurobiobank section performs an analogous harmonization:Links Munich-specific clinical and pathology fields to NACC column names wherever possible.Converts free-text symptom and diagnosis descriptions into structured indicators using keyword rules and pattern matching.Implements proxy rules when multiple candidate variables map to the same NACC column, preferring the least-missing or clinically most specific source.Produces:A NACC-like CSV for the Munich Neurobiobank with standardized naming and coding.A mapping file summarizing source-to-target mappings and any proxy choices.12.3 Joint export with latent pathology factorsThe notebook also contains a companion workflow that merges:Harmonized clinical data from ROSMAP or Munich Neurobiobank.Real pathology variables (when available).Five pre-computed 0–1 latent pathology factors (frozen scaling).into a single NACC-style CSV per cohort, ready for downstream modeling in the main TabPFN/SEM/DDPM notebook.13. RawCode.zipRawCode.zip contains the original development scripts used during the project. These scripts include:Early versions of model training pipelines and correlation analyses.Figure-generation scripts (e.g. cross-correlation heatmaps across NACC tables, raincloud plots, fores