README.md

metadata

license: apache-2.0
tags:
  - pathology
  - foundation-model
  - medical-imaging
  - computational-pathology
  - histopathology
  - vision-transformer
  - dinov2
  - vision
extra_gated_prompt: >

  The OpenMidnight model weights and associated code are released under the
  Apache License, Version 2.0 (the "License"). You may obtain a copy of the
  License at:

  http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
  License for the specific language governing permissions and limitations under
  the License.

  Please note that the primary email used to sign up for your Hugging Face
  account must match your institutional email to receive approval. By
  downloading the OpenMidnight model weights, you attest that all information
  (affiliation, research use) is correct and up-to-date. Downloading the model
  requires prior registration on Hugging Face and agreeing to the terms of use.

  By using the OpenMidnight model, you acknowledge that you have read and
  understood these terms.
extra_gated_fields:
  First and Last Name: text
  Institutional Email (must match your primary HuggingFace email): text
  I agree to the license and terms of use described above: checkbox

OpenMidnight

Overview of the OpenMidnight pathology foundation model

State-of-the-art pathology foundation model trained on 12K slides

Developed by Sophont

Blog Post | GitHub | Demo

---

What is OpenMidnight?

OpenMidnight is our open replication of Kaiko.AI's Midnight, a 1.1 billion parameter Vision Transformer foundation model for computational pathology. OpenMidnight achieves state-of-the-art performance despite being trained on significantly less data than Kaiko.AI's Midnight or comparable models.

Key advantages:

• 🏆 State-of-the-art performance: Achieves 0.775 average score across 14 benchmarks
• ⚡ Efficient training: Trained in ~83 hours on 8× H100 GPUs for only $1,600 USD (estimated)
• 📊 Minimal data requirements: Uses only 12K slides from TCGA for training
• 🔓 Fully open source: Complete model weights, training code, and pipeline publicly available

OpenMidnight is aimed for computational pathology tasks including:

• Tumor detection and classification
• Histological grading
• Tissue segmentation
• Margin assessment
• Clinical outcome prediction

---

Model Description

• Developed by: Sophont
• Model type: Finetuned DINOv2 ViT-G for H&E pathology images
• Training data: TCGA, 12M H&E WSI
• Training repository: https://github.com/MedARC-AI/OpenMidnight/tree/main

Usage

Requirements

pip install torch torchvision huggingface_hub

Recommended: Run on GPU with mixed precision for optimal performance.

Quick Start: Loading the Model

import torch
from huggingface_hub import hf_hub_download

#Downloads to hf cache location
download_location = hf_hub_download(repo_id="SophontAI/OpenMidnight", filename="teacher_checkpoint_load.pt")
model = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14_reg', weights = None)

#Load OpenMidnight weights
checkpoint = torch.load(download_location, map_location = "cpu")

#Required because dinov2 is baseline 392 and we are baseline 224 resolution
pos_embed = checkpoint["pos_embed"]
model.pos_embed = torch.nn.parameter.Parameter(pos_embed)
model.load_state_dict(checkpoint)
model.eval()

print(f"Model loaded with {sum(p.numel() for p in model.parameters()):,} parameters")

Extracting Embeddings from Tissue Patches

from PIL import Image
import torchvision.transforms as transforms

# Standard preprocessing for pathology images
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],  # ImageNet normalization
        std=[0.229, 0.224, 0.225]
    )
])

# Load and preprocess an H&E tissue patch
image = Image.open("path/to/tissue_patch.jpg")
input_tensor = transform(image).unsqueeze(0)  # Shape: [1, 3, 224, 224]

# Extract embeddings
with torch.no_grad():
    embeddings = model(input_tensor)  # Shape: [1, 1536]

print(f"Embedding shape: {embeddings.shape}")
print(f"Embedding norm: {embeddings.norm().item():.4f}")

---

Model Performance

OpenMidnight achieves competitive or superior performance compared to models trained on 8-30× more data:

Benchmark Comparison

Average performance of top pathology foundation models and baselines across computational pathology benchmarks

Detailed Benchmark Results

Model	#WSIs	PCam (10 shots)	BACH	BRACS	BreakHis	CRC-100K	Gleason	MHIST	PCam	Cam16 (small)	Panda (small)	CoNSeP	MoNuSAC	HEST	Average
OpenMidnight (Ours)	12K	0.790	0.916	0.661	0.873	0.961	0.817	0.844	0.938	0.946	0.652	0.631	0.655	0.390	0.775
Midnight	92K	0.900	0.906	0.642	0.850	0.964	0.809	0.825	0.951	0.831	0.633	0.663	0.707	0.384	0.774
UNI-2	350K	0.887	0.914	0.661	0.860	0.965	0.778	0.823	0.949	0.868	0.659	0.628	0.644	0.414	0.773
UNI-2/392	350K	0.821	0.917	0.663	0.829	0.965	0.791	0.849	0.927	0.858	0.653	0.629	0.659	0.407	0.767
Virchow2	3.1M	0.851	0.884	0.624	0.823	0.966	0.778	0.861	0.936	0.865	0.656	0.639	0.676	0.398	0.766
Midnight 92k	92K	0.876	0.896	0.616	0.789	0.966	0.820	0.811	0.950	0.861	0.625	0.629	0.656	0.392	0.761
Midnight 12k	12K	0.791	0.904	0.644	0.841	0.966	0.801	0.807	0.930	0.850	0.663	0.626	0.663	0.395	0.760
H-Optimus-0	500K	0.824	0.757	0.615	0.808	0.956	0.771	0.842	0.942	0.838	0.670	0.644	0.685	0.415	0.751
Kaiko-B8	29K	0.786	0.872	0.617	0.825	0.957	0.748	0.828	0.917	0.831	0.642	0.643	0.686	0.373	0.748
TCGA-100M	12K	0.774	0.864	0.615	0.779	0.967	0.799	0.792	0.927	0.852	0.667	0.622	0.656	0.396	0.747
Prov-GigaPath	171K	0.852	0.766	0.616	0.821	0.951	0.720	0.831	0.942	0.791	0.660	0.626	0.687	0.393	0.743
Hibou-L	1.1M	0.804	0.811	0.637	0.740	0.933	0.763	0.839	0.952	0.823	0.634	0.645	0.668	0.388	0.740
UNI	100K	0.815	0.791	0.593	0.789	0.948	0.757	0.840	0.938	0.822	0.655	0.627	0.659	0.386	0.740
UNI/512	100K	0.737	0.877	0.612	0.732	0.950	0.754	0.814	0.883	0.814	0.654	0.621	0.658	0.364	0.728
Phikon	12K	0.820	0.735	0.568	0.713	0.942	0.729	0.804	0.923	0.809	0.644	0.623	0.644	0.367	0.717
Phikon v2	60K	0.741	0.734	0.600	0.716	0.939	0.755	0.784	0.893	0.803	0.631	0.626	0.645	0.375	0.711
DINOv2-giant (pretrained)	0	0.719	0.725	0.583	0.832	0.935	0.744	0.862	0.874	0.507	0.382	0.564	0.614	0.342	0.668
DINOv2-giant (random)	0	0.649	0.473	0.411	0.427	0.748	0.464	0.569	0.755	0.566	0.308	0.461	0.428	0.172	0.495

Performance comparison of OpenMidnight to existing pathology foundation models on eva+HEST benchmarks. Scores for existing models are taken from Midnight paper. We report balanced accuracy for the classification tasks, Dice score for semantic segmentation (CoNSeP and MoNuSAC), and the average Pearson correlation for the nine HEST regression tasks. Only performance with [CLS] token is reported. Best score per dataset is bolded.

---

Model Details

Architecture

Parameter	Value
Base Architecture	ViT-G/14
Parameters	1.1 billion
Patch Size	14×14 pixels
Input Resolution	224×224 pixels
Embedding Dimension	1536
Number of Layers	40
Number of Heads	16
Initialization	Meta's DINOv2 pre-trained weights

Training Data

• Dataset: TCGA (The Cancer Genome Atlas)
• Slides: 12k FFPE H&E-stained whole slide images
• Cancer Types: 32 different cancer types
• Total Patches: 96 million
• Unique Patches: 29 million
• Stain Type: Hematoxylin and Eosin (H&E)
• Preprocessing: Non-informative patch filtering

Training Configuration

• Hardware: 8× NVIDIA H100 GPUs (80GB each)
• Batch Size: 48 per GPU (384 global batch size)
• Training Steps: 250,000
• Optimizer: AdamW
• Learning Rate: 2.0e-4
• Regularization: KDE regularizer for training stability
• Augmentation: Hematoxylin-Eosin-DAB colorspace transformations
• Training Time: ~83 hours wall-clock time (667 GPU-hrs)
• Training Cost: ~$1,600 USD (at $2.50/H100/hour)

---

Blog Post

For an in-depth discussion of OpenMidnight, read the full blog post.

---

Contact

For questions, feedback, or collaboration opportunities:

• 📧 Email: [email protected]
• 🌐 Website: sophont.med
• 🐦 Twitter/X: @SophontAI
• 💬 GitHub Issues: github.com/MedARC-AI/OpenMidnight/issues

We welcome:

• Bug reports and feature requests
• Contributions to the training code
• Benchmark results on new datasets
• Applications of OpenMidnight to novel tasks

---

Acknowledgments

We thank Mikhail Karasikov for answering questions about Midnight. We thank Nicolas Känzig for answering questions about eva. We thank the members of MedARC and the broader research community for their feedback and support. We are very grateful to FAL AI for granting compute to support this open-source research.

---

Citation

If you use OpenMidnight in your research, please cite:

@article{kaplan2025openmidnight,
            author = {Kaplan, Daniel and Grandhi, Ratna Sagari and Lane, Connor and Warner, Benjamin and Abraham, Tanishq Mathew and Scotti, Paul S.},
            title = {How to Train a State-of-the-Art Pathology Foundation Model with \$1.6k},
            year = {2025},
            url = {https://sophont.med/blog/openmidnight},
          }

---

License

This model is released under the Apache 2.0 License.

---

Terms of Use

Research Use: This model is primarily intended for research purposes in computational pathology, medical imaging, and related fields.

Clinical Use: This model is not intended for use in medical diagnosis, treatment, or prevention of disease of real patients. It should not be used as a substitute for professional medical advice.

Responsible Use: Users should:

• Validate model performance on their specific use cases
• Be aware of potential biases in the training data (TCGA)
• Consider demographic and geographic limitations
• Respect privacy rights and comply with applicable data protection laws
• Follow applicable regulations and ethical guidelines

---