Infra Costs - Training Large Datasets - Deep Learning

Infra and Costs - Link

Insights - Link

• Infra - GTX 1080 TI GPUs and cuDNN
• Dataset - 220,000 carefully annotated hair images

Infra Providers - Cirrascale, Lambda

Training large models - Link

• 4 days to train GPT-3 on 1,024x NVIDIA A100 GPUs.
• With each A100 GPU priced at $9,900, we’re talking almost $10,000,000 to setup a cluster that large
• you can rent A100 GPUs from public cloud providers like Google Cloud, but at $2.933908 per hour, that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days
• Each TITAN X, for example, costs roughly $3,000

Keep Exploring!!!