GitHub / IBM / ModuleFormer
ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2FModuleFormer
Stars: 220
Forks: 11
Open issues: 3
License: apache-2.0
Language: Python
Size: 71.3 KB
Dependencies parsed at: Pending
Created at: over 1 year ago
Updated at: about 1 month ago
Pushed at: about 1 year ago
Last synced at: 17 days ago
Commit Stats
Commits: 15
Authors: 4
Mean commits per author: 3.75
Development Distribution Score: 0.4
More commit stats: https://commits.ecosyste.ms/hosts/GitHub/repositories/IBM/ModuleFormer