This section establishes the identity and purpose of Gemma, positioning it within the broader AI landscape. It explains its relationship to Google's proprietary Gemini models and introduces the core philosophy that drives its development and the ecosystem surrounding it.
1.1 Defining Gemma: Lightweight, Open-Weight Models for the AI Community
Gemma is a family of lightweight, state-of-the-art generative artificial intelligence models developed by Google.1 The name, derived from the Latin word for "precious stone," reflects the models' value and accessibility.2 A defining characteristic of the Gemma family is its "open-weight" nature.1 Google provides free access to the model weights, which are the learned parameters of the neural network, allowing for both individual and commercial use, redistribution, and modification.3 This open approach is a deliberate strategy to foster a rich ecosystem of innovation, collaboration, and responsible AI development within the global research and developer communities.2
The models are engineered to be "lightweight," a term that signifies their relative efficiency in terms of computational and memory requirements.3 This design focus enables Gemma models to be deployed across a diverse spectrum of hardware, from high-end servers equipped with Google Cloud TPUs and NVIDIA GPUs down to consumer-grade hardware like laptops, mobile devices, and single-board computers.1 By lowering the barrier to entry, this efficiency makes advanced AI capabilities more accessible to a broader range of users, from large enterprises to individual hobbyists.5
1.2 The Gemini Lineage: How Gemma Inherits from Google's Flagship Research
The Gemma models are not an independent research initiative but are directly derived from the same foundational research and technology that produced Google's flagship, closed-source, multimodal Gemini models.1 This lineage establishes a strong technical pedigree for Gemma, positioning it as a family of lightweight versions of Gemini that share a "similar DNA".3 This shared heritage means Gemma inherits core architectural principles, data processing methodologies, and advanced training techniques, including instruction tuning and reinforcement learning from human feedback (RLHF).8
The timeline of this technological transfer highlights the rapid pace of development in the field. The Gemini family was first announced in May 2023, with its initial release in December 2023.7 The first generation of Gemma models followed shortly after, launching in February 2024.7 This quick succession demonstrates a strategic decision by Google to port cutting-edge technology from a frontier, proprietary model to an open, community-focused one, accelerating the dissemination of advanced AI capabilities.
1.3 Core Philosophy: Accessibility, Responsible Development, and the "Gemmaverse" Ecosystem
Google's stated mission of "making AI helpful for everyone" serves as the philosophical underpinning for the Gemma project.7 A central pillar of this philosophy is a commitment to responsible development. Accompanying the model weights is the
Responsible Generative AI Toolkit, a suite of resources and tools designed to help developers build safer applications.8 This toolkit provides guidance on safety-tuning and implements guardrails to mitigate risks associated with open models, such as the generation of biased, harmful, or misinformative content.10
The open nature of Gemma has catalyzed the growth of the "Gemmaverse," a vibrant and expanding ecosystem of community-driven innovation.2 This ecosystem comprises tens of thousands of derivative model variants, tools, and applications created by developers and researchers worldwide.11 This flourishing community is enabled by Gemma's broad compatibility with a host of popular machine learning frameworks, including Keras, PyTorch, JAX, and platforms like Hugging Face, Ollama, and Vertex AI.11 The existence of over 60,000 community-created Gemma variants underscores the success of this open strategy in stimulating widespread adoption and collaborative development.11
1.4 Table: Overview of the Gemma Model Family
The following table provides a high-level summary of the major Gemma variants, their parameter sizes, primary modalities, and intended use cases, serving as a roadmap to the diverse and growing family of models.
Data Sources: 1
Section 2: The Architectural Blueprint of Gemma
This section performs a deep dive into the technical architecture of the Gemma models, tracing their evolution and explaining the function and significance of each key component.
2.1 Foundation: The Decoder-Only Transformer
All standard models within the Gemma family are built upon a decoder-only transformer architecture.3 This represents a significant design choice, deviating from the original encoder-decoder structure proposed in the seminal "Attention Is All You Need" paper.4 The function of a decoder-only model is fundamentally autoregressive; it generates output, such as text, in a sequential, token-by-token manner. Each newly generated token is predicted based on the sequence of all tokens that came before it.4 This inherent structure makes decoder-only models exceptionally well-suited for a wide range of generative tasks, including text completion, summarization, question answering, and translation.3
2.2 Architectural Specifics of the First Generation (Gemma 1)
The initial release of Gemma, comprising the 2B and 7B parameter models, established the architectural foundation for the family. These models incorporated several key optimizations that balanced performance with efficiency.
Multi-Query Attention (MQA) vs. Multi-Head Attention (MHA)
A critical architectural distinction between the first two Gemma models lies in their attention mechanisms. The Gemma 2B model utilizes Multi-Query Attention (MQA), whereas the larger Gemma 7B model employs the standard Multi-Head Attention (MHA).4 This was not an arbitrary choice but a deliberate engineering trade-off tailored to each model's intended hardware target.
Standard MHA enhances a model's representational power by allowing each attention "head" to learn independent query, key, and value projection matrices. This enables the model to simultaneously focus on different types of relationships within the data.17 However, this richness comes at a cost: a significant increase in memory bandwidth requirements during inference. At each step of generating a new token, the entire key-value (KV) cache, which stores the key and value vectors for all previous tokens, must be loaded from memory. This can become a major bottleneck, especially for models with many heads.18
MQA was developed to alleviate this bottleneck. It drastically reduces memory bandwidth usage by having all query heads share a single, common set of key and value heads.18 This architectural simplification shrinks the size of the KV-cache and, consequently, the amount of data that needs to be loaded at each decoding step, resulting in significantly faster inference.21 The Gemma 2B model was explicitly designed for deployment on CPUs and on-device applications, where memory bandwidth is a primary constraint.14 Therefore, the selection of MQA for the 2B model was a strategic optimization for speed and efficiency in resource-constrained environments, accepting a potential trade-off in model quality for a substantial gain in performance.18 Conversely, the Gemma 7B model, targeting more powerful hardware like GPUs and TPUs, could afford the higher memory and computational cost of MHA to maximize its representational learning capacity.14
Rotary Position Embeddings (RoPE)
Instead of using traditional absolute positional embeddings, Gemma models incorporate Rotary Position Embeddings (RoPE).4 RoPE injects information about the position of tokens into the model by applying a rotation to the query and key vectors that is dependent on their absolute position in the sequence. This method has been shown to be particularly effective for tasks that require a nuanced understanding of the relative positions of tokens.
GeGLU Activation Functions
Within the feed-forward network (FFN) of each transformer layer, the standard ReLU activation function is replaced by the GeGLU (Gated Linear Unit) activation function.4 This is more than a simple substitution of one non-linearity for another; it introduces a sophisticated gating mechanism. A GeGLU-based FFN splits its first linear projection into two parallel paths. One path is passed through a GELU activation, and the two paths are then multiplied together element-wise.4 This multiplicative interaction acts as a gate, allowing the network to dynamically control the flow of information for each token. It can selectively amplify or dampen features based on the input, effectively learning a token-specific transformation that enhances the model's expressive power and improves gradient flow compared to a static ReLU activation.22
RMSNorm for Training Stability
To ensure stable training, Gemma employs RMSNorm (Root Mean Square Layer Normalization) to normalize the inputs of each transformer sub-layer.14 RMSNorm is a simpler and more computationally efficient variant of the standard Layer Normalization technique, contributing to the overall efficiency of the model.
2.3 Evolution in Gemma 2: Introducing Hybrid Attention
The second generation of Gemma models introduced a more advanced attention mechanism. Gemma 2 pioneered a hybrid approach that alternated between local attention layers and global attention layers.15 This design was an important evolutionary step toward solving the computational challenges posed by very long sequences, serving as a precursor to the more refined system implemented in Gemma 3.
2.4 Architectural Leap in Gemma 3: The Era of Long Context and Multimodality
The Gemma 3 family represents a significant architectural leap forward, introducing capabilities for handling extremely long contexts and multimodal inputs.
Interleaved Global and Local Attention for Extended Context (128K Tokens)
The most striking improvement in Gemma 3 is the massive expansion of the context window from 8,192 tokens in the first generation to 128,000 tokens.11 This 16-fold increase allows the model to process and reason over entire novels, lengthy research papers, or hundreds of images in a single prompt.15
This capability is enabled by a fundamental change to the attention mechanism. A standard global attention mechanism, where every token attends to every other token, has a computational and memory complexity that scales quadratically with the sequence length. Extending this naive approach to 128K tokens would be computationally infeasible for inference due to the enormous size of the resulting KV-cache.15
Gemma 3 overcomes this limitation with a novel interleaved attention mechanism. The architecture is composed of repeating blocks, where five local attention layers are followed by one global attention layer.15 The local attention layers operate with a much smaller sliding window of 1024 tokens, which keeps their individual KV-caches manageable and their computations efficient. The interspersed global attention layers are then responsible for integrating information across the entire 128K token context.15 This hybrid design effectively balances the need to capture both short-range dependencies (via local attention) and long-range dependencies (via global attention), making the 128K context window practical for real-world use.
Advanced Architectural Refinements
Gemma 3 also incorporates several other key upgrades:
QK-Norm: The "soft-capping" mechanism used in Gemma 2's attention layers is replaced with QK-norm. This change leads to both improved accuracy and faster processing speeds.15
New Tokenizer: Gemma 3 adopts an improved SentencePiece tokenizer with an expanded vocabulary of 262,000 tokens. This new tokenizer is better balanced for non-English data, providing out-of-the-box support for over 140 languages.11
Multimodality: Integrating Vision Capabilities
The 4B, 12B, and 27B models in the Gemma 3 family are inherently multimodal, capable of processing both text and image inputs.23 This is achieved through the integration of a
SigLIP vision encoder.12 Images provided as input are first processed by this encoder, which transforms the visual data into a sequence of "soft tokens." These soft tokens are then fed into the language model alongside text tokens, allowing for seamless multimodal reasoning.15
To handle the variety of image resolutions and aspect ratios found in real-world data, Gemma 3 employs a "Pan & Scan" algorithm during inference. This algorithm adaptively crops a high-resolution or non-square image into smaller, square segments. Each segment is then resized and encoded individually. This technique improves the model's ability to perceive fine details in images, though it comes at the cost of some additional computational overhead.12
2.5 Table: Comparative Architectural Specifications
The following table provides a detailed comparison of key architectural parameters across different generations and sizes of the Gemma models, illustrating their technical evolution.
Data Sources: 4
Section 3: Implementation and Customization
This section bridges the gap between architectural theory and practical application, detailing how developers and researchers can deploy, customize, and conceptually reconstruct a Gemma model.
Part A: Practical Deployment and Fine-Tuning
3.1 The Gemma Toolkit: Frameworks and Access
Gemma's design emphasizes broad compatibility and accessibility. The models are natively supported by the most popular deep learning frameworks, including Keras 3.0, PyTorch, and JAX, ensuring that developers can integrate them into their preferred workflows.8
Access to the models is straightforward and available through multiple channels. Developers can download pre-trained and instruction-tuned model weights directly from community hubs like Hugging Face, Kaggle, and Ollama.2 For rapid prototyping and experimentation without the need for local setup, Google provides
AI Studio, a web-based interface for interacting with Gemma models directly.6
3.2 Deployment Strategies: From Local to Cloud
The Gemma family supports a wide array of deployment strategies, catering to different scales and operational requirements:
Local and On-Device Deployment: For applications requiring low latency and offline functionality, Gemma models can be run locally on consumer hardware. Tools like Ollama simplify deployment on laptops, while the Gemma.cpp library provides a lightweight C++ inference engine suitable for mobile devices and embedded systems.5
Managed Cloud Services: For developers seeking to build and scale applications without managing infrastructure, Google Cloud Vertex AI offers a fully managed platform. It provides tools for serving, monitoring, and scaling Gemma models, abstracting away the complexities of MLOps.1
Containerized Cloud Deployment: Organizations with in-house MLOps expertise and existing investments in containerization can deploy Gemma on Google Kubernetes Engine (GKE). This approach offers granular control over the deployment environment, making it ideal for complex AI/ML workloads with specific security, data pipeline, and resource management needs.1
Large-Scale Data Processing: Gemma models can be integrated into large-scale data processing pipelines using Google Cloud Dataflow. This is particularly useful for batch inference tasks, such as performing sentiment analysis on massive datasets.1
3.3 The Art of Fine-Tuning: SFT and RLHF
While the pre-trained Gemma models possess strong generalist capabilities, their true power is often unlocked through fine-tuning. The instruction-tuned variants provided by Google are created using a two-stage process:
Supervised Fine-Tuning (SFT): The base model is trained on a curated dataset of high-quality instruction-response pairs. This teaches the model to follow instructions and engage in dialogue.13
Reinforcement Learning from Human Feedback (RLHF): After SFT, the model is further refined using human preference data. A reward model is trained to predict which of two model responses a human would prefer, and this reward model is then used to optimize the language model's policy, aligning its behavior more closely with human values of helpfulness and safety.13
Developers can perform their own fine-tuning by following a general process: choose a framework, collect data, tune and test the model, and finally, deploy the customized version.28 A key finding is that significant behavioral changes can be achieved with relatively small, high-quality datasets. In some cases, as few as 20 to 200 well-crafted examples are sufficient to specialize a model for a particular task or domain.28
3.4 Resource-Efficient Customization with PEFT and LoRA
Full fine-tuning, which involves updating all of a model's billions of parameters, is computationally expensive and memory-intensive.28 To make customization more accessible, developers can use
Parameter-Efficient Fine-Tuning (PEFT) techniques.
Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods. Instead of updating the entire model, LoRA freezes the original pre-trained weights and injects small, trainable "adapter" matrices into the layers of the transformer. During fine-tuning, only these low-rank adapter matrices are updated, which represent a tiny fraction of the total parameter count. This approach dramatically reduces the memory footprint and computational requirements for training, making it possible to fine-tune large models on consumer-grade GPUs.5 Numerous tutorials and guides are available for implementing LoRA with frameworks like Keras and the Hugging Face PEFT library.6
Part B: Building a Gemma-like Transformer from Scratch
This subsection provides a conceptual walkthrough for implementing a decoder-only transformer that incorporates Gemma's key architectural features, using PyTorch for demonstration. This exercise is pedagogical, aimed at deepening the understanding of the model's inner workings.
3.5 Preliminaries: Tokenization and Embedding
The process begins with converting raw text into a format the model can understand. This involves:
Tokenization: Using a pre-trained tokenizer, such as the SentencePiece tokenizer used by Gemma, to break the input text into a sequence of integer IDs.31
Embedding: Creating an embedding layer (torch.nn.Embedding) that maps each token ID to a dense vector representation of size d_model. A specific detail from the Gemma architecture is to scale these output embeddings by the square root of the model's dimension (sqrt(d_model)) as a normalization step to control the variance of the activations.13
3.6 Implementing Rotary Position Embeddings (RoPE)
Unlike static positional encodings, RoPE is applied within the attention mechanism. The implementation involves:
Pre-computing the sinusoidal rotation matrices for each position up to the maximum context length.
Within the attention block's forward pass, before calculating attention scores, applying these pre-computed rotation matrices to the query and key vectors. This rotation encodes their positional information directly into their representations.
3.7 Implementing Multi-Query Attention (MQA)
To appreciate the efficiency of MQA, one can first implement standard MHA. The key modification for MQA is in the linear projection layers:
In MHA, you would define separate torch.nn.Linear layers for queries (W_q), keys (W_k), and values (W_v), where the output dimension is num_heads * head_dim.
In MQA, you still have a W_q layer for all heads, but you define only a single W_k and W_v layer whose output dimension is just head_dim. These single key and value projections are then broadcast or repeated to be used by all query heads during the attention calculation. This clearly demonstrates the reduction in parameters and subsequent memory load.
3.8 Implementing the GeGLU-based Feed-Forward Network
Contrasting with a standard FFN, which might look like Linear -> ReLU -> Linear, the GeGLU FFN is implemented as follows:
Define three linear layers: gate_proj, up_proj, and down_proj.
In the forward pass, the input x is passed through both gate_proj and up_proj in parallel.
The output of gate_proj is passed through an activation function (like GELU).
The result is then element-wise multiplied with the output of up_proj.
Finally, this gated output is passed through the down_proj layer.4
3.9 Assembling the Decoder Block
A single decoder block combines these components in the correct sequence:
The input first passes through an RMSNorm layer.
The normalized input is fed into the MQA (or MHA) block.
A residual connection adds the output of the attention block back to its input.
The result passes through a second RMSNorm layer.
This is then fed into the GeGLU-based FFN.
A final residual connection adds the FFN's output to its input.
This pre-normalization structure (applying normalization before the sub-layer) is common in modern transformers for improved training stability.15
3.10 Stacking the Decoder and Final Projection
The complete model is constructed by:
Stacking the implemented decoder block N times, where N is the number of layers in the model.
Adding a final linear projection layer (the "language model head") at the end. This layer takes the output from the final decoder block and projects it from the model dimension d_model to the vocabulary size, producing raw logits for each token in the vocabulary.33 A softmax function can then be applied to these logits to obtain a probability distribution for predicting the next token.
Section 4: Performance Analysis and Competitive Landscape
This section provides a quantitative analysis of Gemma's performance, comparing it across its own generations and against key competitors in the open-source landscape.
4.1 Key Evaluation Benchmarks Explained
To provide context for the performance data, the following standard benchmarks are briefly explained:
MMLU (Massive Multitask Language Understanding): A comprehensive benchmark designed to measure a model's general knowledge and problem-solving abilities. It consists of multiple-choice questions across 57 diverse subjects, including humanities, social sciences, and STEM fields.14
GSM8K (Grade-School Math): This benchmark assesses a model's capacity for multi-step mathematical reasoning. It contains word problems that require a sequence of elementary calculations to solve, testing reasoning rather than just rote memorization.14
HumanEval: A benchmark for evaluating the code generation capabilities of language models. It consists of 164 programming problems where the model must generate a correct Python function body from a docstring and function signature.14
LMSYS Chatbot Arena: Unlike automated benchmarks, this is a human-centric evaluation platform. Users chat with two anonymous models side-by-side and vote for the one that provides a better response. This crowd-sourced data is used to calculate an Elo rating, providing a measure of perceived quality in conversational tasks.24
4.2 Generational Improvements: Gemma 1 vs. Gemma 2 vs. Gemma 3
Each new generation of the Gemma family has delivered significant performance improvements. The transition from Gemma 1 to Gemma 2 brought better efficiency and performance. However, the leap to Gemma 3 was particularly substantial. The Gemma 3 models demonstrate superior performance across nearly all benchmarks compared to their Gemma 2 predecessors. This is especially true for tasks involving mathematics, chat capabilities, and multilingual understanding. The improvements are so significant that the instruction-tuned Gemma-3-4B model is competitive with the much larger Gemma-2-27B model, and the Gemma-3-27B model is comparable to Google's proprietary Gemini 1.5 Pro on several benchmarks.24
4.3 Comparative Analysis: Gemma vs. The Open-Source Field
The competitive landscape for open-source models is dynamic and highly dependent on the specific task and model scale. There is no single "best" model; instead, different models exhibit distinct strengths.
An analysis of benchmark data reveals several patterns. At smaller scales, Gemma models often demonstrate a strong aptitude for reasoning tasks. For instance, the Gemma 3 1B model shows stronger mathematical reasoning on the GSM8K benchmark compared to the similarly sized Llama 3.2 1B model, though it may lag slightly on broad knowledge benchmarks like MMLU.36 This suggests that Google's specialized post-training recipes for Gemma have successfully enhanced its reasoning capabilities.
At larger scales, the sheer size of competitor models often gives them an edge. The larger Llama 3 models (e.g., 70B and 405B) frequently outperform Gemma 2 models on complex benchmarks like HumanEval (coding) and MATH.38 The Mistral family of models also presents formidable competition. For example, Mistral Large 2 can outperform the Gemma 3 4B on HumanEval, while the Gemma 3 12B model surpasses Mistral Large 2 on GSM8K.40
Ultimately, Gemma's competitive advantage does not lie in topping every leaderboard with its largest model. Instead, its strength is in providing excellent performance-per-parameter and performance-per-watt. For developers with specific resource constraints or a need for highly efficient inference, Gemma presents a compelling option that delivers state-of-the-art capabilities in a more accessible package. The choice between Gemma, Llama, and Mistral depends heavily on the specific application, the required balance of general knowledge versus specialized reasoning, and the available computational budget.
4.4 Table: Cross-Model Benchmark Comparison
The following table synthesizes benchmark data from multiple sources to provide a clear, quantitative comparison between leading open-source models. Note that evaluation settings (e.g., number of "shots" or examples provided in the prompt) can vary, affecting scores.
Data Sources: 24
Section 5: Recent Developments and Future Trajectory
This final section covers the latest releases in the Gemma family, the emergence of specialized variants, and provides a concluding perspective on Gemma's strategic role in the AI ecosystem.
5.1 The Latest Releases: Gemma 3, ShieldGemma 2, and Gemma 3n
Google maintains a rapid release cadence for the Gemma family, continuously pushing new capabilities to the open-source community.
Gemma 3 (March 2025): This major release introduced multimodality (text and image input), a 128K token context window, native function calling, and a range of sizes from 1B to 27B parameters. These models represent a significant leap in capability and versatility over previous generations.9
ShieldGemma 2 (March 2025): Released alongside Gemma 3, ShieldGemma 2 is a specialized 4B parameter model built on the Gemma 3 foundation. It is an image safety checker designed to classify images against policies for dangerous, sexually explicit, or violent content, providing developers with a ready-made solution for content moderation.2
Gemma 3n (June 2025): The most recent addition, Gemma 3n, is a family of models highly optimized for on-device and low-resource environments. It extends multimodality to include audio input and introduces a novel architecture using "selective parameter activation." This technology allows the models to operate with a smaller effective parameter count than their total number of parameters, significantly enhancing inference efficiency without a proportional drop in performance.1
5.2 Specialized Frontiers: The Rise of Domain-Specific Gemma Models
The open-weight nature of Gemma makes it an ideal foundation for creating highly specialized models tailored to specific domains. This has led to a growing number of official variants:
MedGemma: This collection of models, available in 4B and 27B sizes, has been fine-tuned on extensive medical data. MedGemma is capable of complex tasks like interpreting chest X-rays, analyzing electronic health records (EHRs), and answering medical questions. Technical reports show that it achieves performance approaching that of much larger, specialized medical models while retaining the generalist capabilities of its base model.1
CodeGemma: Fine-tuned on over 500 billion tokens of code, CodeGemma is optimized for programming tasks. It supports code generation, completion, and a unique "fill-in-the-middle" capability that allows it to intelligently insert code between a given prefix and suffix.3
Other Variants: The ecosystem also includes models like DataGemma, which connects to Google's Data Commons to answer statistical queries; RecurrentGemma, which uses the novel Griffin architecture for highly efficient processing of very long sequences; and PaliGemma, a dedicated vision-language model for tasks like image captioning.1
5.3 A Chronological Release History
The following timeline illustrates the rapid evolution of the Gemma family since its inception.
February 21, 2024: Initial release of Gemma (2B, 7B).9
April 9, 2024: Initial release of CodeGemma and RecurrentGemma.9
May 14, 2024: Initial release of PaliGemma.9
June 27, 2024: Initial release of Gemma 2 (9B, 27B).9
July 31, 2024: Release of Gemma 2 (2B) and initial release of ShieldGemma.9
March 10, 2025: Release of Gemma 3 (1B, 4B, 12B, 27B) and ShieldGemma 2.9
May 20, 2025: Release of MedGemma (4B, 27B).9
June 26, 2025: Release of Gemma 3n (E2B, E4B).9
5.4 Concluding Analysis: Gemma's Role and Future Directions
The development trajectory and architectural choices of the Gemma family reveal a clear and consistent strategy. While competitors may focus on releasing ever-larger models to top performance leaderboards, Google's approach with Gemma is centered on maximizing performance within a highly efficient and accessible package. The strategic goal is not necessarily to have the single most powerful open-source model, but to offer the best family of models in terms of performance-per-watt and accessibility.
This focus is evident in every architectural evolution. The use of Multi-Query Attention in the first small model, the introduction of hybrid and interleaved attention to manage long contexts efficiently, and the development of selective parameter activation in Gemma 3n all prioritize the reduction of memory footprint and the acceleration of inference speed. This makes Gemma particularly attractive for real-world applications where cost, latency, and hardware constraints are primary considerations.
Furthermore, the rapid release of specialized, fine-tuned variants like MedGemma and CodeGemma, coupled with a strong emphasis on responsible AI tools, demonstrates a commitment to enabling developers to build practical, safe, and valuable applications. The future of Gemma will likely see a continuation of this trend: further advancements in architectural efficiency, deeper specialization into new domains, and a continued focus on empowering the global developer community to build responsibly on an open, state-of-the-art foundation.
Works cited
Use Gemma open models | Generative AI on Vertex AI - Google Cloud, accessed July 17, 2025, https://cloud.google.com/vertex-ai/generative-ai/docs/open-models/use-gemma
Gemma models overview | Google AI for Developers, accessed July 17, 2025, https://ai.google.dev/gemma/docs
What Is Google Gemma? | IBM, accessed July 17, 2025, https://www.ibm.com/think/topics/google-gemma
Gemma explained: An overview of Gemma model family architectures - Google Developers Blog, accessed July 17, 2025, https://developers.googleblog.com/gemma-explained-overview-gemma-model-family-architectures
Getting Started with Gemma Models - DEV Community, accessed July 17, 2025, https://dev.to/ifihan/getting-started-with-gemma-models-36g8
Get started with Gemma models | Google AI for Developers - Gemini API, accessed July 17, 2025, https://ai.google.dev/gemma/docs/get_started
Gemini (language model) - Wikipedia, accessed July 17, 2025, https://en.wikipedia.org/wiki/Gemini_(language_model)
Difference between Gemma and Gemini - Marvik - Blog, accessed July 17, 2025, https://blog.marvik.ai/2024/07/03/difference-between-gemma-and-gemini/
Gemma releases | Google AI for Developers, accessed July 17, 2025, https://ai.google.dev/gemma/docs/releases
google/gemma-3n-E4B-it-litert-preview - Hugging Face, accessed July 17, 2025, https://huggingface.co/google/gemma-3n-E4B-it-litert-preview
Gemma 3: Google's new open model based on Gemini 2.0, accessed July 17, 2025, https://blog.google/technology/developers/gemma-3/
Gemma 3 Technical Report - arXiv, accessed July 17, 2025, https://arxiv.org/pdf/2503.19786
Gemma: Introducing new state-of-the-art open model by Google | by Shravan Kumar, accessed July 17, 2025, https://medium.com/@shravankoninti/gemma-introducing-new-state-of-the-art-open-model-by-google-caae9fe29972
Google Gemma AI Models: A Developer's Guide - Collabnix, accessed July 17, 2025, https://collabnix.com/google-gemma-ai-models-a-comprehensive-technical-analysis-and-implementation-guide-for-developers/
Gemma explained: What's new in Gemma 3 - Google Developers Blog, accessed July 17, 2025, https://developers.googleblog.com/en/gemma-explained-whats-new-in-gemma-3/
Gemma: Open Models Based on Gemini Research and Technology - arXiv, accessed July 17, 2025, https://arxiv.org/html/2403.08295v1
Exploring Multi-Head Attention: Why More Heads Are Better Than One | by Hassaan Idrees, accessed July 17, 2025, https://medium.com/@hassaanidrees7/exploring-multi-head-attention-why-more-heads-are-better-than-one-006a5823372b
Grouped Query Attention (GQA) vs. Multi Head Attention (MHA): LLM Inference Serving Acceleration - FriendliAI, accessed July 17, 2025, https://friendli.ai/blog/gqa-vs-mha
What is grouped query attention (GQA)? - IBM, accessed July 17, 2025, https://www.ibm.com/think/topics/grouped-query-attention
Attention Variations — MQA vs GQA vs MHA vs MLA | by VerticalServe Blogs - Medium, accessed July 17, 2025, https://verticalserve.medium.com/group-query-attention-58283b337c65
GQA: Training Generalized Multi-Query Transformer Models from ..., accessed July 17, 2025, https://arxiv.org/pdf/2305.13245
[D] Why do GLUs (Gated Linear Units) work? : r/MachineLearning - Reddit, accessed July 17, 2025, https://www.reddit.com/r/MachineLearning/comments/1b6ggpz/d_why_do_glus_gated_linear_units_work/
Gemma 3 model overview | Google AI for Developers - Gemini API, accessed July 17, 2025, https://ai.google.dev/gemma/docs/core
Gemma 3 Technical Report - arXiv, accessed July 17, 2025, https://arxiv.org/abs/2503.19786
gemma3 - Ollama, accessed July 17, 2025, https://ollama.com/library/gemma3
Gemma - Hugging Face, accessed July 17, 2025, https://huggingface.co/docs/transformers/model_doc/gemma
www.techtarget.com, accessed July 17, 2025, https://www.techtarget.com/searchenterpriseai/definition/Gemma#:~:text=Running%20Gemma%20on%20GKE%20enables,GPUs%20and%20Google%20Cloud%20TPUs.
Gemma model fine-tuning | Google AI for Developers, accessed July 17, 2025, https://ai.google.dev/gemma/docs/tune
Workshop: How to Fine-tuning Gemma - Colab - Google, accessed July 17, 2025, https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Workshops/Workshop_How_to_Fine_tuning_Gemma.ipynb
A Beginner's Guide to Fine-Tuning Gemma | by Adithya S K | Medium, accessed July 17, 2025, https://adithyask.medium.com/a-beginners-guide-to-fine-tuning-gemma-0444d46d821c
Transformers from Scratch - DL - Kaggle, accessed July 17, 2025, https://www.kaggle.com/code/auxeno/transformers-from-scratch-dl
LLM Foundations: Constructing and Training Decoder-Only Transformers - Medium, accessed July 17, 2025, https://medium.com/@williamzebrowski7/llm-foundations-constructing-and-training-decoder-only-transformers-bfcc429b43a2
Transformer Implementation in PyTorch: "Attention is All You Need" - GitHub, accessed July 17, 2025, https://github.com/SwastikGorai/transformers_from_scratch
Implementing Transformer Decoder Layer From Scratch - Sanjaya's Blog, accessed July 17, 2025, https://sanjayasubedi.com.np/deeplearning/transformer-decoder/
LLM Evals and Benchmarking – hackerllama - GitHub Pages, accessed July 17, 2025, https://osanseviero.github.io/hackerllama/blog/posts/llm_evals/
Battle of the SLMs: Gemma vs LLama - Embedl, accessed July 17, 2025, https://www.embedl.com/knowledge/battle-of-the-slms-gemma-vs-llama
Which Gemma version is the right one for you? - YouTube, accessed July 17, 2025, https://www.youtube.com/watch?v=qcjrduz_YS8&pp=0gcJCfwAo7VqN5tD
Gemma 2 vs Llama 3: Which Model Is Better for You in 2024? - Novita AI Blog, accessed July 17, 2025, https://blogs.novita.ai/gemma-2-vs-llama-3-which-model-is-better-for-you-in-2024/
Mistral 7B vs. Llama 3 70B vs. Gemma 2 9B: A Comprehensive Benchmark Showdown | by Samir Sengupta | Medium, accessed July 17, 2025, https://medium.com/@samir20/mistral-7b-vs-llama-3-70b-vs-gemma-2-9b-a-comprehensive-benchmark-showdown-9c3128f24b23
Gemma 3 4B vs Mistral Large 2 - LLM Stats, accessed July 17, 2025, https://llm-stats.com/models/compare/gemma-3-4b-it-vs-mistral-large-2-2407
Gemma 3 12B vs Mistral Large 2 - LLM Stats, accessed July 17, 2025, https://llm-stats.com/models/compare/gemma-3-12b-it-vs-mistral-large-2-2407
Open Source LLM Comparison: Mistral vs Llama 3 - PromptLayer, accessed July 17, 2025, https://blog.promptlayer.com/open-source-llm-comparison-mistral-vs-llama-3/
MedGemma: Our most capable open models for health AI development - Google Research, accessed July 17, 2025, https://research.google/blog/medgemma-our-most-capable-open-models-for-health-ai-development/
MedGemma Technical Report - arXiv, accessed July 17, 2025, https://arxiv.org/html/2507.05201v1