EmbeddedKittens: An Evaluation of Code Embeddings for Scratch
This paper explores how to use machine learning to analyze and improve programming education using Scratch, a popular visual programming language for beginners.
This video presentation explains the key concepts from the paper in plain language.
Content & Liability Disclaimer
This article and its accompanying video are automated summaries derived from the original research paper by Unknown authors. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.
The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.
This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.
- 1 To investigate the influence of the natural language for RQ2, we made use of two datasets.
- 2 This paper extends our prior work which evaluated the applicability of only the code2vec embedding model to Scratch.
- 3 We evaluate their performance in comparison to four different pre-trained llms and collect 10 Scratch program datasets each using a different primary natural language.
- 4 We evaluate fine-tuning strategies that adapt pre-trained sprite naming models to be used as classifiers for whole programs.
Research Question
To investigate the influence of the natural language for RQ2, we made use of two datasets.
Methodology
The English-default assumption for keywords and method names can be difficult for international learners. The benchmark of method-level code documentation generation is not feasible in Scratch.
Study Design
Similar challenges arise for method or variable naming tasks since these concepts are rarely used in publicly available Scratch projects.
The structure of Scratch projects allows for the construction of an alternative task.
The choice of sprite naming as an evaluation task highlights another threat to external validity, since by itself it is of limited relevance to the educational community.
Results & Findings
The type of programs found in Scratch differs from those in other languages. This contrasts with the publicly shared open-source code found on GitHub.
- The type of programs found in Scratch differs from those in other languages.
- This contrasts with the publicly shared open-source code found on GitHub.
- Existing code embedding model approaches need to take the differences in the natural languages used by programmers into account.
- Large model training datasets can be created by mining publicly shared Scratch projects since all sprites must be named by the programmer.
- We evaluate potential applications of the models for education-focussed applications based on this foundational evaluation.
Since the meteor metric mainly uses unigrams like bleu, the repeated subtokens of GGNN have a limited impact, especially since the meteor metric is weighted to favour recall over precision .
Whole Scratch projects are also unlikely to be part of the training data, since downloading the original program-json from the Scratch api is a rate-limited two-step process, the result of which would then have to be converted to ScratchBlocks format.
How PDFdigest Helps You Understand Research
Instant Paper Analysis
Get structured summaries and key findings from dense PDFs in seconds.
Visual Explanations
Turn complex methods, figures, and results into clearer visual breakdowns.
AI-Powered Q&A
Ask focused questions and get answers grounded in the paper.
Practical Applications
The assumption that code can be represented similar to natural language text might not hold in these environments. This might not be sufficient to capture common Scratch language constructs since the relations between event emitters and receivers are lost.
Models additionally capturing control-and data-flow information might exhibit better performance on Scratch-based tasks.
Scratch adapts the order of the inputs to embed them as far as possible into the natural grammatical structure of the block text in the user’s language settings.
This might be relevant in both large-and small-scale deployments where the required resources to host an llm are not available or externally hosted tools cannot be used due to the special care required when working with children’s data.
Background
The paper discusses the significance of Scratch as a popular programming language for education, emphasizing its block-based structure and the need for adapted code embedding models for effective learning analytics.
Code Embeddings
Code embedding models are explained as tools that map source code into dense vector representations, capturing semantic and syntactic aspects. The shift towards using large language models for code-related tasks is discussed.
Transforming Scratch Code Into the Model Input Formats
The paper outlines the necessary preprocessing steps to convert Scratch’s visual code into textual formats suitable for various machine learning models, detailing the specific input formats required by different embedding approaches.
Figures Explained
The paper’s visual material highlights the workflow and the main system components.
- Fig. 1: Example of Scratch code and its abstract syntax tree (AST).. Illustrates the structural differences between Scratch code and traditional text-based programming languages, highlighting the need for adapted embedding models.
Limitations and Cautions
A useful limitation and caution is that this article summarizes the available paper text and extracted evidence; readers should consult the source paper before treating any interpretation as definitive.
The paper’s conclusions may depend on its source selection, definitions, assumptions, and the scope of its analysis, so follow-up reading is important.
Conclusion
The unseen specially crafted token representation described in the previous section might therefore not be suitable for llms. Therefore, we only connect the last statement in a procedure back to its declaration.
In our final two research questions, we therefore investigate whether the code embeddings are sufficiently expressive to support more education-focussed tasks, such as predicting program correctness or exercise progress:.
Unlike other general-purpose transformer models, it has therefore been designed specifically for code-related tasks.
Frequently Asked Questions
Custom variables, methods, and sprite names are allowed to contain arbitrary Unicode characters. To investigate the influence of the natural language for RQ2, we made use of two datasets.
We train five different code embedding models on the sprite naming task to evaluate their applicability to Scratch. Earlier code embedding models were trained to solve specific tasks like suggesting method or variable names or generating suitable documentation.
This paper extends our prior work which evaluated the applicability of only the code2vec embedding model to Scratch. We evaluate their performance in comparison to four different pre-trained llms and collect 10 Scratch program datasets each using a different primary natural language.
This might not be sufficient to capture common Scratch language constructs since the relations between event emitters and receivers are lost. In our final two research questions, we therefore investigate whether the code embeddings are sufficiently expressive to support more education-focussed tasks.
Whole Scratch projects are also unlikely to be part of the training data, since downloading the original program-json from the Scratch api is a rate-limited two-step process, the result of which would then have to be converted to ScratchBlocks format (e.g. using.
This paper explores how to use machine learning to analyze and improve programming education using Scratch, a popular visual programming language for beginners.