PROPER ERRORS

Errors are the one thing that I knew were terrible but I could never build up the courage to fix.
So I just fixed them.

Now all errors in the language are formatted nicely, show you exactly where the error happened and give you a nice description.

Along with this, I made a new system for collecting errors, so now the compiler won’t stop at the first error and will continue going and give you all errors at once. This makes fixing code a lot easier.

This also opens up the possibility of an LSP server later on
(before and after in images)

More demos and QOL

Try it out: https://scratch.mit.edu/projects/1387804691/

Demos

• Math lexer, parser and interpreter: New demo that lets you input math expressions (eg: 1 + 1) and get the result.
• 3d cube renderer: a spinning 3d cube and player controls

QOL

• Smarter inlining cost calculation
• A math centered optimization for removing identities
• Get rid of PromoteToGlobals (temp)
• GC documentation
• Codegen fixes (receiver functions generate the same block, causing scratch to get confused)
• Stack size optimization (reuse stack slots of the same type)
• Fix crash when the garbage collector is disabled

Demo programs and general improvements

I added demo programs to scratcher, these show the language actually being used

Try one out! https://scratch.mit.edu/projects/1385339395/

General improvements

• Bug fixes
• Dummy sprite
• Make the type checker ignored in inlined functions
• Make topLevelInit smaller and remove unused functions from final compilation
• TurboWarp specific compilation mode with faster operations on strings/lists (not compatible with vanilla scratch if enabled)

Indexed repeat and general improvements

• Added a new indexed for and repeat statement which allows way easier index tracking:

//Before:
int index = 0;
repeat(10) {
    ...

    index++;
}

//After:
repeat(10) { index -> 
    ...
}

I also made the PooledList get garbage collected with the refcount gc
Did some really small refactors aswell

List pools

Static lists are cool, but they are very limited expressiveness wise because they must be known at compile time

Pools fix this by having a pool of static lists that the user code can allocate at any time

PooledList a = allocate();

But because the pool is limited, the pooled list must be freed when not needed, the garbage collector will not handle it(yet)

a.free();

Other than that, the PooledList behaves just like a List and can do almost anything a List does

• I also added proper documentation for Out of memory errors.

QOL

I added a shit ton of quality of life stuff:

• Tons of bug fixes
• Big array syntax:

int[] myArray = [1, 2, 3, 4, 5, 6];

• Safe casting for enums(previously a panic):

Result.Success? a = res as? Result.Success;

• Out of memory detection: Scratch caps lists at 200k elements, after that it silently stops adding stuff to the list
• Compact int list: Store pretty much infinite integers without wasting heap space
• SHORT CIRCUITS: DONT EXECUTE RIGHT SIDE OF && or || if left side is false/true!
• If statements without braces:

if(a) b++;

• CONTINUE AND BREAK:

while(true) {
     if(a) break;
     if(b) continue;
}

• STATIC LISTS: store data outside of the heap in scratch lists so that you dont run out of heap space:

static str[] myList;

This also allows for file reading:

static str[] myList = "hello.json";

So you can properly store data(like ROMs)

• CASE SENSITIVE EQUALS:
  DID YOU KNOW THAT SCRATCH “=” BLOCK IS CASE INSENSITIVE??? And its pretty hard to make it case sensitive
  Now you can do “===” and its case sensitive:

'a' === 'A' 
RETURNS FALSE

DOCUMENTATION

I do not want to touch any markdown ever again

Privates

Added a “private” keyword, anything declared private cannot be accessed from outside the current file.

This prevents the user from messing with internal states of libraries they are using

For example, the list library from the standard library.
Now you can’t directly mess with the list pointer

triangles.ptr[5] = ...;

This now results in an error because “ptr” is private!
I also added some small optimizations for lambdas.

Operator overloading

Added operator overloading, now you can define overloads for specific operators so that code looks cleaner

Just use the operator keyword with the correct signature and you’re done!

You can overload the operators: +, -, /, *, %, [], []=
For example:

warp operator <T> void T[].set(int index, T item) {        
     self::replace(this, item, index);
}

//use
array[index] = item

//compiler translates to
set(array, index, item)

I also realized that my previous refactor of arrays broke my nullability system:

int[] a = Array(int, 5);
say(a[3]);

This would print null, which my type system missed

I fixed this by moving arrays to a standard library module, the same array import works, but you have to change how you create the array:

//before:
int[] a = Array(int, 5); //doesn't need to be marked as null!!!

int[] a = Array(int, 5);
a[0] = ...;
a[1] = ...;
...

//after
int?[] a = arrayOfNulls(5); //null filled

int[] a = arrayOf(5, (int index) -> {
     return index * 5; //value of the array at that index
});

Import some

import names into unqualified co ntext

Before:

//import
import list;

//use
list::List<str> myList = list::newList();

After:

//import
import list::*;

//use
List<str> myList = newList();

Pattern matching for sealed enums

Previously if you wanted to check sealed enums types and use them, you had to do very ugly syntax for actually getting the variant instance:

auto out = tryGet();
when(out) {
    Result.Success -> {
        auto suc = out as Result.Success;
        looks::say("Success! ${suc.out.x}");

Now you don’t have to do all of this:

when(tryGet()) {
    Result.Success suc -> {
        looks::say("Success! ${suc.out.x}");

This is just syntax improvement, doesn’t change the generated code or anything, just helps the developer

ACTUAL ARRAYS!!

• Previously, scratcher “arrays” were just lists, you could add more items, it would dynamically resize, it was written in the kotlin DSL, so technically it was just a list.
• I removed the previous list code, reused some of it and added a new Array system
• Along with that, the normal lists got rewritten in 100% scratcher with no internal functions
• The new system is almost fully backwards compatible in source code, you just have to make a few changes

The new implementation is more clear, more expressive and more like regular programming languages

I’m sorry

For over 2 months, scratcher has had an “export” keyword, which was fully unused, until today:

I implemented the export keyword on functions, which allows for code written in plain scratch to interact with scratcher code:

export warp void myFunction() {  ... }

Export requires that your function arguments are all primitives and that the return is also a primitive, but other than that, it does not change anything, just exposes the function cleanly to regular scratch code.
However, scratcher type checking still applies, so if the native scratch code passes in a string to a float argument, it will still cause a panic

Even when you have obfuscation enabled, exports will stay and let native scratch use it with its original name

Sealed enums and faster GC

• I added a new type of object, sealed enums:

sealed enum Animal {
     Cat(str name, int age, bool isMeowing),
     Dog(str name, int age),
     Unknown
}

They allow for storing data inside enums, they’re very similar to rust enums:

sealed enum Result<T> {
     Success(T out),
     Failure
}

• I also refactored lambdas to use a sealed enum instead of a big struct:

//before
struct LambdaCaptures(Lambda0Captures? c0, Lambda1Captures? c1, Lambda2Captures? c2);

//after
sealed enum LambdaCaptures {
    Lambda0Captures(Box$int capture0, ...),
    Lambda1Captures(Box$float capture0, ...)
}

Sealed enums only store a tag and a pointer, so if you use a lot of lambdas, this will save a TON of heap space

• Faster garbage collector!
  I made the marking phase of the mark and sweep collector faster by using an epoch based cleanup approach
  Now, instead of clearing and re-populating the marked list, we don’t ever clear it and we invalidate it by just increasing the epoch.
  This makes the isMarked check go from O(n) to O(1) while also improving marking speed because scratch doesn’t have to resize our list

MORE FUCKING ABSTRACTIONS!!!

I added receiver/extension functions. These allow you to write code that takes in a type as a special argument
You define them just like how it works in kotlin:

warp <T, R> R T.let((T) -> R action) {
    return action(this);
}

It just compiles to a regular function

warp <T, R> R let(T this, (T) -> R action) {
    return action(this);
}

But the compiler lets you call it with the special argument:

thing.let((int a) -> ...);

I also added a very big standard library module (written in scratcher!) with a ton of these functions built-in, they allow you to write more expressive code:

triangles
     .filterNotNull()
     .filter((Triangle<float> t) -> t.x1 > 150)
     .filter((Triangle<float> t) -> t.y1 < 150)
     .forEach((Triangle<float> tri) -> ...)

I also rewrote the entirety of string lib during this, now you have to use the extensions library to use strings, but its way more powerful (and written in scratcher)!
I also added a new “char” type, which stores a single character (wow really???)
I also added “safe dot calls” that allow you to use stuff as if they’re not null:

triangles.forEach((Triangle<float>? t) -> {
    t?.x1?.let((float x1) -> {
        looks::say("x1: ${x1}");    
    });
});

This only calls the let function if t?.x1 is not null

There is a big problem with all this powerful new code though… the garbage collector can no longer keep up, a single mark and sweep takes over 1 second! Which is especially bad because the garbage collector needs the entire application to stop in order to run. You do not want your code to completely stop for a whole second(maybe you do? weirdo), so I have to fix this in my next few devlogs…

Lambdas

I added lambdas to Scratcher!
They allow you to write more expressive code like map(list, a -> a + 1), which literally adds 1 to every element of list
I also allow using outer variables within lambdas

int a = ...;    
map(list, b -> {    
     a++;    
     b + a    
});

I allow this through boxing, the local variables become boxes and the lambdas get the local variable’s box, this makes lambdas more powerful than most languages as this allows for variables to be shared mutable state across lambdas, this also allows the local variables to get garbage collected.

Of course, scratch doesn’t support this natively, so lambdas are 100% compiler sugar and just lowers to regular functions and uses function references(which is also 100% compiler sugar).

void lambda::defunc@lambad@0(LambdaCaptures c, args...) {  
  ...  
}

Fixed the garbage collector… turns out I broke it accidentally while cleaning up code in my last devlog, 3 line change but took a long time to find

Made it so the stack allocator doesn’t do redundant operations like 1+1 and inlines them as 2 instead

100 hours!

Evil hack part 4: Death of evil hack

So I was using a really terrible hack to implement stuff like
FunctionInlining, Safe dot and the elvis expression, the hack is
basically (ab)using the when expression to do stuff like run code inside
expressions and use local variables inside expressions.

So to stop this from becoming a hack, I added it directly to the AST as
StatementExpression(not to be confused with ExpressionStatement), which
can be used to contain statements inside expressions.

All of my evil hacks have been replaced by StatementExpression.

I also added some small optimizations for StatementExpressions!

General language improvements

• Elvis expressions:
  If a value is nullable, it is not usable by anything that only accepts non-null values, to solve this, I added the elvis expression, which allows you to give a fallback for null values(compilation in image 2):

//int doesn't have to be nullable because we have a fallback value!
int x = functionThatReturnsNull()?: 5;

• String boxing and nullable primitives
  Previously, scratcher only supported having structs, enums and function references as nullable, now it also supports having primitive types as nullable:

str? x = functionThatReturnsNullableString();

//distinguish between literal null and string null
if(x == null) looks::say("null!");
if(x == "null") looks::say("null as string!");
//the compiler achieves this by boxing the string return with a struct:
//struct StringBox(str value)
//the actual type behind the scenes is:
//StringBox? x = ...;

• Safe dot traversal
  If a struct instance is nullable, you cannot use regular struct.member because that may result in a NullPointerException, to solve this I added ?. which allows for null safe traversal(compilation in image 1):

//if nullableTriangle == null, return null
//if nullableTriangle != null, return nullableTriangle.x1
int? x1 = nullableTriangle?.x1;

• Deepseek fix
  I noticed that FunctionInlining completely ignored execution order, so I asked deepseek to solve it(this happened after I added string boxing).
  But I found a better solution to the same problem later on, so I reverted deepseek’s fix and wrote my own, which is 260 lines compared to deepseek’s 360 lines, over 100 lines shorter!

Second garbage collector!

Added a second garbage collector! If your app does not contain any cyclic objects and needs high performance, you can now enable the ARC collector!

This garbage collector works by automatically detecting when objects go out of scope and decrementing their reference count, if an object’s reference count goes to 0, it is freed.

This is more performant than the mark and sweep collector because the objects are freed at the moment they are unused instead of stopping the entire program to collect garbage.

This works for all cases, except for cases where there is a cyclic object. Because the cyclic object owns itself, so its reference count is always at 1.