Fixed Fragment shaders broken with Turkish locale on Android builds

[Timur]

This error occurs when running on an Android target (General Mobile G512, Android 11, Locale: Turkish)

This error is called once a new canvas has been made:

04-15 00:37:56.381 12746 12785 I [Cerberus]: Failed to compile fragment shader:0:4: P0001: Invalid identifier 'B3D_AMBİENT'

As the locale of the device is in Turkish, it uses İ (I with dot above) instead of "I" for the capital of "i".
I have examined the code and the way to fix it is to add

.Replace("İ", "I")

to line 773 and 775 on

modules/mojo2/graphics.cxs

However, this might not be a priority but considering the problems it could create possible problems on Android devices running with a Turkish locale.
I'll be trying to creating a pull request on GitHub related to this.

 

[Phil7]

Good catch! Did you try to look at the shader code itself and try to fix the spelling there? Or is this a common problem with Turkish locale?

 

[Timur]

Good catch! Did you try to look at the shader code itself and try to fix the spelling there? Or is this a common problem with Turkish locale?

Well, what I did was just trying to run an on my phone, but the app would instantly crash. Upon checking on the console I saw that it couldn't find the said shader. Confusing I and İ in computer apps are common, for example the Chrome search function wouldn't accept an "i" to search for "I" (for example, to find "MARIO" on a page, you'd have to type it in uppercase or as "marıo" for it to find)
After that, I just looked a bit down and started examining the code and we'll, found that replace should be added.

 

[MikeHart]

What I don’t get is why the I is changed to the i in the first place. Do they share the same ASCII code? I would not think so.

 

[Phil7]

I had a look and I think the reason for this to happen is that "b3d_ambient" is transformed into "B3D_AMBIENT" with String.ToUpper()
This method works language specific which leads to "İ" instead of "I" because the former is the correct capital letter of "i" in Turkish.
The fix IMHO should use a ASCII compliant .ToUpper Function and make sure only ASCII characters are used for those identifiers.
Another approach could be a different way of mangling for these identifiers that doesnt involve capitalisation at all.

 

[dawlane]

Turkish is one of those languages that are a bit of a headache. There are also a few other languages that would have the same issue.
A search with unicode to ascii Turkish locale should show a fair few items on the subject. But as this is Android specific, then it's a java issue that needs to be addressed. This specific issue with the `i` character is mentioned in the Android docs. https://developer.android.com/reference/kotlin/java/util/Locale

 

[dawlane]

After a bit of a think. Phil's idea of a function my be the way to go, but instead of a dedicated ascii compliant ToUpper, which would then need the equivalent ToLower. I would suggest more in the line of functions to deal with getting and setting locales.

In this way, CX internals and get and store the current local, switch to a more suitable locale and then restore it after the operation is completed.

Probably the place to make a start into researching this.

Edit: This may also provide some insights.

 

[dawlane]

@MikeHart
Right I have a solution to the locale issue.
As the glsl files are using English, only those files need to be processed.

Before making any changes. Create an AVD with Android Studio, start it and set it's locale to Turkish.
Compile and run the Mojo2 shadereffect example. It should fail because it cannot find the identifier b3d_ClipPosition.

The changes:
Start by creating a new file in the modules/mojo2 called locales.cxs and add this code.

#If TARGET="android"
Import "native/locales.${LANG}"
Extern
Class Locales
    Function ToUpperASCII:String( str:String )="toUpperASCII"
    Function ToLowerASCII:String( str:String )="toLowerASCII"
End
#Endif

Now create a new directory called native in modules/mojo2 and then inside this directory create a new file called locales.java with this code.

import java.util.Locale;

class Locales{
    static public String toUpperASCII( String str ) {
        return str.toUpperCase(Locale.ENGLISH);
    }
 
    static public String toLowerASCII( String str ) {
        return str.toLowerCase(Locale.ENGLISH);
    }
}

Now the changes to the file graphisc.cxs file in modules/mojo2
Open the file and at the top, import the locales module.

#If TARGET="android"
Import locales
#End

Now the change to the problem code by replacing the if/Endif block at line 771 with

            If id
#If TARGET<>"android"
                If id.StartsWith( "gl_" )
                    vars.Insert "B3D_"+id.ToUpper()
                Else If id.StartsWith( "b3d_" )
                    vars.Insert id.ToUpper()
                Endif
#Else
                If id.StartsWith( "gl_" )
                    vars.Insert Locales.ToUpperASCII( "B3D_"+id )
                Else If id.StartsWith( "b3d_" )
                    vars.Insert Locales.ToUpperASCII( id )
                Endif
#Endif
                Continue
            Endif

Once done. Delete the Cerberus Game from the AVD and clean out the shadereffect.build folder, then rebuild and run.

At some point in the future. The Locales class should be created into a full public module for those targets that can have issue locales.

 

[Phil7]

@dawlane I am absolutely fine with a module for those issues in other places, where localisation is necessary. In this specific case for mojo2.graphics I really prefer something simple and reduced to ASCII. Basically "if a_to_z subtract a-A, leave the rest alone".
The main reasons for me are that I don't have an overview of how letters are matching lower case and upper case. But I know that there are definitely issues with unicode just by looking at the few languages I know. In addition to this I don't want to risc performance issues with those localized implementations. They might be well optimized but they still have to cope with a much more complicated job to get the localisation and the amount of symbols right.
The downside is that we limit GLSL identifiers to ASCII, which is not in the standard but recommended style AFAIK. To me this is not a big issue as Cerberus identifiers are limited to this as well.

 

[dawlane]

Basically "if a_to_z subtract a-A, leave the rest alone".

Wouldn't be as simple as that. Android OS deals with setting the locales of an application, unless the application it's self override the users locale. And that from all accounts of what I've read up to press, that can causes it's own problems.

Could possibly try to use Constant string "A_to_Z" as a look up. Then parse the id string. And if a character is in the range a_to_z; subtract the character with the ASCII value of `a` to get the index of the constant look up.

Edit:
Ignore the above.

 

[dawlane]

Two additional ways. One is the Cerberus way. And the other is doing it the java way.
As using ToUpper (which gets mapped to the standard java string method toUpperCase(), that in turn calls toUpperCase(Locale.getDefault())) messes things up with the Android OS getting in the way (that is unless you specifically use the locale version of toUpperCase) and so cannot be used. So a dedicated method/function would have to be used to do the conversion. That is providing nobody writes any more glsl shader scripts out side of the Cerbeus IDE using their own locale and a different uni-code format.

The Cerberus way. Which could be changed from using ToChars to an actual array and doing a character-by-character copy:

Method ToUpperASCII:String( str:String )
    Local s:=str.ToChars()
    For Local i:=0 Until s.Length
        If (s[i]>=97 And s[i]<=122) s[i]~=$20
    Next
 
    Return String.FromChars( s )
End

The java way. Which would have to be bound similar to the locale that I posted earlier.

class ASCII{
    static public String toUpperASCII( String str ) {
        char[] s = str.toCharArray();
        for (int i = 0; i < s.length; i++) {
           if(s[i]>=97 && s[i]<=122){
               s[i] ^= 0x20;
            }
        }
        return new String( s );
    }
 
}

To determine which of the methods from all that I've posted would be the better in performance. That would require that the examples be compiled and loaded into Android Studio for testing and profiling with real hardware.