Fixed CerberusX reporting incorrectly number of line.

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
Could you explain, what the but is causing and in which files to look for the problem .
The two files are the preprocessor and toker (aka lexer). Both work in conjunction with each other.

As it stands, it cannot handle nested preprocessor directives correctly within remark blocks.
You wouldn't be able to do something like.
Cerberus:
#Rem
    You can use the preprocessor directive below to start selective compilation. You must terminate any conditional blocks with #End
    #If CONFIG="debug"
   
    The CONFIG preprocessor variable can be either debug or release.
#End

Function Main()
    Print "END"
End

You would get the Error: Function 'Main' not found as the preprocess is still pulling in the #If conditional.

The current method I was trying was just through the lexer only as this is called first, but I have an idea that may work that I will try tomorrow with a fresh mind and set of eyes.

Would it still be possible to nest rem blocks, if you change #end to #endRem?
If you are asking can you do this:
#rem
#rem
#endrem
#endrem

Then yes, all you have to do is on the first #rem set a flag to ignore everything and count the nesting's of the rems/endrem blocks only.
When the count is zero then turn the flag off.
 

MikeHart

Administrator
Joined
Jun 19, 2017
Messages
3,210
That would cause some incompability with 3rd party stuff but i would be ok with it. But then we should change also #if #end to #if #endif.
And the regular stuff that ends with END.
CX2 here we come.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
That would cause some incompability with 3rd party stuff but i would be ok with it. But then we should change also #if #end to #if #endif.
And the regular stuff that ends with END.
Messing with #end/#endif would make it really bad. Any new cx should keep backward compatibility with the old language, or you would end up with another mx2.

I may have a solution that could get round it by doing a little extra lexing so that a remark block can use #End and also ignore and preprocessor directives within. The solution would be to just terminate a remark block with #End Rem if a preprocessor conditions directive is detected.
 

Rich

Well-known member
CX Code Contributor
3rd Party Module Dev
Tutorial Author
3rd Party Tool Dev
Joined
Sep 9, 2017
Messages
451
Just as an idea.. I had a look at preprocessor yesterday and can see the case statement for rem. Couldn't a loop go in there to only exit when an end has been detected? This means it also wouldnt count lines and new lines.. it might mean that nested rem blocks wouldn't work (unless ident is also counted)

I really don't like the idea of endif
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
Just as an idea.. I had a look at preprocessor yesterday and can see the case statement for rem. Couldn't a loop go in there to only exit when an end has been detected? This means it also wouldnt count lines and new lines.. it might mean that nested rem blocks wouldn't work (unless ident is also counted)
You would still hit the same problem of which #End would be the terminator. But I can see a possibility of simplifying by cutting down a lot of conditional testing.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
Well those ideas failed. There is no way that I can think of that to get round the fact the #END preprocess directive is doing more than closing just preprocess conditional blocks. Looking through Marks code, not once I've I seen any conditional block, be it a preprocess directive or otherwise terminating with an End.

Side notes:
The preprocessor doesn't check test explicitly for and EOF token. So if there is an incorrect termination of preprocess directive blocks within a code body block, you should get an unexpected token error. That so far looks like it can be easily fixed.

The current github developer branch fails to build from source on Linux. Would help if only standard C++ functions are used and any platform specific code is filtered out. Last time I checked, Linux didn't have a MultiByteToWideChar function or icon images embedded in the executable.

Over the next day or so. I will post a more elegant fix to the preprocess and tokenizer with comments on how it works.

Messing with #end/#endif would make it really bad. Any new cx should keep backward compatibility with the old language, or you would end up with another mx2.
I think I've changed my mind on that Idea. All conditional blocks should be terminated correctly with a specific terminator. First it should make writing the lexer, error checking and reporting code easier. And Second it will make anyone not familiar someone else's code to able to follow what's going on when you have nested code blocks within nest code blocks etc.
 

MikeHart

Administrator
Joined
Jun 19, 2017
Messages
3,210
The current github developer branch fails to build from source on Linux. Would help if only standard C++ functions are used and any platform specific code is filtered out. Last time I checked, Linux didn't have a MultiByteToWideChar function or icon images embedded in the executable.
Yes you are correct. Sorry about this. I will need to be more careful what to except from PRs.
I think I've changed my mind on that Idea. All conditional blocks should be terminated correctly with a specific terminator. First it should make writing the lexer, error checking and reporting code easier. And Second it will make anyone not familiar someone else's code to able to follow what's going on when you have nested code blocks within nest code blocks etc.
Of course that would make life much easier.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
Yes you are correct. Sorry about this. I will need to be more careful what to except from PRs.
It may be a good idea to put a notice up about excepted coding standards when people are wishing to contribute code, especially for the core code.

eg.
Cerberus:
Please keep in mind that Cerberus is a cross platform language
and that Cerburus code and native code must use the languages preprocessor directive for
both Cerberus and the native target language.

All conditional code blocks using If/Else/Else If, including preprocessor code blocks must be terminated with `Endif`.
e.g.
#If CONFIG="debug"
    Print "HELLO DEBUG WORLD"
#Else
    Print "HELLO RELEASE WORLD"
    #If SOMECONFIGVAR="SOMEVALUE"
        Print "What?"
    #Endif
#Endif

Tabulation should also be used within non conditional as well other types of
conditional code blocks e.g.
Select StarTrek
    Case TNG
        If Riker
            Print "Make it so Number One"
        Else
            Print "Make it so"
        Endif
    Default
        Print "There's Klingons on the starbord bow Jim"
End

These types of blocks are permitted to use `End`. Or for better readability,
`End` with the use the opening statement of the code block. Especially if they are going to be nested.
e.g.
Select StarTrek
    Case TNG
        If Riker
            Print "Make it so Number One"
        Else
            Print "Make it so"
        Endif
    Default
        Print "There's Klingons on the starbord bow Jim"
End Select
 
Last edited:

MikeHart

Administrator
Joined
Jun 19, 2017
Messages
3,210
It may be a good idea to put a notice up about excepted coding standards when people are wishing to contribute code, especially for the core code.

eg.
Cerberus:
Please keep in mind that Cerberus is a cross platform language
and that Cerburus code and native code must use the languages preprocessor directive for
both Cerberus and the native target language.

All conditional code blocks using If/Else/Else If, including preprocessor code blocks must be terminated with `Endif`.
e.g.
#If CONFIG="debug"
    Print "HELLO DEBUG WORLD"
#Else
    Print "HELLO RELEASE WORLD"
    #If SOMECONFIGVAR="SOMEVALUE"
        Print "What?"
    #Endif
#Endif

Tabulation should also be used within non conditional as well other types of
conditional code blocks e.g.
Select StarTrek
    Case TNG
        If Riker
            Print "Make it so Number One"
        Else
            Print "Make it so"
        Endif
    Default
        Print "There's Klingons on the starbord bow Jim"
End

These types of blocks are permitted to use `End`. Or for better readability,
`End` with the use the opening statement of the code block.
e.g.
Select StarTrek
    Case TNG
        If Riker
            Print "Make it so Number One"
        Else
            Print "Make it so"
        Endif
    Default
        Print "There's Klingons on the starbord bow Jim"
End Select
Just noticed the End Select. If we change that, it should be a one word statement. EndSelect, EndIf.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
Just noticed the End Select. If we change that, it should be a one word statement. EndSelect, EndIf.
I haven't gone through the Syntax Analyser yet, but End Select, End Class, End Function, End Method work as they should as these are keywords and treated differently than the preprocessor directives in the Lexical Analyser (aka preparser/toker).

Edit: Plus `End Select` is much easier to read than the keywords being bunched up as EndSelect.
 
Last edited:

Phil7

Administrator
CX Code Contributor
3rd Party Tool Dev
Joined
Jun 26, 2017
Messages
659
Just noticed the End Select. If we change that, it should be a one word statement. EndSelect, EndIf.
I am also voting for this in the long run. We already had a discussion about what end statements are best for readability and usability.
In the docs I am going for what is possible with CS as it is and beeing consistent: EndIf, End (for Select), Next, Wend, End (for Class, Function, Method)
For now I think we should break as little code as possible by only changing the preprocessor to #endif.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
For now I think we should break as little code as possible by only changing the preprocessor to #endif.
I think you mean the removal of #End as a standalone terminator for everything. The preprocessor does permit `#End If`. Internally it get translated to endif.

In the docs I am going for what is possible with CS as it is and beeing consistent: EndIf, End (for Select), Next, Wend, End (for Class, Function, Method)
Code:
Function Main()
    WithCorrectEndTerm()
End

Function WithCorrectEndTerm()
    For Local x:=0 To 9
        For Local y:=0 To 4
            Print "WithCorrectEndTerm"
        Next x                            ' This will throw an error that the variables is miss match for the for loop. Swap the variables round so that x is y etc.
    Next y
   
End Function                            ' Any code block ending with a different terminator that doe not match the initiator will throw an expecting declaration error.
Code:
Function Main()
    Quirky()
End

Function Quirky()
   
    Select Rnd(2)
        Case 1
            Print "1"
        Case 2
            Print "2"
        Default
            Print "Default"
    End If                                ' This Should throw an error, but it doesn't until it hits another command and an error that has no relation to this error.
   
    Print "OK"
   
End Function
 
Last edited:

MikeHart

Administrator
Joined
Jun 19, 2017
Messages
3,210
OK preparser and toker for testing.
All code that I used to analyse the state of variables should have been removed.
Any preprocessor conditional directives within a remark block still have to be terminated correctly.
Hello dawlane,
thank you for your files. I see you put a lot of effort into this. So I tested your changes on Windows10 with just doing a F6 on all the examples.
The mojo2 section does not build at all, looks like an error with a missing #endif. But in the mojo1 section some of the examples don't have this problem and they can't be build. So I advice that you look at your changes again.

Example: These M.Sibly example show these problems.

mak/audiotest
mak/suspendtest

Regards
Michael Hartlef
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
Thank's Mike.
I'll look into it.

I was more transfixed on detecting broken code, that of I didn't take into account code that works. I didn't take into account multiple elseif preprocessor directives. I'll just keep a simple back trace for open preprocessor directives, by only keeping tract of those directives that increase the nesting count and not elseif. Though I may see if I can use two string stacks to keep track of those that increase nesting and elseif to merge the two later on the event of a EOF token.

A quick fix is to remove the `preptrace.Push _errInfo` out of the case of `elseif`.

Not sure why mojo2 isn't building. Have to double check that doing a chop and change and adding tabs to make M.Sibly code more readable isn't the cause of it.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
@MikeHart :
What were the exact errors for both mojo and jojo2. And which Cerberus build were you using?
I've just test the build using MinGW TDM 5.1.0 with the current develop branch, after removing the `preptrace.Push _errInfo` out of the case of `elseif`, and it works.
 

MikeHart

Administrator
Joined
Jun 19, 2017
Messages
3,210
Of course I was using my development version here, but I just tested this now with the 2019-10-13 release again. MingW TDM-GCC-64 version which we provide here.

mak/audiotest shows this:

1587217852559.png


The console shows this:

1587217913282.png


Looks like you have some debugging messages left there.

mak/suspendtest shows this in the console:

1587217961282.png


The mojo2 examples all crap out because they all depend on the opengles2 implementation.....

1587218121683.png


It comes down to not handling the #elseif or #else if statements correctly...

Anyway I hope I was of help.
 
Last edited:

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
It comes down to not handling the #elseif or #else if statements correctly...
It's because any #elseif was being pushed onto the same string stack and not taking into account that they will be needed to be popped off. Removing the offending `preptrace.Push _errInfo` out of the preprocessor section `elseif` select/case will only make note of the last `#If` directive.

Though I have found a weird bug that will be hard to trace with the string stack class, or the for loop. Push an array from another string stack isn't being updated correctly, or picked up when you iterate the modified stack. Only seems to do it in transcc. A stand alone C++ Tool works as it should. Could be a possible GC issue. I'll have to see if I can work round it and if not, only focus on the start of an `#if/#endif` block.
 

dawlane

Well-known member
CX Code Contributor
Joined
Jun 21, 2017
Messages
874
@MikeHart
OK. Think I have got it working with some back tracing/tracking.
The toker file should be OK. The issue was with the preprocessor file. Just back up the old working one and replace with to one in the zip file.
preprocessor.zip

Once it's confirmed that the affected files work. It's just a case of someone writing some code to see if it can be tripped up.
Note that #if/#else/#elseif/#end in remark blocks will still have to be terminated correctly. That is unless plain #end is dropped from preprocessor conditionals.
 

MikeHart

Administrator
Joined
Jun 19, 2017
Messages
3,210
@MikeHart
OK. Think I have got it working with some back tracing/tracking.
The toker file should be OK. The issue was with the preprocessor file. Just back up the old working one and replace with to one in the zip file.
preprocessor.zip

Once it's confirmed that the affected files work. It's just a case of someone writing some code to see if it can be tripped up.
Note that #if/#else/#elseif/#end in remark blocks will still have to be terminated correctly. That is unless plain #end is dropped from preprocessor conditionals.
Looks like you did it. Thank you.
 
Top Bottom