Publish the source code of all in-game widgets

[mvgulik]

...

[shubla]

I've posted about this issue previously, but I guess it bears repeating in one post.

The reason I put code into resources is because it has a number of advantages. Most importantly, it's incredibly convenient for me to be able to push small changes (some minor tooltip, some custom graphics) without having to do a full client update and boot people who haven't updated. In that way, it helps custom clients as well, since the client author doesn't have to merge changes for every other patch just for the client to remain usable.

I do deeply understand why it's a pain for client modders, however. Precisely because the kind of code I tend to put into resources is "on the edge", so to speak (it tends to be the visible part of many mechanics), it's exactly the kind of code that client modders would tend to be the most interested in, and also important to interact with for the kind of client modifications that I can even agree are the most legitimate of modifications. As such, I'd honestly want to provide a better way to interact with it, and I've spent some amount of time trying to think of one (to no success thus far). It is of course true, nonetheless, that it's not a problem that I'm personally afflicted by, so it's not the kind of thing that I spend every day thinking about. As such, if you have any good ideas for how to achieve a better system for handling these things, I'm interested. That being said, it would have to be something that preserves the advantages that I derive from the current system.

What exactly prevents you from adding these resource files with code in them to the git repository? It would be enough to expose the source code so custom client developers can then modify and copy the code as needed into their clients.

In a case that you have the code in a separate folder where all other resource files lie, it could be even another repository with some proper .gitignore to ignore everything else than the code sources. This kind of solution would also require minimal effort from you when updating the files.

It would be helpful if you told what your current process with the resource files that contain code is, so we could figure out if there would be some good solution to it, if you are too busy to come up with an one.

[loftar]

What exactly prevents you from adding these resource files with code in them to the git repository? It would be enough to expose the source code so custom client developers can then modify and copy the code as needed into their clients.

Since the source code is already available in the resources. I don't quite understand what the advantage of maintaining a separate Git tree with their source code would be.

It would be helpful if you told what your current process with the resource files that contain code is, so we could figure out if there would be some good solution to it, if you are too busy to come up with an one.

Not sure what you mean by "process". I just write the code alongside the other files that constitute the resource and it gets compiled into the resource along with those other files (meshes, textures, &c). The nice things about it are that 1) there isn't much of a process to it, and 2) the code gets automatically downloaded and updated by the client just by virtue of the higher version number on the resource.

[shubla]

What exactly prevents you from adding these resource files with code in them to the git repository? It would be enough to expose the source code so custom client developers can then modify and copy the code as needed into their clients.

Since the source code is already available in the resources. I don't quite understand what the advantage of maintaining a separate Git tree with their source code would be.

Yes it is technically available from the resources, but to access it I have to see considerable effort:
first I must find out the name of the resource that is responsible for some certain feature in the client, and see it so my client updates the resource
then I must decompile the resource file, now I must attach the resource file to my client code directory and do the possible modifications to it,
now if the resource ever changes in the future, my client will crash with some (possibly obfuscated) error message and I will have to debug for quite some time to find out where the issue is, and then find that resource file again from some folder.. and decompile it.. modify it and publish a new update... tedious.

Now think about this: what if instead of doing all the things above, I could just write "git pull" or something, and get the source code without digging around, decompiling etc. and instantly see if there is some change related to resource source file that my client uses. I want the file visible in my IDE without having to first write my own decompiler for resource files etc. etc.

If I understood correctly, your only motive for the current resource thing is that clients don't need to be explicitly updated if they have not modified the resource file, and you can keep some of the things "secret", to only appear when the player sees it. Of course after the first player sees them, they are not that secret anymore.

Even if you still don't completely understand what the advantage in having them in some more easily available form is, you should still do something, as it would greatly ease development and maintenance of custom clients.
The preferred way would be of course to have them in the git like all the other files of the client. I'm sure that there is some way where you could still update them via resource files without updating the whole jar-file if you make some small change to them.

[shubla]

What exactly prevents you from adding these resource files with code in them to the git repository? It would be enough to expose the source code so custom client developers can then modify and copy the code as needed into their clients.

Since the source code is already available in the resources. I don't quite understand what the advantage of maintaining a separate Git tree with their source code would be.

I would certainly like to know how am I supposed to get the source code from them in some not-so-painful manner? Decompiling is similar effort in my opinion.

This is the method that I've been using so far.

1. Grep files from folder which caches all resource files until you find the correct one (may take a while)
2. Notice that the code depends on another class which is in another file so often go back to 1.
3. Open the resource with vim, find these oddly separated sourcecodes and copypaste them to client, resulting in the same as they'd originally been in the upstream git repo on the first place
4. Hope that they never change in the future in the server side, because that will break everything and the process must be done again

Very very tedious. One option is of course to write some kind of script that automates all this but oh well... Trying to interpret some custom format that the source codes are in is tedious also. Cannot understand why you just can't put them in github!

Is this some standard format ? What does "/* $use: ui/tt/q/buff */" for example mean, or "> tt: Quality"

[Aoyama]

Mr Loftar,

What is the single byte before the name in the "src" layer?
It's driving me nuts.

Code: Select all

    @LayerName("src")
    public class Source extends Layer
    {
        public String name;
        public byte[] code;
        public Source(Message buf)
        {
            //src
            //73 72 63 00
            //length
            //3F 02 00 00
            //What is this byte? Presents as 01 in this case
            buf.skip(1);
            //Armpen.java
            //41 72 6D 70 65 6E 2E 6A 61 76 61 00
            name = buf.string();
            //The rest of the bytes is the java code
            code = buf.bytes();
        }
        @Override
        public void init() { }
    }


[shubla]

Mr Loftar,

What is the single byte before the name in the "src" layer?
It's driving me nuts.

Code: Select all

    @LayerName("src")
    public class Source extends Layer
    {
        public String name;
        public byte[] code;
        public Source(Message buf)
        {
            //src
            //73 72 63 00
            //length
            //3F 02 00 00
            //What is this byte? Presents as 01 in this case
            buf.skip(1);
            //Armpen.java
            //41 72 6D 70 65 6E 2E 6A 61 76 61 00
            name = buf.string();
            //The rest of the bytes is the java code
            code = buf.bytes();
        }
        @Override
        public void init() { }
    }


Version number or number indicating type of the following data uint8. Of course loftar has not documented anywhere what data the file contains. Stating that source code is available in resource file, but then having to reverse-engineer/guess the whole format is just ridiculous.

[loftar]

Right, my recollection was that I just stuffed the source code right in there, but yes, you're right. The first byte is a version number in case of future changes, followed by a string indicating the filename, then followed by the actual source code. With the addition of the version number, your decoder seems completely correct.

Is this some standard format ? What does "/* $use: ui/tt/q/buff */" for example mean, or "> tt: Quality"

That's just the format I use to make the resource compiler construct the metadata, like the codeentry layer.

Cannot understand why you just can't put them in github!

I don't think my reasons or opinions have changed since we discussed it in this thread.

[shubla]

Right, my recollection was that I just stuffed the source code right in there, but yes, you're right. The first byte is a version number in case of future changes, followed by a string indicating the filename, then followed by the actual source code. With the addition of the version number, your decoder seems completely correct.

Is this some standard format ? What does "/* $use: ui/tt/q/buff */" for example mean, or "> tt: Quality"

That's just the format I use to make the resource compiler construct the metadata, like the codeentry layer.

Cannot understand why you just can't put them in github!

I don't think my reasons or opinions have changed since we discussed it in this thread.

I would still like to know how am I supposed to modify these codes, for example when I copy the Quality.java class with intention to change the color or the displayed quality number on items, I get stuff like

Code: Select all

Exception in thread "Haven UI thread" java.lang.RuntimeException: Cannot fetch resource of non-resloaded class class res.ui.tt.q.quality.Quality


any modifying to these classes is just absurdly difficult only because of the fact that code is packed in res files.

Like how am I for example supposed to do that small UI change without making some dramatic changes / seeing too much effort?

This isn't the only problem. If I replace the Quality.java with my own class, the "show" variable doesn't get changed, because now I must also somehow replace the ShowQuality.java with my own, which I haven't figured out how to do yet..

edit: in the end I found a way to do it, but its not very nice..
I had to replace every "Resource.classres" with Resource.remote.loadwait of the original unmodified resource. And make a terrible workaround to resource for loading my own code files. God knows what happens if devs decide to change any of the classes that I replaced in the future.

[Aoyama]

Arguably the real problem with that solution is just getting javac in the development environment to be able to get code from resources onto its classpath, and I really don't know how to solve that issue.

So I attempted to make a little tool that recompiles resource files, and indeed I was worried about the classpath issue. However I accidentally stumbled upon something that works.
javax.tools.JavaCompiler uses the current classpath when issuing the javac command. As my tool already uses Resource.remote() to load and inspect resources the offending resources were already loaded into the classpath and I was able to genereate a new resource.

So the remaining work I have to do in order to make more significant modifications is to auto generate Code Entries for the resource.