Code Processor - Make bigger levels, smaller codes, and edit ANY level (Rotation safe)

So this is a little thing I have been working on, this program has two major functions. It can:

1) Compress level codes.  Using this tool level codes can be compressed to approximately half size, allowing you to post more levels per post on the forums, but more importantly circumventing Exit Path's character limit, allowing your levels to be twice as big!

2) Convert your codes to the older level format, which designer can open properly, even if rotations and scaling are used. Edit any level!

Use should be pretty straightforward and obvious, just a few caveats and tips:

• .NET Framework version 4.0 is REQUIRED (The lighter weight client profile version is sufficient)
  
• Converting a level to old code strips any options set in the new designer out, so make sure to re-enable your flow and chat settings before you export your edited code from designer
  
• Old style level codes cannot be used as input for this program. Use designer to convert them to new style codes first.
  
• The name field is optional, if it is left blank the name of the code will be unchanged on conversion. If it is used, it will override the old name.
  
• It has limits, extremely large codes may still not compress enough to be loaded by Exit Path
  
• Actual compression amount will vary from level to level, but seems to be nearly a 50-60% character reduction in general.
  

So what's the catch?
Basically there is none. Loading the codes should be fast as ever, actually generating them is theoretically slower than generating less compressed code, but it is probably on the order of micro if not nano seconds.
So yeah, the catch is that for some reason yck used shitty compression in designer, so you have to use (and I had to make) this program to enable us to do this.

Personally I would recommend compressing all of your codes to save space and make them easier to post, but I know it's a pain in the ass. I can make a batch compressor if there is sufficient demand.

DOWNLOAD HERE

So the compress feature isn't rotation friendly, but the "make old code" is?

Pretty wicked stuff, Don. ^^

I like it,GOOD WORK

Retrocrystallix wrote:So the compress feature isn't rotation friendly, but the "make old code" is?

Pretty wicked stuff, Don. ^^

Yeah pretty much. Compress makes a code exactly like designer, but shorter. Rotations are fine in game, as with normal codes, but don't load into designer properly.

Make old code is rotation friendly for loading into designer.

Don, thank you very much, you are awesome.

This needs to be stickied.

Made it to Global Announcement, which will be sticky instead after some time

I'm kind of interested to know exactly how exactly the level codes are parsed so that compression works. Have you documented any of this or could you briefly explain how you figured out how to do this?

princevladlin wrote:I'm kind of interested to know exactly how exactly the level codes are parsed so that compression works. Have you documented any of this or could you briefly explain how you figured out how to do this?

I figured it all out by decompiling the game.

The key to it all is that normal level codes are already compressed, just not very well. When a level is loaded it is first decoded from the Base64 level code into a string of bytes. Those bytes represent the level information compressed via the deflate algorithm (a decompressor for which is packaged with actionscript). The compress function of this program works much the same, I decode the level from Base 64, and decompress it. Then I simply recompress it (better), and encode to Base 64. The compression/decompression in my program is provided by a .NET class provided by Microsoft, so I didn't have to do any of the really hard stuff.

The reason such drastic gains are seen is that the compression used by designer is very bad. Due to the nature of the deflate algorithm (you can look it up, though beyond a basic level it gets pretty hard to read), greater compression can be seen at the cost of doing more analysis on the data, specifically to see which characters (byte values) have greatest frequency, and adjusting the alphabet of the algorithm accordingly (to represent common bytes with less data than less frequent values). Obviously the implementation used in designer doesn't do that analysis well, (which is strange because the version I used is built into .NET, which designer uses. Maybe it's the difference between 3.5 and 4.0, or maybe yck used a third party or even his own implementation). It is worth noting that Microsoft's implementation is not the best out there. When I was figuring this stuff out I was doing it on a mac, and using zlib which gives better results (the 50% I stated in my first post is probably overly optimistic for this program, but with zlib it's a reality most of the time). Unfortunately zlib proved trickier to integrate than I hoped in this version of the program.


To further complicate things, the old style codes, while using the same encoding and compression, store the actual level information differently. I'll just list the differences:

• No flow/chat mode flag
  
• Old style codes contain the name of every object in the level (as set in designer), new style only contains the names for those where it matters to the game (teleporters, popspikes and axes)
  
• Old style stores all position/rotation/scaling information for each object individually, new style only stores each distinct rotation/scaling once. Position and a reference to which rotation/scaling is used are all that is stored for each object. (Theoretically there is a limit of 2147483647 distinct rotation/scaling schemes for a new style code. This limitation doesn't exist in old codes)
  
• Old style codes use utf-8 encoded characters to identify object types, new style just uses 8-bit integers
  
• New style codes have a few validity checks at the beginning of the code, without which no attempt to actually load the level will be made
  
• New style codes are identified by the presence of the '~' at the end, and use "~#-" to separate name from code. Old style uses just a '#' to separate name from code.
  

So the code conversion part of the program actually deals with the data, changing the format before recompressing.

Ah crap, why is that edit button so close to the quote button.

Sorry about messing up your post vlad, I'm terrible...


~Don

I don't remember what I said, but it probably wasn't important. I always accidentally quote myself on other forums instead of editing so..

~ Prince

Later I'll post a format docs of the code.

yck wrote:Later I'll post a format docs of the code.

Thanks yck.

Now this is what I was TRYING to do...


Yeah, I know how it's parsed :p

In actuality, I started looking at this stuff with the plan to make a mac designer as a project to learn Obj-C. Still considering it, though I would probably go Java to make it cross platform. I also had an interesting thought about making a parser so that you could just make levels in excel or something (I haven't researched this at all. Not sure about any details, or if it would even work, I can barely even use excel).


As for the compression ratios, I just did an experiment w/ Xak, and it turns out that compression ratio of the windows program is dependent on... something (probably either .NET version or processing power). My program on his computer is about the same as my zlib/mac version, slightly better even. 40% is really good, I only usually get just better than 50% (mac of course). Not sure why, maybe processing power matters more to zlib than I thought. Could be happen to be an easily compressible level as well.


yck, any thought of putting this stuff into designer? Makes  what I did worthless, but that's okay :p
Also what deflate implementation did you actually use?

Also yck, it's been driving me nuts, whats with &'ing the tile id with 127.... it will never change anything :p

To quote the documentation:

Starting with the .NET Framework 4.5, the DeflateStream class uses the zlib library. As a result, it provides a better compression algorithm and, in most cases, a smaller compressed file than it provides in earlier versions of the .NET Framework.

Most likely the compression difference is due to it.

The documentation is now available. It should answer your question.

Also, I've just uploaded a new version of designer, it should address most problems your tool reveals.

I just took a look on the new designer :

- Rotation bug fixed
- Level code is about half the size of previously (as does the program DonGato made)

Quick question : How does the Compression Strenght field work ? (btw filling it with '0' or negative values makes the designer crash)

Conclusion : @yck and DonGato : thanks for your awesome work guys, I'm impressed and grateful, really



EDIT : I just tested something with the 'Move' button : for very tall levels, the result is a drama lol, due to the bug that makes the blocks with Canvas.Left below 3000 AND Canvas.Top above 3000 become invisible.
I'm afraid that someone who made a very big level on the right of the red square attempts to suicide after having mistakenly clicked that 'Move' button ^^

Well here's a python implementation. It allows for batch conversion. Also, I got even lower sizes for some codes (around 35% of original size).

yck wrote:To quote the documentation:

Starting with the .NET Framework 4.5, the DeflateStream class uses the zlib library. As a result, it provides a better compression algorithm and, in most cases, a smaller compressed file than it provides in earlier versions of the .NET Framework.

Most likely the compression difference is due to it.

The documentation is now available. It should answer your question.

Thank you; this is great!

Don.Gato wrote:In actuality, I started looking at this stuff with the plan to make a mac designer as a project to learn Obj-C. Still considering it, though I would probably go Java to make it cross platform. I also had an interesting thought about making a parser so that you could just make levels in excel or something (I haven't researched this at all. Not sure about any details, or if it would even work, I can barely even use excel).

Parsing excel files would probably be a pain in the arse. What would you do about objects that have more freedom of movement than blocks? Give the option to specify an offset from the cell? It might be easier to just use plain text and have letters refer to the different objects and specify properties below the letter grid for positions. Neither would be really user friendly, especially for big levels. I've never worked with guis and probably won't be able to do anything, but I'm going to try messing around with generating the binary more with python. Good luck if you do attempt to make a designer in java.

why did chrome tell me the exe file is evil???

I think it is because the file isn't downloaded by a lot of people, so as the programm isn't used often, there is a risk. (for chrome)

I keep getting "invalid input" for all of my levels. Does anybody know what that's about?
Edit: It only worked for my more recent levels... Interesting.

Don.Gato wrote:

• Old style level codes cannot be used as input for this program. Use designer to convert them to new style codes first.
  

This.

Also, with the new designer this program is pointless, it takes care of compression and properly loads rotations...

oh duhhhh. I have level codes from Designer 1.0 xD

Made this a sticky, instead of a global announcement, as dragon had said.

moved to normal convo in general discussion.