Unofficial PAPA file format

Discussion in 'Mod Support' started by LennardF1989, June 4, 2013.

  1. LavaSnake

    LavaSnake Post Master General

    Messages:
    1,620
    Likes Received:
    691
    Seams to work fine on windows. Very nice!
  2. ozonexo3

    ozonexo3 Active Member

    Messages:
    418
    Likes Received:
    196
    this what happend when I'm trying to import Delta commander textures using this:

    Attached Files:

  3. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    You mean it's not supposed to be like that? ;)

    Yeah, I know. I only figured out the A8R8G8B8 format. The ones you are trying to load are DXT1. As you can see, I haven't quite figured that one out yet. I applied the DXT1 format from the Microsoft MSDN pages, but there is something fishy going on.
  4. superouman

    superouman Post Master General

    Messages:
    1,007
    Likes Received:
    1,139
    When I open papadumb and papatran, it closes immediatly on win7 x64. I need to try on my other comp with linux.


    And DeathbyDenim, you should open a thread for this mod and not highjack this one. :p


    By the way, there is a second skybox in Planetary Annihilation\PA\media\pa\terrain\sky\textures
    As I understand, the game selects only skybox_01 so if you edit skybox_02 to skybox_01 file names (don't forget to change the file name in the json file), you get the skybox you want.
    Is there a way to call the skybox you want from an other file if you have multiple skyboxes in your texture folder?
    Once we will have many skyboxes, we need a script which randomly selects one skybox for every game.
    [​IMG]
    Last edited: February 4, 2014
    lokiCML likes this.
  5. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    Hmm, getting closer with the DXT1 format, but it's not quite right.
    imperial_delta_headlight.png imperial_dxt1.png
    The left is the png from build 53072 (just before they put everything in papa files) and the right is what my DXT1 decoder makes of the current papa files. This specific picture is for the headlights of the imperial delta commander. The shape seems ok. I can even read out the subsequent minimaps in the papafile and they have that shape. However, I can't seem to figure out the colours. According to the Internet, the two extreme colours for DXT1 should be
    Code:
    union colour_t
    {
      quint16 value;
      struct
      {
        quint16 red : 5;
        quint16 green : 6;
        quint16 blue : 5;
      } fourintotalcolour;
      struct
      {
        quint16 red : 5;
        quint16 green : 5;
        quint16 blue : 5;
        quint16 alpha : 1;
      } threeintotalcolour;
    };
    But that clearly doesn't work. It may be wrong though. If I look at the example on msdn I am quite puzzled:
    [​IMG]
    Here it says that red is 255, which in 16-bit is 11111111 00000000, which according to the RGB 5:6:5 format is
    red: 11111 (5 bits)
    green: 111000 (6 bits)
    blue: 00000 (5 bits)
    That's red-greenish though. So possibly (fine, probably) I am doing it wrong and messing up the order of the bits?

    See GitHub for my full decoding code. Any ideas?


    It's still proof of concept code. I.e. figuring out the PAPA file format.
  6. Dementiurge

    Dementiurge Post Master General

    Messages:
    1,094
    Likes Received:
    693
    My guess would be that they're just saying that red is the maximum value, not that it's 255 in bits.
    So it shouldn't be 11111111 00000000, it should be 11111000 00000000, at least according to this: http://msdn.microsoft.com/en-us/library/windows/hardware/ff566496(v=vs.85).aspx

    Then again, I did have to look up what "most significant bit" means, so I'm probably 99% wrong.
  7. maxpowerz

    maxpowerz Post Master General

    Messages:
    2,208
    Likes Received:
    885
  8. maxpowerz

    maxpowerz Post Master General

    Messages:
    2,208
    Likes Received:
    885
    maybe it's raw PNG data minus the png header from the point of the texture offset onward in the papa file,,
    I tried looking for a few common image headers inside the papa but only found the papa header so far.
    (im using Sweetscape 010)
  9. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    Ah, yeah. That might be it. It doesn't make much sense otherwise.

    Interesting links, maxpowerz. It seems trying out glCompressedTexImage2DARB is worth a shot.

    I'm pretty sure it's not raw PNG data though. I checked the length of the texture data against how much a DXT1 texture would take with all the minimaps and it matches exactly. That is, for a 512 x 512 texture the first minimap takes 512 x 512 / 2 = 131072 bytes, the next one 256 x 256 / 2 = 32768 bytes, etcetera until 2 x 2 and 1 x 1, which still takes 8 bytes due to the fact that everything is stored in texels of 4 x 4. Added together these give exactly the length of the texture data. Also, if you zoom in very closely to the "decoded" picture above, you can see that the 4x4 blocks match their neighbours in shape(, but not in colour). That indicated to me that 4 x 4 texels are indeed used, which is something that PNG doesn't use as far as I know.

    edit: Changed 4 bytes to 8 bytes for 1 x 1 and 2 x 2 mipmaps.
    Last edited: February 5, 2014
    maxpowerz likes this.
  10. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    No, it doesn't look like there is. It's hardcoded in the PA binary. If you have a hexeditor, you can change the skybox at position 0x005F0D70 where it says "/pa/terrain/sky/textures/skybox_01.json". Changing that to "/pa/terrain/sky/textures/skybox_02.json" lets you load the other skybox. (Which is breathtakingly ugly by the way. :))
  11. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    Yay, I figured it out. The texture format is indeed DXT1. It's not the one from MSDN though. Maxpowerz link pointed me in the right direction. The MSDN one has an alpha value, while the OpenGL one also has a flavour without alpha and guess which one PA uses... Also, there was an error in the colour interpolation. At some point I needed to take the average between two colours, so I followed the instructions to the letter and averaged the uint16 values. It turns out they meant the average for each colour component. Which makes sense in retrospect of course (and the same for interpolation).
    dxt1headlights.png
    The code is on GitHub as well as Linux and Windows binaries. It only decodes DXT1 at the moment, but DXT5 should be similar.

    For completeness' sake, I looked at the papadump binary and it seems there are loads and loads of other possible formats. I ran papadump on every texture though there aren't any other than DXT1, DXT5, A8R8G8B8 or X8R8G8B8 (yet?).
    Code:
    enum
    { // This comes from inside the papadump binary at position 0x0055E0DE
      Invalid = 0,
    A8R8G8B8,
    X8R8G8B8,
    A8B8G8R8,
    DXT1,
    DXT3,
    DXT5,
    R32F,
    RG32F,
    RGBA32F,
    R16F,
    RG16F,
    RGBA16F,
    R8G8,
    D0,
    D16,
    D24,
    D24S8,
    D32,
    R8I,
    R8UI,
    R16I,
    R16UI,
    RG8I,
    RG8UI,
    RG16I,
    RG16UI,
    R32I,
    R32UI,
    Shadow16,
    Shadow24,
    Shadow32
    } Format;
    Anyway, this means that the texture format has been successfully reverse engineered. So go nuts with writing tools and such. I'll go finish the papa texture editor.

    EDIT: Oh, oops. I forgot about the SRGB transform that is determined by the srgb bit. That's a simple transformation though. See Wikipedia.
    Last edited: February 6, 2014
  12. superouman

    superouman Post Master General

    Messages:
    1,007
    Likes Received:
    1,139
    On two different computers, i can't use papadump and papatran, they directly close.

    On the first computer (DxDiag_1), i also tried launching papadump on ubuntu (dual boot) but it doesn't launch either.

    I am stuck :(

    Attached Files:

  13. cwarner7264

    cwarner7264 Moderator Alumni

    Messages:
    4,460
    Likes Received:
    5,390
    [​IMG]
    tatsujb and LavaSnake like this.
  14. superouman

    superouman Post Master General

    Messages:
    1,007
    Likes Received:
    1,139
    I think he is.
    [​IMG]
  15. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    Drat, did my avatar give it away? :)
    (Doh, ninja'd by superouman)

    Anyway, raevn and LennardF1989 did most of the work by finding out the overall structure of the papa files.


    I think the problem is that you expecting a GUI. They are command line utilities and as such need to be run from the command line. Starting them from the file browser will make it appear they don't start.

    On Ubuntu, open the "Terminal" program, then type
    Code:
    cd <PA Installation directory>/tools
    ./papadump
    ./papatrans
    Running those commands will show you more info on how to run use them.

    It's similar on Windows, but I don't really know how to open a command line there anymore since I haven't used any Windows newer than XP.
    maxpowerz likes this.
  16. maxpowerz

    maxpowerz Post Master General

    Messages:
    2,208
    Likes Received:
    885
    To use "Paptrans.exe"

    Have a png or FBX model in same directory as papatrans.exe.

    Then use papatrans via commandline,

    Usage
    papatrans input-filename -o output-filename

    Examples

    papatrans image.png -o image.papa
    papatrans model.fbx -o model.papa

    papatrans automatically detects the content of the file it's trying to convert to PAPA format.
  17. bgolus

    bgolus Uber Alumni

    Messages:
    1,481
    Likes Received:
    2,299
    Most textures are going to be DXT1 (w/o alpha) or DXT5. DXT1 with out alpha is far more common than DXT1 with alpha in the games industry in general, even though documentation on DXT formats frequently treat the w/ alpha format as the main mode. Technically they're part of the same format as if the texture should be interpreted as having alpha or not is determined by the color palette byte order (if I remember correctly). DXT5 is going to be used frequently as well for the diffuse texture on most units and effects textures. Eventually we'll be using uncompressed pngs, DXT5 or DXTn for normals, but we haven't made that change yet.

    DXT5 is a non alpha DXT1 texture with a second DXT1 like format for storing a smooth alpha. DXTn is an odd format that is just two channels stored in the same format as alpha in the DXT5. It's supported by most PC hardware and the Xbox 360, but famously not on the PS3 which caused a bit of headache for cross platform devs in the past. (Both the Xbone and PS4 support it.)
  18. superouman

    superouman Post Master General

    Messages:
    1,007
    Likes Received:
    1,139
    Ah thank you very much guys, I don't have much knowledge so you spared me a lot of time trying to figure it out.
    However i still didn't manage to convert a file. I place my png file into the tools folder and write in the console as you can see on the screenshot.[​IMG]
    I thought I had to write the entire path of the png file but there is not difference.
    I guess when you write "papatrans" with an "s", this is a typo.
  19. DeathByDenim

    DeathByDenim Post Master General

    Messages:
    4,327
    Likes Received:
    2,125
    Right, I forgot to mention that. It's quite a finicky program. It expects a default.settings to be there. You can copy the one from <PA Install dir>/media/pa/ or just place your sky_top.png in the appropriate directory in the <PA Install dir>/media/pa folder. For sky_top.png I would expect you would want <PA Install dir>/media/pa/terrain/sky/textures.
    superouman, Raevn and maxpowerz like this.
  20. superouman

    superouman Post Master General

    Messages:
    1,007
    Likes Received:
    1,139
    Thanks to the DeathByDenim and maxpowerz's advice and patience, i finally managed to import a skybox made in Space Engine. It was quite a pain to assemble it the right way since they don't have the same structure. Anyway, here are some screenshots of the skybox as well as the papa files.

    [​IMG]

    [​IMG]

    Download link
    Extract the archive in \Planetary Annihilation\PA\media\pa\terrain\sky\textures
    Be careful, before you extract the archive, make a backup of the standard skybox if you want to keep it.

    All this is cool but there are big conversion issues, the quality of the skybox pictures falls drastically in the papa format. The original png files are very good. Does anyone know something?
    DeathByDenim, in the skybox pictures you posted, it doesn't look like there are quality losses, especially on the one with your avatar.
    Last edited: February 11, 2014

Share This Page