[new formats] File formats

Project to create a new client based on new file formats and eventually a new 3D engine.

Moderators: osRose dev team, ospRose dev team, osiRose dev team, Moderators

[new formats] File formats

Postby lmame on Tue Aug 10, 2010 10:29 am

Ok let's do a list of file formats known for now and the one which should be changed.

Client server data files:
STB: remains the same
STL: remains the same
LTB: remains the same
AIP: remains the same
QSD: remains the same
CON: remains the same (not used by server)



client "map" files:
IFO: changed to .xml(s),
ZON: changed to .xml,



others:
DDS: remains the same (textures),
jpg/tga: remains the same (loading screens),
CUR: remains the same,
lua: remains the same,



To be determined:
ZMD: bones,
ZMS: meshes,
ZSC: "object linker". I guess this one could be changed to .xml quite easily no?,
TSI + xml: interface (?),
ZCA: cameras ?,
ZMO: animation,
EFT: effect,
VSO: shaders,
HLP: help stuff (epileptic...)
MRP: ?


Map related:
MOV: ?
HIM: ?
TIL: ?
LIT: ?
DAT: ?
The world is full of love and peace ^_^
Image
User avatar
lmame
Admin
Admin
 
Posts: 8997
Joined: Mon Aug 06, 2007 4:42 pm
Location: July City

Re: [new formats] File formats

Postby lmame on Tue Aug 10, 2010 10:29 am

Perhaps we could split in several .xml the .ifo and .zon:

For one map:
-> 1 xml for npcs + objects + monster spawns,
-> 1 xml for telegates / respawnareas (or mix it with the one above though as they are coming from .zon... would be easier at first to make a separate one),
-> 1 xml for all that concerns only the client (constructions, trigger events, sounds and so on),

That would mean about 3 to 5 XMLs per map and though should be fast enough to read them and friendly enough rather than having 100s of files...


As for their filenames we should use different names for all maps, at least as "root" so people could copy them to the server.
That would avoid having to copy the whole folder structure and since people are lazy I can only imagine they would just copy everything leading to a 800 MB server thing...
I guess they would already copy all the xml rather than the ones actually needed if we left the ".xml" extension...
Or we could use several file extensions like ".serv" for files to be copied to server and ".cli" for files to be left to client?

The name could be "JPT01_xxxx.yyy" with JPT01 the folder name for example, the one we can find in the STB.
For now when they try to make several maps from one (like clan wars thing) they keep the same map structure so I guess we won't have a problem on this side. The difference is done server side with AIP and QSD summoning more or less strong monsters and so on.
At server loading server would load the stb at each map and would look for the according ".serv" files like "mapname_npc.serv" and would load it.

We shouldn't give the mapid directly in the xml so people can change it in the STB directly. The "name" (JPT01) though is usually only done once and for all and would be changed rarely.

Example for a xml "npc + object + monster spawns", named "JPT01_npc.serv":
  1.  
  2. <map id="JPT01" name="Junon Polis map">
  3.     <slice x="31" y="31">
  4.         <npclist>
  5.             <npc type="1024" x="5011" y="5052" dir="180" dialog="npc_1024.con">Judy</npc>
  6.             <npc type="1025" x="5000" y="5000" dir="0" dialog="npc_1025.con">Spero</npc>
  7.         </npclist>
  8.         <objlist>
  9.             <obj id="1" x="5051" y="5022" dir="180" qsd="hourglass_5067">hour glass</obj>
  10.         </objlist>
  11.         <spawnlist>
  12.             <spawn x="5200" y="5200" range="10" rtime="10" limit="10" tactic="200" day="0" active="1">
  13.                 <group monster="301" nb="2" tactic="0"/>
  14.                 <group monster="302" nb="1" tactic="0"/>
  15.                 <group monster="303" nb="1" tactic="0"/>
  16.                 <group monster="301" nb="4" tactic="1"/>
  17.                 <group monster="302" nb="3" tactic="1"/>
  18.             </spawn>
  19.         </spawnlist>        
  20.     </slice>
  21. <slice x="31" y="32">
  22.         <npclist>
  23.             <npc type="1024" x="5011" y="5052" dir="180" dialog="npc_1024.con">Judy</npc>
  24.             <npc type="1025" x="5000" y="5000" dir="0" dialog="npc_1025.con">Spero</npc>
  25.         </npclist>
  26.         <spawnlist>
  27.             <spawn x="5300" y="5300" range="10" rtime="10" limit="10" tactic="200" day="0" active="1">
  28.                 <group monster="301" nb="2" tactic="0"/>
  29.                 <group monster="302" nb="1" tactic="0"/>
  30.                 <group monster="303" nb="1" tactic="0"/>
  31.                 <group monster="301" nb="4" tactic="1"/>
  32.                 <group monster="302" nb="3" tactic="1"/>
  33.             </spawn>
  34.         </spawnlist>        
  35.     </slice>    
  36. </map>
  37.  


for telegates and respawn zones, named "JPT01_gates.serv":
  1.  
  2. <map id="JPT01" name="Junon Polis map">
  3.     <telegatelist>
  4.         <telegate id="2" x="5200" y="5200" dmap="10" dx="5100" dy="5100"/>
  5.     </telegatelist>
  6.     <respawnlist>
  7.         <respawn id="3" x="5000" y="5000" type="0">
  8.     </respawnlist>
  9. </map>
  10.  


If we want to continue the slice system instead of simply doing:
  1.    <slice x="31" y="31">


We could insert another entry so a generic XPath query would find the according data quickly, even in a single file (like "//slice_31_31" or even a root query "slice_31_31"):
  1.  
  2.     <slice_31_31>
  3.         <slice x="31" y="31">
  4.             <npclist>
  5.             </npclist>
  6.             <objlist>
  7.             </objlist>
  8.             <spawnlist>
  9.             </spawnlist>        
  10.         </slice>
  11.     </slice_31_31>  
  12.  


The other entry would be used by the server, I would need it to calculate map size or we would have to do this that could lead to confusions if someone puts wrong coordinates:
  1. <map id="JPT01" lx="4500" ly="4500" width="1024" height="600" name="Junon Polis map">

I know how to get real maps coordinates from slices so I would prefer that server fetches it rather than letting people do this. Experience shows lately that people just don't update the map_grid.csv so we'll have to do it for them.
So I guess they wouldn't do it for their xml...


Now that beeing said, the problem with xmls is that you can't go in database and tell "ok let's stop this event" or see the spawns that are overall on your server, making mysql queries for example.
It's especially true for the npcs who not only have a dialogid but also an eventid that subsets their dialogs...
That would mean keeping at least a list_npc mysql database storing the events for the npcs. Though instead of storing all npcs, we could just store the ones who have their eventid changed.
So I'll have to change that but shouldn't be a big deal.

Another possible issue is the ".serv" size for the file with monster spawns. Could be big.


I don't know exactly what client needs in the .ifo so I can't really tell :? but the same way could apply, I think.
Don't forget for now several Objects can have the same ID in the same map since it's "reseted" when you change from tile, actually it uses the tile itself as a "key", like "31" and "31". So it's referenced as "object 2 from tile 31 31" in qsd for example.
Object beeing union warp gate or walls...
The world is full of love and peace ^_^
Image
User avatar
lmame
Admin
Admin
 
Posts: 8997
Joined: Mon Aug 06, 2007 4:42 pm
Location: July City

Re: [new formats] File formats

Postby Raavi on Tue Aug 10, 2010 11:35 am

HLP files contains text for example the epleptic screen text is inside a HLP file.
Raavi
Rackie
Rackie
 
Posts: 243
Joined: Sun May 30, 2010 4:15 pm

Re: [new formats] File formats

Postby lmame on Tue Aug 10, 2010 12:43 pm

Ok :)
The world is full of love and peace ^_^
Image
User avatar
lmame
Admin
Admin
 
Posts: 8997
Joined: Mon Aug 06, 2007 4:42 pm
Location: July City

Re: [new formats] File formats

Postby PurpleYouko on Tue Aug 10, 2010 3:07 pm

I am liking this XML format.
I'm not really familiar with XML but from the examples it looks very workable.
why not just have a small mysql event table that contains only the relevent NPC number (from list_npc.stb) and the event id. Load it is seperately from the XML
Need to lookup information on NARose items, skills, quests?
Now featuring a newly completed skill tree for all classes
Formatting fixed for different resolutions
Image

"A Gazelle is nothing but a giraffe plotted logarithmicaly"
User avatar
PurpleYouko
Rose Guru
Rose Guru
 
Posts: 4733
Joined: Fri Aug 10, 2007 2:05 pm

Re: [new formats] File formats

Postby Aukemon0NL on Tue Aug 10, 2010 4:32 pm

lmame nice work on the xml files.
This looks good to me.
I like that idea.
But the problem would be why not make everything in xml.
this will go alot fast and you don't have to use stb and so on.
But than again I will need to make an editor that does the job so people won't make to much mistakes.
and an converter will be handy 2 -,-" So that will be the problems else it will be alot easier to create all xml files.
and either call them .serv or .cli
based on what's needed in the server and what not.
Image
User avatar
Aukemon0NL
Pomic
Pomic
 
Posts: 107
Joined: Fri May 16, 2008 8:53 pm
Location: The Netherlands

Re: [new formats] File formats

Postby lmame on Tue Aug 10, 2010 5:28 pm

PurpleYouko wrote:I am liking this XML format.
I'm not really familiar with XML but from the examples it looks very workable.
why not just have a small mysql event table that contains only the relevent NPC number (from list_npc.stb) and the event id. Load it is seperately from the XML

It can be workable, my concern was rather the "friendly loading part" C++ side though it seems libxml++ would do the trick and it's quite easy to add to an existing project.
Yeah that's what I was thinking for the events :)

Aukemon0NL wrote:lmame nice work on the xml files.
This looks good to me.
I like that idea.
But the problem would be why not make everything in xml.
this will go alot fast and you don't have to use stb and so on.
But than again I will need to make an editor that does the job so people won't make to much mistakes.
and an converter will be handy 2 -,-" So that will be the problems else it will be alot easier to create all xml files.
and either call them .serv or .cli
based on what's needed in the server and what not.


I'll try to explain the problem I have with the "all xml" thing.
If you do just everything in xml you'll need to do also editors and tools to export from STB, STL, LTB, QSD, AIP and so on to .xml... That'll do a LOT of tools and editors.

You think everyone will begin from scratch though it won't be true and it's not a scratch game too. People will want to import back their "old" files (or latest naRose's files) or keep their old editing habits or whatever.
Besides, you'll find very difficult for some formats to just export everything to .xml. I'm thinking about QSD and AIP for example. It'll be quite hard and painful to just recode everything and not so fast in the long run. Look at the massexport output (qsd and aip exporter bundled in dev rev), it's like 9 MB text. If you want to do a qsd "xml" friendly that's the kind of file size in total you'll have to face and it'll take time to load...
Look at the STBs, STLs, LTBS too. It's just a bunch of rows. In xml it wouldn't be really efficient since there's no structure at all, I guess a mere .csv file would be enough for them in fact.

Now talking about performances, osRose dev rev II loads all QSDs, AIPs, STBs, STLs, LTBs in less than 4 seconds with the massexporter on, meaning I'm exporting all QSDs and AIPs back to a text file for references (and fetching item names, stats, LTB sentences and so on). And in a vmware and loading also spawns and data from the mysql database.
So in a real life server it's about 2 seconds or less and it's not even optimized.
So perhaps it would be faster with .xml (if the .xml structure is well done) but I wouldn't be so sure of that. After all, the libs like libxml++ just are text or DOM parsers and I honnestly don't think loading a STB would be slower than loading the XML version, same for QSD. Their strong point is that is relying on offsets and data sizes most of the time, which is fast. And instead of reading from file, we could even load parts into memory to make it faster.
XMl on the other side needs to be parsed or traversed. For the IFO it'll be great since it'll be possible to begin by fecthing NPCs then other than other and so on. But on a STB... I fail to see the point. You'll just have that:
  1. <record id="1">
  2.  <col id="1">value</col>
  3. ...
  4. </record>
  5. <record id="2">
  6. ...
  7.  

hardly a structure...

More over let's not forget that the difference between those files and .ifos .zons and others that matter client side, is that all the STBs, STLs, QSDs, AIPs, LTBs are loaded only once at startup.
So it would be a lot of pain and work for just (perhaps) saving some time (like 0.5 second) just to get .xml... At the expense of multiple editors.
And when you see 80% of questions here concern simple file copy or .sql generation they forgot to do...


My point is, why would we change well known and robust (or pain in the xxx) file formats when loaders / interpreters are working perfectly now and when editors are working perfectly or nicely too?
A contrario, ifos, zons and others like zms, zsc, zmo (and so on) have very few or none public editors and none of them are user friendly, though rippers are working ok for some of them so for some doing xml versions would be a treat.
The world is full of love and peace ^_^
Image
User avatar
lmame
Admin
Admin
 
Posts: 8997
Joined: Mon Aug 06, 2007 4:42 pm
Location: July City

Re: [new formats] File formats

Postby Aukemon0NL on Tue Aug 10, 2010 10:45 pm

Thanks for the update.
hmmm
but than we need to build something like a table in some other language.
I personally hate stb and so on because all those extra tools you have to download.
That's why I was suggesting it.
Image
User avatar
Aukemon0NL
Pomic
Pomic
 
Posts: 107
Joined: Fri May 16, 2008 8:53 pm
Location: The Netherlands

Re: [new formats] File formats

Postby lmame on Tue Aug 10, 2010 11:50 pm

Yeah if we were about to do a brand new game why not. Though it's not the case here. And you would need tools and editors anyway.
Here you're just adding layers of editors and rippers, and overall it wouldn't really be user friendly sadly.

In fact if we were in the same position as iRose or prose I would have say "perhaps" since the game is "dead". But Evo has some "living" official servers and well, naRose, even if it could do a better job, does some updates, maps, quests and items.
If we don't let us and players the possibility to quickly update their own server to latest naRose files, I mean, it wouldn't be that interesting.

And to be honnest, the file format I dislike the most is the STL one which is kinda very weird on string lengths... But as for the others (STB / LTB / QSD / AIP) once you understand why they did it let's say they are efficient.
Of course they're not designed to be loaded like "load record 3" but rather "load once and for all" but it's understandable. I prefer loading all in memory rather than accessing hard disk all the time anyway.

Why do you speak about a table in another language?
If you think about porting STB (and others) reading in other coding language it's fairly easy. I did it in PHP and in another language a while ago. The file format is very easy (STL tricky side apart).

About extra tools... Well... Honnestly I wouldn't go and edit directly xml files without an editor. It would be the highway to fuck it up.
And don't forget that some of the files have strings inside, like STL, LTB and even STB and some are in unicode...
That would mean escaping some characters. A "normal" person would easily miss it and that would be quite bad.
That's why in all cases an editor would be needed so why not using the ones we already have ;)
The world is full of love and peace ^_^
Image
User avatar
lmame
Admin
Admin
 
Posts: 8997
Joined: Mon Aug 06, 2007 4:42 pm
Location: July City

Re: [new formats] File formats

Postby xqnp7 on Fri Nov 12, 2010 12:36 pm

When I was a moderator at SevenHearts Online (an older, dead Arcturus server) the ZSC editing tool we had converted the ZSC to and from xml. Having firsthand experience using xml as a ZSC, I have to say it was massively easier than using a ZSC.

So, I'm definitely in support of using xml for ZSC. Actually, I would strongly suggest it.
AKA Sneaux, VXJP1, Sanctum
xqnp7
Jelly Bean
Jelly Bean
 
Posts: 31
Joined: Tue Jun 17, 2008 9:42 pm
Location: Tennessee, USA

Next

Return to [Project] Custom client (new 3d engine, new files formats)

Who is online

Users browsing this forum: No registered users and 5 guests

cron