Difference between revisions of "Engine/NMPR"

From Minetest Developer Wiki
Jump to navigation Jump to search
m (Rubenwardy moved page Core Architecture to Engine/NMPR)
(45 intermediate revisions by 8 users not shown)
Line 1: Line 1:
[[Category:Core]]
+
== The base (NMPR) ==
[[File:minetest-0.3-dfd-visio.png|500px|thumb|right|0.3 data flow diagram - still relevant.]]
+
Everything is built on a small core, that was the original network multiplayer release of Minetest (let's call it NMPR; the 2010-10-24 version). Being around 10000 lines of code, it contains:
== General architecture ==
+
* The '''Map''': Voxel storage + lighting + rendering
 +
* The '''Environment''': Contains the map and the players, handles the simulation of the world
 +
* The '''Client''' + '''Server''' logic
 +
* The '''main loop''': Invokes the client, the server, the environment and the rendering.
 +
* A bunch of wrappers for OS-dependent things, and utilities.
  
Minetest always runs as a server and zero or more clients connected to it.
+
As the current code still largely bases on the NMPR, it is useful to look at how it works.
  
== Functions ==
+
NMPR has been made available at [https://github.com/celeron55/minetest_nmpr https://github.com/celeron55/minetest_nmpr] (easiest to build). Older source release is [http://c55.me/random/2013-01/minetest_10-10-24_16-33-41_wonderful.tar.gz here (source)] (build like [http://gist.github.com/4578183 this]). Also the [http://c55.me/random/2010-10/old/minetest-c55-win32-101024164856.zip original win32 release] is available (works in wine).
  
=== main() ===
+
=== Map (the voxels) ===
main.cpp
+
[[File:minetest_voxel_storage.png|400px|thumb|right|Minetest Voxel Storage]]
 +
* The main content of the Map is a <code>map<v2s16, MapSector*></code> container.
 +
* The main content of a MapSector is <code>map<s16, MapBlock*></code>.
 +
* The main content of a MapBlock is a linear array of 16x16x16 MapNodes.
  
When Minetest starts, main() comes up from main.cpp. It handles all the command-line arguments (server and client) and launches and processes the results of the main menu (client).
+
These form a relatively performant storage of voxel data. In addition to these containers, the latest fetched MapBlock is cached in each MapSector, and the last fetched MapSector is cached in Map, resulting in very useful sequential access speed through the whole abstraction layer.
  
=== dedicated_server_loop() ===
+
=== Environment ===
server.{h,cpp}
+
The content of the environment in NMPR is:
 +
* Map
 +
* List of Players
  
When the dedicated server is run, main() ends up running this and when it returns, Minetest quits.
+
Players contain:
 +
* Position and speed
 +
* An Irrlicht scene node (that is rendered by Irrlicht)
 +
* move() method with collision detection
  
It basically just feeds time to the server instance created by main() and reacts to SIGKILL etc.
+
In later versions of Minetest, the environment also contains things like [[ActiveObject|ActiveObjects]] and [[ABM|ABMs]].
  
=== the_game() ===
+
=== Network protocol ===
game.{h,cpp}
+
The high-level network protocol of NMPR is delightfully simple. There are four commands for the server, and four commands for the client. Since this, a lot has been added and changed, but the basic idea stays the same.
  
Launched when you start the game from the main menu.
+
==== Client -> Server ====
 +
{| class="wikitable"
 +
|TOSERVER_GETBLOCK
 +
|v3s16 p
 +
|Ask the server to send the data of a block
 +
|-
 +
|TOSERVER_ADDNODE
 +
|v3s16 p, MapNode node
 +
|Inform the server of a placed node
 +
|-
 +
|TOSERVER_REMOVENODE
 +
|v3s16 p, MapNode node
 +
|Inform the server of a removed node
 +
|-
 +
|TOSERVER_PLAYERPOS
 +
|v3s32 p*100, v3s32 speed*100
 +
|Inform the server of the positon of the local player
 +
|}
  
This creates the client (and the server, if needed), handles the waiting of loading stuff from the server, and runs the game in a somewhat badly structured main loop (which works well, though).
+
==== Server -> Client ====
 +
{| class="wikitable"
 +
|TOCLIENT_BLOCKDATA
 +
|v3s16 p, MapBlock data
 +
|Send the content of a block (16x16x16 nodes)
 +
|-
 +
|TOCLIENT_ADDNODE
 +
|v3s16 p, MapNode node
 +
|Add a node
 +
|-
 +
|TOCLIENT_REMOVENODE
 +
|v3s16 p, MapNode node
 +
|Remove a node
 +
|-
 +
|TOCLIENT_PLAYERPOS
 +
|foreach(player){
 +
:u16 player_id, v3s32 p*100, v3s32 speed*100}
 +
|Update players on client
 +
|}
  
== Threads ==
+
Minetest uses it's own reliability layer on top of UDP. It isn't well documented at the moment, and thorough understanding of it isn't that important, so let's skip it as of now.
  
=== Stand-alone server ===
+
=== Server ===
* main
+
The NMPR server logic is 287 lines of code that runs in two threads:
:* Doesn't do much
 
* ServerThread (Server)
 
:* Runs the server
 
* EmergeThread (Server)
 
:* Fetches and generates world
 
 
 
=== Client-only ===
 
* main
 
:* Runs almost everything in main game loop
 
* MeshUpdateThread (Client)
 
:* Does mesh updates in the background
 
 
 
=== Singleplayer ===
 
* main
 
:* Runs almost everything except server in main game loop
 
* MeshUpdateThread (Client)
 
:* Does mesh updates in the background
 
* ServerThread (Server)
 
:* Runs the server
 
* EmergeThread (Server)
 
:* Fetches and generates world
 
 
 
== Classes ==
 
 
 
=== IGameDef ===
 
An interface for fetching pointers to the managers of things. It is passed to almost everything.
 
 
 
It is implemented by Client and Server. Neither implements all interfaces.
 
 
 
Generally these can be accessed by referring to IGameDef:
 
* TextureSource
 
* ItemDefManager
 
* NodeDefManager
 
* SoundManager
 
* MtEventManager
 
 
 
This is the main difference between 0.3 and 0.4. In 0.3 this does not exist, because all content is defined in static tables in source code.
 
 
 
gamedef.{h,cpp}
 
 
 
=== TextureSource ===
 
Fetches, generates and caches textures.
 
 
 
tile.{h,cpp}
 
 
 
=== ItemDefManager ===
 
Stores the definitions of items, by item name. Content is set up at server startup, and transferred from server to client at beginning of connection.
 
 
 
itemdef.{h,cpp}
 
 
 
=== NodeDefManager ===
 
Stores the definitions of nodes and the mapping between node ids and names. Content is set up at server startup, and transferred from server to client at beginning of connection.
 
 
 
nodedef.{h,cpp}
 
 
 
=== SoundManager ===
 
Stores and plays sounds on the client.
 
  
sound.{h,cpp}; sound_openal.{h,cpp}
+
'''Main thread:'''
 +
* Run a simulation step of the environment (= move players)
 +
* Store the time passed from last time for the server thread
 +
'''Server thread'''
 +
* Receive and handle network packets
 +
* Handle network timeouts
 +
* Send player positions
  
=== MtEventManager ===
+
==== Packet handler ====
A minimal event manager currently only used for triggering sounds on the client.
+
The packet handler handles the TOSERVER_* commands coming from clients.
  
event.{h,cpp}
+
{|
 +
|TOSERVER_GETBLOCK
 +
|Serialize the content of a MapBlock and send it (TOCLIENT_BLOCKDATA)
 +
|-
 +
|TOSERVER_REMOVENODE
 +
|Set a node to be air and echo to other clients
 +
|-
 +
|TOSERVER_ADDNODE
 +
|Set a node to the type provided and echo to other clients
 +
|-
 +
|TOSERVER_PLAYERPOS
 +
|Update the position and speed of a player in the server environemnt
 +
|-
 +
|}
  
 
=== Client ===
 
=== Client ===
Contains a lot of stuff. Most considerable members are listed here.
+
Most of what the client does is very obvious, but there is one thing to note:
* TextureSource
+
{|
* ItemDefManager
+
|'''NMPR'''
* NodeDefManager
+
|The client "catches" accesses to unknown MapBlocks, and requests them from the server based on those accesses
* SoundManager
+
|-
* MtEventManager
+
|'''Later'''
* MeshUpdateThread
+
|The server sends MapBlocks based on where the client's player is located and what it hasn't sent yet.
* ClientEnvironment
+
|}
:* ClientMap
 
:* Players
 
:* ClientActiveObjects (CAOs)
 
* Connection
 
 
 
Implements IGameDef.
 
 
 
client.{h,cpp}
 
 
 
=== Server ===
 
Contains a lot of stuff. Most considerable members are listed here.
 
* ServerEnvironment
 
:* ServerMap
 
:* Players
 
:* ServerActiveObjects (SAOs)
 
* Connection
 
* BanManager
 
* Lua State
 
* ItemDefManager
 
* NodeDefManager
 
* CraftDefManager
 
* ServerThread
 
* EmergeThread
 
 
 
Implements IGameDef.
 
 
 
server.{h,cpp}
 
 
 
=== Lua State ===
 
The single Lua interpreter on the server. Built-in code and mods are loaded into it at startup.
 
 
 
=== Connection ===
 
Connection (client->server or server->clients)
 
 
 
=== ClientEnvironment (Environment) ===
 
Contains most of the actual game environment (players, objects, map...)
 
 
 
* ClientMap
 
* Players
 
* ClientActiveObjects (CAOs)
 
 
 
environment.{h,cpp}
 
 
 
=== ServerEnvironment (Environment) ===
 
Contains the actual game environment (players, objects, map, time of day, ...)
 
  
* ServerMap
+
=== Rendering ===
* Players
+
Rendering of players and GUI is done by regular Irrlicht stuff ([http://irrlicht.sourceforge.net/tutorials/ Irrlicht tutorials]). What is interesting is how the voxel world is rendered.
* ServerActiveObjects (SAOs)
 
  
environment.{h,cpp}
+
In NMPR, rendering of the Map works like this:
  
=== ClientMap (Map) ===
+
'''Cache step''' (runs in the background, in a thread managed by Map)
 +
* List MapBlocks in displayed area that have been modified
 +
* Update lighting in them
 +
* foreach(MapBlocks that were modified in the previous steps):
 +
:* Go through the MapBlock, detecting all air/non-air surfaces, collecting them to a list of faces
 +
'''Render step'''
 +
* Create a list of MapBlocks in rendering range
 +
* foreach(MapBlocks in rendering range):
 +
:* If it is not in front of the player, skip it
 +
:* Draw the faces of the MapBlock
  
map.{h,cpp}, clientmap.{h,cpp}
+
This is how Minetest works up to this day, except:
 +
* At some point, instead of storing faces, Minetest was made to store [http://en.wikipedia.org/wiki/Polygon_mesh meshes].
 +
* The mesh generator is managed by the Client.
 +
* In Minetest 0.3.1, [http://en.wikipedia.org/wiki/Hidden_surface_determination#Occlusion_culling occlusion culling] was added to the render step
 +
* In Minetest 0.4.3, the list of MapBlocks to be rendered is cached, and sorted by texture
  
=== ServerMap (Map) ===
+
=== Main loop ===
 +
'''Initialization:'''
 +
* Basic game stuff: Initialize graphics, load textures, set up camera, hide cursor
 +
* Start server if hosting a game (or playing a local game)
 +
* Start client and connect it to a server
 +
'''Loop'''
 +
* Read input
 +
* Run client (also steps the environment)
 +
* Run server
 +
* Update camera
 +
* Calculate what block is the crosshair pointing to
 +
* If the player left/right clicked, send a remove/add node command to server
 +
* Render scene
  
map.{h,cpp}
+
== Hmm... ==
 +
TODO: Describe newer things like world saving, inventories, scripting (definition managers) and sound.
 +
[[Category:Core Engine]]

Revision as of 19:28, 18 June 2020

The base (NMPR)

Everything is built on a small core, that was the original network multiplayer release of Minetest (let's call it NMPR; the 2010-10-24 version). Being around 10000 lines of code, it contains:

  • The Map: Voxel storage + lighting + rendering
  • The Environment: Contains the map and the players, handles the simulation of the world
  • The Client + Server logic
  • The main loop: Invokes the client, the server, the environment and the rendering.
  • A bunch of wrappers for OS-dependent things, and utilities.

As the current code still largely bases on the NMPR, it is useful to look at how it works.

NMPR has been made available at https://github.com/celeron55/minetest_nmpr (easiest to build). Older source release is here (source) (build like this). Also the original win32 release is available (works in wine).

Map (the voxels)

Minetest Voxel Storage
  • The main content of the Map is a map<v2s16, MapSector*> container.
  • The main content of a MapSector is map<s16, MapBlock*>.
  • The main content of a MapBlock is a linear array of 16x16x16 MapNodes.

These form a relatively performant storage of voxel data. In addition to these containers, the latest fetched MapBlock is cached in each MapSector, and the last fetched MapSector is cached in Map, resulting in very useful sequential access speed through the whole abstraction layer.

Environment

The content of the environment in NMPR is:

  • Map
  • List of Players

Players contain:

  • Position and speed
  • An Irrlicht scene node (that is rendered by Irrlicht)
  • move() method with collision detection

In later versions of Minetest, the environment also contains things like ActiveObjects and ABMs.

Network protocol

The high-level network protocol of NMPR is delightfully simple. There are four commands for the server, and four commands for the client. Since this, a lot has been added and changed, but the basic idea stays the same.

Client -> Server

TOSERVER_GETBLOCK v3s16 p Ask the server to send the data of a block
TOSERVER_ADDNODE v3s16 p, MapNode node Inform the server of a placed node
TOSERVER_REMOVENODE v3s16 p, MapNode node Inform the server of a removed node
TOSERVER_PLAYERPOS v3s32 p*100, v3s32 speed*100 Inform the server of the positon of the local player

Server -> Client

TOCLIENT_BLOCKDATA v3s16 p, MapBlock data Send the content of a block (16x16x16 nodes)
TOCLIENT_ADDNODE v3s16 p, MapNode node Add a node
TOCLIENT_REMOVENODE v3s16 p, MapNode node Remove a node
TOCLIENT_PLAYERPOS foreach(player){
u16 player_id, v3s32 p*100, v3s32 speed*100}
Update players on client

Minetest uses it's own reliability layer on top of UDP. It isn't well documented at the moment, and thorough understanding of it isn't that important, so let's skip it as of now.

Server

The NMPR server logic is 287 lines of code that runs in two threads:

Main thread:

  • Run a simulation step of the environment (= move players)
  • Store the time passed from last time for the server thread

Server thread

  • Receive and handle network packets
  • Handle network timeouts
  • Send player positions

Packet handler

The packet handler handles the TOSERVER_* commands coming from clients.

TOSERVER_GETBLOCK Serialize the content of a MapBlock and send it (TOCLIENT_BLOCKDATA)
TOSERVER_REMOVENODE Set a node to be air and echo to other clients
TOSERVER_ADDNODE Set a node to the type provided and echo to other clients
TOSERVER_PLAYERPOS Update the position and speed of a player in the server environemnt

Client

Most of what the client does is very obvious, but there is one thing to note:

NMPR The client "catches" accesses to unknown MapBlocks, and requests them from the server based on those accesses
Later The server sends MapBlocks based on where the client's player is located and what it hasn't sent yet.

Rendering

Rendering of players and GUI is done by regular Irrlicht stuff (Irrlicht tutorials). What is interesting is how the voxel world is rendered.

In NMPR, rendering of the Map works like this:

Cache step (runs in the background, in a thread managed by Map)

  • List MapBlocks in displayed area that have been modified
  • Update lighting in them
  • foreach(MapBlocks that were modified in the previous steps):
  • Go through the MapBlock, detecting all air/non-air surfaces, collecting them to a list of faces

Render step

  • Create a list of MapBlocks in rendering range
  • foreach(MapBlocks in rendering range):
  • If it is not in front of the player, skip it
  • Draw the faces of the MapBlock

This is how Minetest works up to this day, except:

  • At some point, instead of storing faces, Minetest was made to store meshes.
  • The mesh generator is managed by the Client.
  • In Minetest 0.3.1, occlusion culling was added to the render step
  • In Minetest 0.4.3, the list of MapBlocks to be rendered is cached, and sorted by texture

Main loop

Initialization:

  • Basic game stuff: Initialize graphics, load textures, set up camera, hide cursor
  • Start server if hosting a game (or playing a local game)
  • Start client and connect it to a server

Loop

  • Read input
  • Run client (also steps the environment)
  • Run server
  • Update camera
  • Calculate what block is the crosshair pointing to
  • If the player left/right clicked, send a remove/add node command to server
  • Render scene

Hmm...

TODO: Describe newer things like world saving, inventories, scripting (definition managers) and sound.