Find what lags

Let the agent use the profiler

FXServer lets a resource start its profiler only if server.cfg allows it. The block the panel hands out when you connect a server has these lines already. On a server you connected before, add them and restart the fivedock resource:

add_ace resource.fivedock command.profiler allow
add_ace resource.fivedock command.record allow
add_ace resource.fivedock command.saveJSON allow

fivedock_check in the server console says whether the profiler is allowed. Without these lines the report still counts commands and samples FPS, and tells you which lines are missing.

Profile the server

Run it while the server lags, from a machine that backs the box up (fivedock servers add). The agent records for 10 seconds, or 3 to 20 with --seconds, counts every resource’s commands, samples every player’s game and reads the client scripts and UI pages players download. --json prints the same report for scripts.

fivedock perf my-city
fivedock perf my-city --seconds 20

Read the report

A report from a server that a garbage job stalls every few seconds, and whose emote pack registers a command for every emote:

my-city  profiled 10 s · 214 players online
  server thread  late by 3.8 ms on average, 212 ms at worst in the last minute

Server  198 ticks, the slowest took 212 ms
  resource          total   per tick  worst tick
  jobs_garbage   1,236 ms     6.2 ms      205 ms
  hud              412 ms     2.1 ms      6.3 ms
  oxmysql           96 ms     0.5 ms       14 ms
  emotes            12 ms     0.1 ms      0.4 ms

Where exactly  the threads, events and exports that took longest
  jobs_garbage     611 ms  thread · server/main.lua:88-131 · 3× · worst 203 ms
  jobs_garbage     598 ms  thread · server/main.lua:12-40 · 198× · worst 6.1 ms
  hud              380 ms  event hud:sync · server.lua:52-77 · 1,420× · worst 4.2 ms
  oxmysql           88 ms  export · dist/build.js:310 · 402× · worst 14 ms

Commands  on the server, and in the game of 3 sampled players
  resource     server  players
  emotes            –    2,034
  admin_menu       61       12
  hud               9        4
  total           248    2,310
  A command per emote or item adds up fast. Check whether that resource needs them all.

Players  207 of 214 answered · 5 players in the pause menu or loading, left out
  FPS          24 for the slowest tenth · 47 median · 71 for the fastest tenth
  1% low       19 median · 64 players had a frame over 50 ms
  around them  980 objects · 310 peds · 180 vehicles (median)
  slowest
    Player 112   13 fps  1% low 4 · 9 frames over 50 ms · 4,210 objects  at 1693, 3245
    Player 17    16 fps  1% low 6 · 7 frames over 50 ms · 3,980 objects · a UI open  at 1702, 3251

Where exactly, in players’ games  the client functions that cost most, in the game they cost most
  hud              3.8 ms  thread · client/status.lua:14 · 9 players
  jobs_garbage     2.4 ms  thread · client/bags.lua:52 · 4 players
  phone            0.9 ms  NUI getMessages · client/nui.lua:88 · 2 players
  hud              0.6 ms  event hud:updateStatus · client/main.lua:120 · 12 players
  Measured in 31 traced resources, per frame.

Entities scripts made  in a player’s own game, the most any one of them had
  jobs_garbage  2,940 objects
  hud           12 objects
  A script that spawns props or peds and never deletes them slows every frame after.

Client loops that never sleep  each runs on every frame, in every player’s game
  hud         2  client/status.lua:14 · client/minimap.lua:41
  A while true do whose only Wait is Wait(0), or a setTick. resmon 1 in F8 shows what each costs.

UI pages  each loads in every player’s game on join and stays loaded
  phone      41.8 MiB  video ui/wallpapers/live.mp4 · endless animation ×4 ui/assets/index.css:1 · backdrop blur ×6 ui/assets/index.css:1
  hud         0.6 MiB  endless animation html/hud.css:212
  An endless animation, a backdrop blur or a video makes the game’s browser repaint every frame while it shows.

Streamed to every player  the heaviest packs, from the last asset scan
  [cars]       2.4 GiB · 23 texture dictionaries over 16 MiB
  sandy_mlo  479.7 MiB · 2 texture dictionaries over 16 MiB
  fivedock optimize <folder> shows what shrinking them saves, and --apply does it.

  Escrowed, so their client code could not be read: vehicle_shop

Exact cost per client resource  only a player’s own profiler can see it
  Ask Player 112 to type these two lines in F8:
    profiler record 500
    profiler saveJSON lag.json
  It saves into FiveM Application Data › citizen. fivedock perf --file lag.json reads it.

Stalls the agent caught  each one recorded for 10 s after it happened
  21:04  212 ms stall  then jobs_garbage took 203 ms in one tick
         jobs_garbage · server/main.lua:88-131

FPS drops the agent caught  players below 40 fps and well under their usual, newest first
  21:12  Player 112   58 → 14 fps  jobs_garbage
         jobs_garbage spawned 1,800 objects in the minute before
  20:51  Player 17    71 → 22 fps
         driving at 170 km/h, so the game streams fast
  20:30  Player 64    66 → 19 fps  phone
         phone started in the minute before · a UI had focus · the phone UI has a video and an endless animation

Server

The resources that used the most script time while recording. per tick is what they cost each server frame on average, worst tick their slowest single frame. Above 50 ms, players feel it as a stall.

Where exactly

That time split into threads, event handlers and exports, with the file, the lines, how many times each ran and its slowest run. A thread that runs every tick and costs milliseconds each time usually waits too little between rounds.

Commands

Every resource’s commands, on the server and in the game of three players. One resource with hundreds of them usually registers a command for every emote or item.

Players

The FPS of the slowest tenth of players, the median and the fastest tenth, their 1% low (the FPS of their slowest frames) and how many had a frame over 50 ms, what is around them, and the five slowest players with where they stand. Players in the pause menu or still loading are left out. When the slow ones stand in the same place, look at what is streamed there.

Where exactly, in players’ games

In resources with the client profiler, the functions that cost most per frame, in the game where they cost most, with the file, the line and how many players’ games had them. Without the profiler, the report says how to add it.

Entities scripts made

Props, peds and vehicles that a resource’s client scripts created in a player’s game, the most any one player had. Hundreds from one resource usually mean it spawns them and never deletes them.

Client loops that never sleep

Client loops that wake on every frame and can never sleep: a while true do whose only wait is Wait(0), or a setTick. Loops that break out, sometimes wait longer or only run in debug are left out. resmon 1 in a player’s F8 console shows what each one costs.

UI pages

Every resource’s UI page loads in every player’s game when they join and stays loaded. The report names the heavy ones and what keeps the game’s browser repainting while they show: endless CSS animations, backdrop blur and video, with the file and line, plus images over 1 MiB and pages loaded from another site.

Streamed to every player

The heaviest streamed packs and their texture dictionaries over 16 MiB, from the agent’s last asset scan. fivedock optimize shrinks those textures.

Stalls the agent caught

The stalls the agent recorded by itself, newest first.

FPS drops the agent caught

The drops the FPS watch logged, newest first, each with what changed in that player’s game. With the watch off, the report says how to turn it on.

Stalls it records by itself

When the server thread stops for 150 ms or more, the agent records the next 10 seconds without being asked, at most once every 15 minutes, and keeps the last three until the resource restarts. If the stall did not come back while it was recording, the report says so rather than blaming whatever happened to run.

FPS drops it logs by itself

With these lines the agent watches every player’s FPS. Their game reads its frame counter once a second, so watching costs nothing per frame. When a player falls below the floor and well under their own usual FPS for five seconds, the agent logs a player.low_fps warning to the Black Box: how low, from what, and what changed in their game just before. That can be a resource that spawned props or peds, one that had just started, a UI with focus, or driving fast enough to outrun streaming. A player is logged at most once every ten minutes.

set fivedock_fps_watch 1
set fivedock_fps_floor 40

fivedock_fps_floor is the floor, 40 fps unless you set it. Turning the watch on or off in the console takes effect within 30 seconds, without a restart. Filter the Black Box by player.low_fps to see every drop; fivedock perf lists the last ten.

See the client function behind each player’s FPS

FiveM keeps each resource’s timing inside the game, out of reach of scripts. fivedock trace puts a small profiler into every resource it can read: one line at the top of its manifest and a fivedock_trace.lua beside it. It stays idle until fivedock perf or the FPS watch asks, then times every thread, event handler, NUI callback and command in that resource for a few seconds. Escrowed, Skalp-protected and obfuscated resources are left alone, and the list says why.

fivedock trace C:\FXServer\server-data\resources
fivedock trace C:\FXServer\server-data\resources --apply

Restart the server so players load it. From then on the report and every player.low_fps warning name the resource, the file, the line and how many milliseconds a frame it took. On Windows, the desktop app does the same under Tools, Client profiler. To take it out again, every manifest back to the byte:

fivedock trace C:\FXServer\server-data\resources --remove --apply

One player’s FPS

What a script costs inside a player’s game is only visible to that game. When one player has low FPS, ask them to record it in F8 and send you the file:

profiler record 500
profiler saveJSON lag.json

fivedock perf --file reads it the same way as a server recording:

fivedock perf --file lag.json

For the sample, players get a few lines with the agent. When you ask, their game counts frames for three seconds, notes whether it is paused, loading or has a UI open, counts the props, peds and vehicles around the player with the resource that created each one, lists the commands it has registered, and collects what the client profiler measured where you added it. It reads nothing else, and sends nothing on its own unless you turn on the FPS watch.

What it cannot see

• A stall that comes back less often than a recording lasts. Run fivedock perf --seconds 20 around when it is due, or let the agent catch it.
• Time spent outside scripts, such as the network, the database server or the game’s own streaming. A script that blocks while it waits on the database does show up, as its own time.
• Time inside a resource without the client profiler: escrowed, Skalp-protected and obfuscated resources, and those written in JavaScript or C#. A player’s own F8 recording still shows them.
• What code does only some of the time. The report judges loops and UI pages by reading the files, not by timing them, so a loop that is heavy only now and then is not named, and code in escrowed resources cannot be read at all. The report lists those resources by name.