How to Stop MongoDB 8.x.x Log Files from Exploding on Debian 12/13

MongoDB 8.x.x is one of those services that can run quietly for a long time, until one day you check disk usage and realize the log file has grown into something unreasonable. Not because MongoDB 8.x.x is broken, but because the logging behavior is doing exactly what it was told to do, keep writing, keep appending, and never stop unless someone outside the process manages rotation properly.

This is a common situation on Debian-based servers, Debian 12/13, especially when MongoDB 8.x.x is configured to write logs into a dedicated storage path and logAppend: true is enabled. At first glance, everything looks normal.

The server runs. Replication works. Authentication is enabled. Nothing crashes. But the log file quietly grows and grows until it starts eating storage much faster than expected.

In this article, we will look at why this happens, what MongoDB actually means by log rotation, and the clean Debian-friendly way to keep the log file under control.

The Configuration That Looks Fine, Until It Isn’t

A typical MongoDB 8.x.x configuration on Debian might look like this:

systemLog:
  destination: file
  logAppend: true
  path: /mnt/disk-log/mongodb/mongod.log
  logRotate: reopen
Code language: JavaScript (javascript)

At a glance, this seems like log rotation is already configured. That assumption is where many administrators get trapped.

The key thing to understand is that logRotate: reopen does not mean MongoDB 8.x.x will rotate the log automatically by itself. It only defines how MongoDB 8.x.x behaves when a log rotation event is triggered externally. In other words, MongoDB 8.x.x is prepared to reopen the log file, but something else still needs to perform the actual rotation.

With logAppend: true, MongoDB 8.x.x keeps appending to the same file across restarts. That is usually desirable, because you do not want logs truncated every time the service restarts. But without a proper rotation mechanism, the file will simply continue growing forever.

Why the Log File Keeps Growing

There are two separate concerns here, the first is where MongoDB 8.x.x writes logs, the second is who rotates them.

MongoDB 8.x.x can write to a file, and it can react when rotation happens. But on Linux, especially on Debian 12/13, the actual rotation is typically handled by the operating system through logrotate.

This means your current configuration is only half of the solution.

MongoDB 8.x.x is ready to reopen the log file after rotation. But if you do not configure Debian’s 12/13 logrotate to rename the file and notify MongoDB 8.x.x, then nothing ever rotates.

The result is predictable, one endlessly growing mongod.log.

The Right Way on Debian 12/13

The clean and standard solution is to let Debian’s logrotate manage the file, while MongoDB stays configured with logRotate: reopen.

That pairing is important, MongoDB 8.x.x should continue using

systemLog:
  destination: file
  path: /mnt/disk-log/mongodb/mongod.log
  logAppend: true
  logRotate: reopen
Code language: JavaScript (javascript)

Then create a logrotate policy such as

/mnt/disk-log/mongodb/mongod.log {
    daily
    rotate 14
    maxsize 100M
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    create 640 mongodb mongodb
    postrotate
        /bin/kill -SIGUSR1 $(pidof mongod) 2>/dev/null || true
    endscript
}
Code language: JavaScript (javascript)

This does several useful things at once.

It rotates the log daily.
It also rotates when the file exceeds 100 MB.
It keeps 14 archived logs.
It compresses older logs to save disk space.
And after rotating the file, it tells MongoDB to reopen its log handle using SIGUSR1.

That last step is the critical piece. Without it, MongoDB 8.x.x may continue writing to the old file descriptor even after rotation.

Why reopen Matters

MongoDB 8.x.x supports different log rotation behaviors, but on Linux the practical pattern is simple: let the OS rename the log file, then signal MongoDB 8.x.x to reopen it.

That is exactly what logRotate: reopen is for.

This mode works well with logrotate because the external tool handles the rename, while MongoDB 8.x.x responds by reopening the file path and continuing cleanly into a fresh log file.

This is also why using copytruncate is generally not the preferred choice here. copytruncate copies the log contents to a rotated file and then truncates the original file in place. It can work for some applications, but for MongoDB 8.x.x the cleaner method is rename plus signal.

In short, if you already use logRotate: reopen, stay consistent and let logrotate do the rename properly.

Reducing Log Noise

Rotation solves the storage growth problem, but sometimes the logs are not just large. They are also noisy.

If the file is growing unusually fast, you may also want to reduce how much MongoDB writes in the first place.

One conservative option is to add

systemLog:
  destination: file
  path: /mnt/disk-log/mongodb/mongod.log
  logAppend: true
  logRotate: reopen
  quiet: true
Code language: JavaScript (javascript)

The quiet setting reduces certain routine log messages. It will not eliminate important warnings or errors, but it can help cut down unnecessary chatter on busy systems.

At the same time, make sure you are not running with increased debug verbosity. Unless you are actively troubleshooting, there is no reason to raise MongoDB 8.x.x log verbosity above the default.

Higher verbosity levels can cause logs to balloon much faster than expected.

When Big Logs Are Actually a Symptom

Sometimes a giant MongoDB log is not just a rotation problem. It is also a sign that the server is trying to tell you something repeatedly.

Common causes of excessive MongoDB log growth include

  • Repeated authentication failures
  • Noisy connection attempts from applications or monitoring tools
  • Replication instability or frequent state changes
  • Aggressive health checks
  • Temporary debugging settings left enabled
  • Problematic clients connecting and disconnecting in high volume

So while log rotation is the immediate fix, it is also worth glancing through the log itself before archiving everything away. If the same message appears thousands of times, the real problem may be somewhere else.

A small log that rotates normally is healthy, a huge log that rotates normally may still be warning you.

How to Apply and Test It

On Debian, once you save the logrotate file, test it before waiting for the daily scheduler:

sudo logrotate -d /etc/logrotate.d/mongod

That performs a dry run.

If the output looks correct, force a rotation test:

sudo logrotate -f /etc/logrotate.d/mongod

Then verify the result

ls -lh /mnt/disk-log/mongodb/

You should see the active mongod.log plus rotated files, and older ones may appear compressed depending on your settings.

This simple validation step is worth doing immediately. It confirms that the file path is correct, ownership is correct, and MongoDB 8.x.x is reopening the log as expected.

A Practical Recommendation

If you are running MongoDB 8.x.x on Debian 12/13 and storing logs in a custom path, the safest practical setup is this

Use logAppend: true
Use logRotate: reopen
Manage rotation with Debian logrotate
Signal MongoDB with SIGUSR1 after rotation
Optionally enable quiet: true if routine log noise is excessive

That gives you predictable log growth, retained history, compressed archives, and no surprise disk explosion months later.

This is not a flashy optimization. It is not the kind of change that makes dashboards look better or benchmark scores improve. But it is exactly the kind of quiet operational hygiene that keeps a production server sane.

And often, that is what matters most.

How to Upgrade Wazuh on Ubuntu 24.04 and Fix a wazuh-manager Timeout During the Process

Upgrading Wazuh on Ubuntu 24.04 should be routine. Refresh the repository, install the updated packages, restart the services, and move on.

But sometimes an upgrade does not fail loudly. It stumbles at the very end, in a way that makes the whole platform look broken even though most of it has already upgraded successfully.

That was the situation here.

The environment was upgraded from Wazuh 4.13.1 to 4.14.4 on Ubuntu 24.04, using a deployment that had originally been installed with the Wazuh installation assistant, not Docker.

The package upgrade downloaded and unpacked correctly. The dashboard came back. The indexer came back. Filebeat came back. But the process still stopped with an error because wazuh-manager did not complete its post-install step.

At first glance, it felt like a failed upgrade. In reality, it was much narrower than that.

The real issue was that wazuh-manager needed more time to start than the default systemd unit allowed. Once that was understood, the fix became simple and the upgrade completed cleanly.

The Original Installation Method Matters

This server had originally been installed using the Wazuh assistant script:

curl -sO https://packages.wazuh.com/4.13/wazuh-install.sh && sudo bash ./wazuh-install.sh -a
Code language: JavaScript (javascript)

That matters because once Wazuh is already installed, the correct way to upgrade it is not to re-run the installer script. The correct path is to upgrade the installed packages through APT.

In this case, the target versions were:

  • wazuh-manager4.14.4-1
  • wazuh-indexer4.14.4-1
  • wazuh-dashboard4.14.4-1
  • filebeat7.10.2-2

Updating the Wazuh Repository on Ubuntu 24.04

The first step was to make sure the server used the current Wazuh 4.x repository.

curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH \
  | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import

sudo chmod 644 /usr/share/keyrings/wazuh.gpg

echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" \
  | sudo tee /etc/apt/sources.list.d/wazuh.list

sudo apt-get update
Code language: JavaScript (javascript)

Then the upgrade itself was started with

sudo apt-get install wazuh-manager wazuh-indexer wazuh-dashboard filebeat
Code language: JavaScript (javascript)

At this point, the process looked healthy. Packages downloaded, unpacked, and started configuring. But at the end, the upgrade stopped at the most uncomfortable point possible, almost finished, yet still incomplete.

The package manager reported an error on wazuh-manager.

The Symptom That Makes the Upgrade Look Worse Than It Is

After the upgrade attempt, the package state told the story.

dpkg -l | egrep 'wazuh-manager|wazuh-indexer|wazuh-dashboard|filebeat'
Code language: JavaScript (javascript)

The result looked like this

ii  filebeat
ii  wazuh-dashboard
ii  wazuh-indexer
iF  wazuh-manager

That iF state is the important clue. It means the package is installed, but configuration failed.

This created a confusing situation. Three components were already upgraded, but the whole upgrade still could not be considered complete because wazuh-manager had not successfully finished its post-install phase.

At the same time, the dashboard still opened normally in the browser. Login worked. But once inside, the Wazuh app reported that no API was available.

That can be misleading when you first see it. The dashboard frontend may be alive, but the Wazuh Manager API behind it can still be unavailable.

Why the Dashboard Can Work While Wazuh Is Still Broken

This is one of the most deceptive parts of the process.

wazuh-dashboard can start independently and still present the login page, even when the Wazuh Manager service behind the API is not fully ready. So the ability to log in does not prove that the whole platform is healthy.

If the dashboard loads but complains that there is no API available, the first place to look is not the dashboard. It is wazuh-manager.

That is where the actual problem was hiding.

Checking the Manager Service Properly

The first useful command was:

sudo systemctl status wazuh-manager --no-pager -l

This showed something subtle but important. wazuh-manager was not crashing immediately. It was beginning to start, loading modules, spawning API-related Python processes, and then getting terminated by systemd before it had fully completed startup.

That distinction matters.

This was not a corrupted package or a fatal binary crash. It was a startup timing problem.

Inspecting the unit file made the reason much clearer:

sudo systemctl cat wazuh-manager

The default unit included:

[Service]
Type=forking
LimitNOFILE=65536
TimeoutSec=45

There it was. 45 seconds.

On a platform like Wazuh, after a version upgrade, that may simply not be enough time for wazuh-manager to complete initialization, especially when several internal processes need to come back up together.

The Real Fix, Increase the systemd Startup Timeout

Once the problem was understood, the solution was straightforward, let wazuh-manager have more time to start.

Create a persistent systemd override

sudo systemctl edit wazuh-manager

Then add the following:

[Service]
TimeoutStartSec=300
TimeoutStopSec=300

Save the file, then reload systemd

sudo systemctl daemon-reload

This creates an override file at

/etc/systemd/system/wazuh-manager.service.d/override.conf

You can verify it with

sudo systemctl cat wazuh-manager

Once the override is in place, it remains persistent across reboots, restarts, and future service starts.

Restarting the Manager After the Override

With the longer timeout applied, restarting the manager allowed it to complete startup properly

sudo systemctl restart wazuh-manager
sudo systemctl status wazuh-manager --no-pager -l

This time, the manager reached the state it had been trying to reach all along

Active: active (running)
Code language: HTTP (http)

And beneath it, all the expected internal Wazuh processes appeared

  • wazuh-authd
  • wazuh-db
  • wazuh-execd
  • wazuh-analysisd
  • wazuh-syscheckd
  • wazuh-remoted
  • wazuh-logcollector
  • wazuh-monitord
  • wazuh-modulesd
  • multiple wazuh_apid.py processes

At that point, the upgrade was no longer blocked by service startup. What remained was simply finishing the interrupted package configuration.

Completing the Upgrade Cleanly

Once wazuh-manager was actually running, the pending package configuration could be completed

sudo dpkg --configure -a

This is the step that finally clears the half-finished package state.

Checking package versions again showed the clean result:

dpkg -l | egrep 'wazuh-manager|wazuh-indexer|wazuh-dashboard|filebeat'
Code language: JavaScript (javascript)

Now the output showed

ii  filebeat
ii  wazuh-dashboard
ii  wazuh-indexer
ii  wazuh-manager

That ii status across all packages is what confirms the upgrade is fully complete.

Final Validation Steps

Once the package state was clean, I still wanted to verify the platform as a whole.

First, check that all services are active

systemctl is-active wazuh-manager wazuh-indexer wazuh-dashboard filebeat

The ideal result is

active
active
active
active

Then test the API locally

curl -k https://127.0.0.1:55000
Code language: JavaScript (javascript)

If the API responds and the dashboard no longer shows No API available to connect, then the platform is back in a healthy state.

The Small Lesson Hidden Inside This Upgrade

What made this upgrade frustrating was not that it failed completely. It was that it failed only partially.

That is often worse.

A full crash gives clarity. A partial success creates doubt. The dashboard works, but not really. The manager starts, but not completely. The packages are upgraded, but not cleanly. Everything looks almost fine, which makes troubleshooting slower than it should be.

In this case, the fix was not a reinstall, not a rollback, and not a deep rebuild of the stack. It was simply allowing wazuh-manager enough time to finish starting, then letting dpkg complete the unfinished package configuration.

Sometimes that is all a stubborn upgrade needs.

Command Summary

For quick reference, here is the complete flow that worked

curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH \
  | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import

sudo chmod 644 /usr/share/keyrings/wazuh.gpg

echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" \
  | sudo tee /etc/apt/sources.list.d/wazuh.list

sudo apt-get update
sudo apt-get install wazuh-manager wazuh-indexer wazuh-dashboard filebeat

sudo systemctl edit wazuh-manager
sudo systemctl daemon-reload
sudo systemctl restart wazuh-manager

sudo dpkg --configure -a

dpkg -l | egrep 'wazuh-manager|wazuh-indexer|wazuh-dashboard|filebeat'
systemctl is-active wazuh-manager wazuh-indexer wazuh-dashboard filebeat
curl -k https://127.0.0.1:55000
Code language: PHP (php)

Closing Thought

Ubuntu 24.04 is stable, modern, and perfectly capable of running Wazuh well. But not every post-upgrade service startup will fit neatly into the timeout values inherited from a package unit file.

Sometimes the service is healthy, the platform is healthy, and the only thing missing is enough time.

In this case, once wazuh-manager was allowed to finish starting, everything else followed naturally.

That is what made this upgrade feel broken at first, yet ultimately turn out to be completely recoverable.

How to Sync Only MariaDB Users and Passwords from Master to Slave

Sometimes I want a MariaDB slave to have the same user accounts and passwords as the master, but I do not want it to inherit the same permissions.

This is common when the slave is used for reporting, internal access, standby, or a limited failover role. The accounts may need to match, but the grants should remain different.

In that case, the goal is simple: sync only the users and password hashes, and leave the slave’s privileges alone.

MariaDB Is Slightly Different from Old MySQL Habits

In modern MariaDB, user accounts, passwords, and global privileges are stored in mysql.global_priv. The older mysql.user still exists, but it is now a view that references mysql.global_priv. (MariaDB)

That detail matters because many older MySQL-oriented guides still treat mysql.user as the main storage table. In MariaDB 10.4 and newer, that is no longer the real source of truth. MariaDB’s own documentation states that account data is now stored in mysql.global_priv, while mysql.user remains for compatibility. (MariaDB)

So for this use case, the main object to keep in mind is:

mysql.global_priv
Code language: CSS (css)

What You Should Not Sync

If the slave must keep different permissions, do not copy the privilege tables that define access rules beyond the account itself.

Examples include

mysql.db
mysql.tables_priv
mysql.columns_priv
mysql.procs_priv
mysql.proxies_priv
mysql.roles_mapping
Code language: CSS (css)

MariaDB documents mysql.db as database-level privileges and mysql.tables_priv as table-level privileges, while GRANT stores global privileges in mysql.global_priv and database privileges in mysql.db. (MariaDB)

If you copy those tables, you are no longer syncing only users and passwords. You are also cloning authorization behavior, which is exactly what you said you do not want.

The Better Method

The cleaner method is to generate SQL statements from the master and run them on the slave, that means.

  • use the master as the source of account information
  • generate CREATE USER statements if accounts do not exist yet
  • generate ALTER USER statements if accounts already exist
  • keep password hashes intact
  • do not import the other grant tables

This approach is safer than copying system tables directly, especially when master and slave may not be perfectly identical in role or version.

MariaDB documentation also recommends using account management statements such as CREATE USER and GRANT instead of directly updating privilege tables. (MariaDB)

Step 1, Check the Accounts on the Master

Use this query to list the accounts from the actual privilege storage layer:

SELECT Host, User
FROM mysql.global_priv
ORDER BY User, Host;
Code language: CSS (css)

That confirms which accounts exist on the master. mysql.global_priv is the real table that stores users, account properties, and global privileges in modern MariaDB. (MariaDB)

Step 2: Read the Authentication Data

For practical export work, mysql.user can still be useful because it is exposed in a familiar format as a compatibility view. MariaDB says this view exists specifically so tools that inspect mysql.user can continue to work. (MariaDB)

A practical query is:

SELECT User, Host, plugin, authentication_string
FROM mysql.user
ORDER BY User, Host;
Code language: CSS (css)

This gives you the account, host, authentication plugin, and password-related value where applicable. (MariaDB)

Step 3, Generate CREATE USER Statements

If the users do not exist yet on the slave, generate SQL on the master like this.

SELECT CONCAT(
  'CREATE USER IF NOT EXISTS ''', User, '''@''', Host,
  ''' IDENTIFIED BY PASSWORD ''', authentication_string, ''';'
) AS stmt
FROM mysql.user
WHERE User <> ''
  AND authentication_string <> ''
ORDER BY User, Host;
Code language: PHP (php)

MariaDB documents CREATE USER and supports using password hashes with IDENTIFIED BY PASSWORD. It also notes that creating an account updates the mysql.user view and the underlying mysql.global_priv table. (MariaDB)

Save the output, review it, and run it on the slave.

Step 4, Generate ALTER USER Statements

If the accounts already exist on the slave, it is usually better to update only the passwords

SELECT CONCAT(
  'ALTER USER ''', User, '''@''', Host,
  ''' IDENTIFIED BY PASSWORD ''', authentication_string, ''';'
) AS stmt
FROM mysql.user
WHERE User <> ''
  AND authentication_string <> ''
ORDER BY User, Host;
Code language: PHP (php)

Use this when you want to keep the existing account structure on the slave and only refresh the password hashes. MariaDB documents ALTER USER for modifying authentication settings and passwords. (MariaDB)

Important Caveat, Not Every Account Stores a Password Hash

This part is easy to overlook.

MariaDB documents that some authentication plugins, including unix_socket, named_pipe, gssapi, and pam, do not store passwords in mysql.global_priv the way password-based accounts do. For those accounts, there may be no usable password hash to export with this method. (MariaDB)

So this method is best for normal password-based users. If an account authenticates through socket or external auth, you may need to recreate it with the same plugin instead of trying to sync a password hash.

Why This Works Well

This method gives you exactly the level of sync you want.

Do sync

  • account names
  • host definitions
  • password hashes
  • authentication settings where relevant

Do not sync

  • database grants
  • table grants
  • column grants
  • routine grants
  • role mappings

That keeps the slave aligned at the login level, but independent at the authorization level. For a reporting slave or restricted standby server, that is usually the correct design. (MariaDB)

My Practical Recommendation

If your requirement is to “Make the slave use the same MariaDB users and passwords as the master, but keep different permissions on the slave,”

Then this is the path I would use.

  1. Query the master for users and authentication data.
  2. Generate CREATE USER statements for accounts that do not exist on the slave.
  3. Generate ALTER USER statements for accounts that already exist.
  4. Apply only those statements on the slave.
  5. Do not dump or restore the other privilege tables.

It is focused, easy to review, and much safer than cloning the full mysql privilege setup.

Final Thought

This is one of those cases where MariaDB looks familiar but behaves differently enough that older MySQL habits can be misleading.

If you only need accounts and passwords, do not copy more than necessary. Use MariaDB’s account statements, keep the slave’s grants separate, and make the synchronization as narrow as possible.

That keeps the setup cleaner and avoids accidentally changing access rules on the replica.

Useful Queries Recap

List accounts from the real MariaDB privilege store

SELECT Host, User
FROM mysql.global_priv
ORDER BY User, Host;
Code language: CSS (css)

Read account authentication data through the compatibility view

SELECT User, Host, plugin, authentication_string
FROM mysql.user
ORDER BY User, Host;
Code language: CSS (css)

Generate CREATE USER statements

SELECT CONCAT(
  'CREATE USER IF NOT EXISTS ''', User, '''@''', Host,
  ''' IDENTIFIED BY PASSWORD ''', authentication_string, ''';'
) AS stmt
FROM mysql.user
WHERE User <> ''
  AND authentication_string <> ''
ORDER BY User, Host;
Code language: PHP (php)

Generate ALTER USER statements

SELECT CONCAT(
  'ALTER USER ''', User, '''@''', Host,
  ''' IDENTIFIED BY PASSWORD ''', authentication_string, ''';'
) AS stmt
FROM mysql.user
WHERE User <> ''
  AND authentication_string <> ''
ORDER BY User, Host;
Code language: PHP (php)

Source Notes

MariaDB documents that mysql.global_priv stores user accounts and global privileges, and that mysql.user remains as a compatibility view in modern versions.

It also documents CREATE USER, ALTER USER, and the cases where some authentication plugins do not store password hashes in the same way. (MariaDB)

Moving a MariaDB Database Between Servers (The Simple Way)

Sometimes the hardest problems in infrastructure are actually the simplest ones. Not because they are complicated, but because we forget the exact commands when we need them.

Migrating a MariaDB database from one server to another is one of those things. You’ve done it many times, but when you need it again six months later, you end up searching your own notes.

Not as dificlut or tedius restoring master slave without GTID, but sometimes I forgot how to reproduce the steps.

So here’s a quick and practical note for future me (and maybe useful for others too).

The Classic Method: Dump > Transfer > Restore

The safest and most common approach is:

  1. Dump the database from the old server
  2. Copy the dump file to the new server
  3. Restore the database

Simple and reliable.

Step 1, Dump the Database

On the source server, create a dump file.

Basic command

mysqldump -u root -p dbname > dbname.sql
Code language: CSS (css)

But for production systems, it’s better to use a few additional flags.

mysqldump -u root -p \
  --single-transaction \
  --quick \
  --routines \
  --triggers \
  --events \
  dbname > dbname.sql
Code language: CSS (css)

Why these options?

  • –single-transaction
    Keeps the dump consistent without locking tables (important for InnoDB).
  • –quick
    Streams rows instead of loading everything into memory.
  • –routines / triggers / events
    Includes stored procedures, triggers, and scheduled events.

Without these flags, some database logic might silently disappear.

Step 2, Transfer the Dump File

Copy the dump file to the new server, You can use FIlezilla, SCP, XFtp and others

Example using scp.

scp dbname.sql user@newserver:/tmp/
Code language: JavaScript (javascript)

If the database is large, compress it first.

mysqldump -u root -p dbname | gzip > dbname.sql.gz
scp dbname.sql.gz user@newserver:/tmp/
Code language: JavaScript (javascript)

Compression can reduce transfer time significantly.

Step 3, Create the Database on the New Server

Login to MariaDB on the destination server:

mysql -u root -p

Create the database:

CREATE DATABASE dbname
  CHARACTER SET utf8mb4
  COLLATE utf8mb4_unicode_ci;

Then exit the MariaDB console.

Step 4, Restore the Database

Restore from the SQL dump:

mysql -u root -p dbname < dbname.sql
Code language: CSS (css)

If you used compression:

gunzip < dbname.sql.gz | mysql -u root -p dbname

After this finishes, your database should be fully restored.

Dumping All Databases

If you need to migrate the entire server, use:

mysqldump -u root -p --all-databases > alldb.sql
Code language: CSS (css)

And restore with:

mysql -u root -p < alldb.sql
Code language: CSS (css)

Alternatives, direct Server to Server Migration

If both servers can connect via SSH, you can skip the intermediate file entirely.

mysqldump -u root -p dbname | ssh user@newserver "mysql -u root -p dbname"
Code language: JavaScript (javascript)

This streams the dump directly into the new database, It’s fast and very convenient for quick migrations.

Things I Always Double-Check

After restoring a database, I usually run a few quick checks.

List tables:

SHOW TABLES;

Check approximate row counts:

SELECT table_name, table_rows
FROM information_schema.tables
WHERE table_schema='dbname';
Code language: JavaScript (javascript)

Just to make sure nothing obvious is missing.

Final Thoughts

Database migrations sound like a big operation, but most of the time they come down to a very simple workflow:

dump > copy > restore

It’s one of those small operational tasks that every sysadmin or developer does regularly, yet still forgets the exact commands.

So this post is mostly a note for my future self, because simple things deserve good documentation too.

Docker SFTP Port Works on localhost but Times Out from Outside (Debian 12 + ferm + Linode NAT)

I hit a classic “it must be Docker” moment.

  • The container was healthy, the port was clearly listening and nc 127.0.0.1 22222 said open.
  • But from outside the server, connections to PUBLIC_IP:22222 timed out.
  • And the weirdest part, if I stopped ferm, it worked instantly.

This post documents what was actually happening and the minimal, correct fix.

The setup

  • Host is Debian 12
  • Firewall, ferm (tight rules, default DROP)
  • Provider, Linode with NAT
  • Host SSH port, 22222
  • Docker SFTP container, atmoz/sftp published as 22222:22

Symptom, “Local open, remote timeout”

On the server:

sudo ss -lntp | egrep ':(22|22222)\b' || true
sudo docker ps --format 'table {{.Names}}\t{{.Ports}}'
nc -vz 127.0.0.1 22222
Code language: JavaScript (javascript)

Everything looked correct: Docker was listening on 0.0.0.0:22222, and localhost could connect.

But from outside, connections to port 22222 timed out.

First principle, don’t guess, but watch packets

On the server, I ran

sudo tcpdump -ni any tcp port 22222

Then I attempted to connect from outside.

I saw repeated SYN packets arriving (something like):

IP <client_ip>.<port> > <server_private_ip>.22222: Flags [S] ...
Code language: HTML, XML (xml)

That single observation is huge

  • The NAT / provider network was not blocking the port.
  • Packets reached the server.
  • The server simply did not respond (no SYN-ACK), this is almost always a firewall/ruleset problem.

Why allowing INPUT is not enough for Docker-published ports

This is the key concept.

When you publish a Docker port like this:

  • Host is 22222
  • and Container is 22

The incoming packet flow is roughly

  1. Client connects to PUBLIC_IP:22222
  2. Host receives the packet and Docker applies DNAT to the container:
    • destination becomes 172.18.x.y:22
  3. That packet must now traverse the FORWARD path to the Docker bridge interface (br-...)

So even if you allow 22222 in INPUT, the connection can still fail if your firewall has:

  • FORWARD policy DROP (very common in hardened ferm setups)
  • and no exception for forwarding traffic to Docker networks

That was my case.

The fix, allow FORWARD to the Docker bridge (port 22 after DNAT)

After DNAT, the traffic is targeting container port 22 (not 22222).
So the correct fix is a FORWARD rule that allows forwarding to the Docker bridge interface for TCP dport 22.

Below is a complete example ferm.conf you can use as a reference.

Full ferm.conf example (with 22060 + 22061)

Replace:

  • YOUR_ADMIN_IPv4 with your real public IP (or your office/VPN IP)
  • YOUR_WG_IPv4 if you have another trusted source
  • br-xxxxxxxxxxxx with your Docker bridge name (find it via ip -br a or docker network ls + ip link)
domain (ip ip6) {
  table filter {

    chain INPUT {
      policy DROP;

      # Allow established connections and localhost
      mod state state (ESTABLISHED RELATED) ACCEPT;
      interface lo ACCEPT;

      # Allow ping
      proto icmp ACCEPT;
      @if @eq($DOMAIN, ip6) {
        proto ipv6-icmp ACCEPT;
      }

      # SSH and Docker SFTP from a trusted public IP (example)
      @if @eq($DOMAIN, ip) {
        # Admin/VPN/Office IP
        proto tcp saddr YOUR_ADMIN_IPv4 dport 22221 ACCEPT;   # SSH on 22221
        proto tcp saddr YOUR_ADMIN_IPv4 dport 22222 ACCEPT;   # Docker-published SFTP on 22061

        # Optional second trusted source (example)
        proto tcp saddr YOUR_WG_IPv4 dport 22222 ACCEPT;
      }

      # Optional IPv6 allow examples (fill as needed)
      @if @eq($DOMAIN, ip6) {
        # proto tcp saddr YOUR_ADMIN_IPv6 dport 22221 ACCEPT;
      }

      LOG log-prefix "ferm INPUT drop: " log-level warning;
      DROP;
    }


    chain OUTPUT {
      policy ACCEPT;
      mod state state (ESTABLISHED RELATED) ACCEPT;
    }


    chain FORWARD {
      policy DROP;

      # Allow return traffic
      mod state state (ESTABLISHED RELATED) ACCEPT;

      # IMPORTANT:
      # Docker port publishing (22222 -> container:22) needs FORWARD allowance to the Docker bridge.
      # After DNAT, destination port is 22, and the packet is forwarded to br-xxxxxxxxxxxx.
      @if @eq($DOMAIN, ip) {
        # Allow ONLY from the same trusted admin IP
        outerface br-xxxxxxxxxxxx proto tcp saddr YOUR_ADMIN_IPv4 dport 22 ACCEPT;
      }

      LOG log-prefix "ferm FORWARD drop: " log-level warning;
      DROP;
    }

  }
}

Apply safely

  1. Dry-run syntax check
sudo ferm -n /etc/ferm/ferm.conf
  1. Restart ferm
sudo systemctl restart ferm

How to find the Docker bridge name

Run:

ip -br a

Look for an interface like:

  • br-93736b42dd62 UP 172.18.0.1/16

That br-93736b42dd62 is what you put into the FORWARD rule as outerface.

Verify the fix

From outside:

  • sftp -P 22222 user@PUBLIC_IP
  • or ssh -p 22222 user@PUBLIC_IP (you should see SSH negotiation if reachable)

On the server, you can confirm handshake progress:

sudo tcpdump -ni any tcp port 22061

You should start seeing SYN followed by SYN-ACK and further packets.

Notes on “Do I need to whitelist the NAT?”

In this case: no.

tcpdump showed the real client IP as the source. The NAT was forwarding traffic to the server, and the server was dropping it on the forwarding path. The correct whitelist target is your client IP, and the correct fix is allowing FORWARD to Docker bridge.

Reseeding MariaDB Master–Slave Replication (Non-GTID) While the Master Stays Online

MariaDB replication breaks at the worst possible time, right when you need the database to be calm, predictable, and boring.

If you’re running classic (non-GTID) master–slave replication, the fastest way out of a messy drift is usually not to “skip errors” or “restart services” endlessly.

The cleanest recovery is to reseed the slave from a fresh snapshot of the master, then let replication catch up from the exact binlog coordinates that match that snapshot.

The good news, you can do all of that while the master keeps running.

This article is a practical runbook for reseeding non-GTID replication using mysqldump with --single-transaction and --master-data=2, plus one extra option that makes the whole process safer: --add-drop-table.

Why reseed instead of “fixing” replication?

When replication breaks, you might see errors like

  • Duplicate key / duplicate entry
  • Cannot find record (missing row)
  • Foreign key issues
  • Random drift that keeps reappearing after you “skip” one event

Those symptoms usually mean one thing: the slave is no longer an exact logical copy of the master.

In that situation, skipping transactions might get you a green status for a few minutes, but it also guarantees you’ll carry silent inconsistencies forward.

A reseed resets the slave to a known-good snapshot and restores a clean replication path.

The fear, “but the master is still running and the binlog keeps moving”

Yes, the binlog will keep moving. That’s normal.

When you dump the master with:

  • --single-transaction (consistent snapshot for InnoDB), and
  • --master-data=2 (records the master’s binlog file+position that matches that snapshot)

…the dump file contains a single “bookmark” for where replication should start. Even if the master continues writing changes during the dump, the slave can start from that recorded position and replay everything that happened after the snapshot.

What --master-data=2 actually does

--master-data=2 writes the master coordinates inside the dump file as a comment, like this:

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000123', MASTER_LOG_POS=456789;
Code language: JavaScript (javascript)

The key part is: it matches the snapshot. It’s the exact point-in-time reference you use to configure the slave.

Why =2? Because it’s safer. With =2, the command is not executed automatically during import; you extract it and run CHANGE MASTER TO manually.

The clean reseed steps (non-GTID)

1, Stop replication and reset slave metadata (on the SLAVE)

STOP SLAVE;
RESET SLAVE ALL;

At this point, you’ve cleared the old master connection info and relay logs. You’re ready to load a fresh snapshot.

2, Dump the master (on the MASTER)

Dump all replicated databases in one file (recommended). Replace db1 db2 db3 with your actual DB list.

mysqldump -u root -p \
  --single-transaction \
  --routines --triggers --events \
  --master-data=2 \
  --add-drop-table \
  --databases db1 db2 db3 > /tmp/reseed.sql
Code language: JavaScript (javascript)

Why these flags?

  • --single-transaction
    Gives a consistent snapshot without locking for InnoDB tables.
  • --master-data=2
    Records the exact binlog file/pos for this snapshot.
  • --add-drop-table
    Makes the dump drop existing tables on the slave before recreating them — useful if you prefer not to manually drop databases.
  • --routines --triggers --events
    Because production databases aren’t just tables.

Sample how You copy it to the slave, you can use other method.

scp /tmp/reseed.sql user@SLAVE_IP:/tmp/reseed.sql
Code language: JavaScript (javascript)

3, Import the dump (on the SLAVE)

mysql -u root -p < /tmp/reseed.sql
Code language: JavaScript (javascript)

Because we included --add-drop-table, the import will overwrite tables cleanly (inside the databases you included in the dump), without you having to manually DROP DATABASE.

4, Extract the binlog coordinates from the dump (on the SLAVE)

grep -m 1 "CHANGE MASTER TO" /tmp/reseed.sql
Code language: JavaScript (javascript)

You’ll get a line similar to:

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000123', MASTER_LOG_POS=456789;
Code language: JavaScript (javascript)

5, Point the slave to the master, then start replication (on the SLAVE)

Replace host/user/password and the file/pos.

CHANGE MASTER TO
  MASTER_HOST='MASTER_IP',
  MASTER_PORT=3306,
  MASTER_USER='repl',
  MASTER_PASSWORD='repl_password',
  MASTER_LOG_FILE='mysql-bin.000123',
  MASTER_LOG_POS=456789;

START SLAVE;
SHOW SLAVE STATUS\G
Code language: JavaScript (javascript)

Success means:

  • Slave_IO_Running: Yes
  • Slave_SQL_Running: Yes
  • Last_IO_Error: empty
  • Last_SQL_Error: empty

Do you need to reseed per database?

No , not conceptually.

If you replicate multiple databases, you’ll get the cleanest result by making one dump containing all replicated databases, that ensures they all reflect a consistent snapshot.

Multiple separate dumps taken at different times can create weird cross-database inconsistencies, especially if your app writes across schemas.

Notes and sharp edges

1, InnoDB vs MyISAM/Aria

--single-transaction is ideal for InnoDB. If you still have MyISAM/Aria tables that are actively changing, you may need a different strategy (brief read lock, or convert them).

2, Keep the slave read-only

A replica that accepts writes becomes a future incident waiting to happen. If you can, keep it read-only:

SET GLOBAL read_only=ON;
SET GLOBAL super_read_only=ON;
Code language: PHP (php)

3, Replication filters matter

If your slave uses replicate-do-db or wildcard filters, align your mysqldump --databases ... list to the same scope, dump what you replicate.

Closing thought

When replication breaks, it’s tempting to keep applying band-aids until the status turns green.

But green isn’t the same as correct.

A reseed is one of those maintenance tasks that feels heavy… until you do it once, watch it work, and realize you just bought back your peace of mind.

Install Node.js 24 & PM2 on Debian 13, Including Log Rotation

If you’re running Debian 13 for a production server, Node.js 24 + PM2 is a practical setup, simple deployment, easy process management, and predictable restarts.

Below is a production-friendly installation guide, including PM2 log rotation.

1. Install Node.js 24 (via NodeSource)

Update packages and install prerequisites:

sudo apt update
sudo apt install -y ca-certificates curl gnupg

Add the NodeSource GPG key:

curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key \
  | sudo gpg --dearmor -o /usr/share/keyrings/nodesource.gpg
Code language: JavaScript (javascript)

Add the Node.js 24 repository:

echo "deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" \
  | sudo tee /etc/apt/sources.list.d/nodesource.list
Code language: PHP (php)

Install Node.js:

sudo apt update
sudo apt install -y nodejs

Verify:

node -v
npm -v

2. (Optional) Install Build Tools

Some npm packages (native modules) require compilation

sudo apt install -y build-essential python3 make g++

3) Install PM2

Install PM2 globally

sudo npm install -g pm2

Verify:

pm2 -v

4. Run Your App with PM2

If your app uses npm start

cd /var/www/myapp
pm2 start npm --name "myapp" -- start
Code language: JavaScript (javascript)

Check status

pm2 status

View logs:

pm2 logs myapp

5) Install PM2 Log Rotation (pm2-logrotate)

This prevents logs from growing endlessly and filling up disk space.

Install the module, donot use ‘sudo’, because it will install and run under roor, not current user

pm2 install pm2-logrotate

Confirm it’s installed

pm2 status
pm2 show pm2-logrotate

pm2 conf pm2-logrotate

6. Recommended Log Rotation Settings (Production-Friendly)

Example: rotate when logs reach 50MB, keep 14 files, compress old logs.

pm2 set pm2-logrotate:max_size 50M
pm2 set pm2-logrotate:retain 14
pm2 set pm2-logrotate:compress true
pm2 set pm2-logrotate:dateFormat YYYY-MM-DD
pm2 set pm2-logrotate:workerInterval 30
Code language: JavaScript (javascript)

Optional: rotate on a schedule (cron). Example: rotate daily at midnight

pm2 set pm2-logrotate:rotateInterval '0 0 * * *'
Code language: JavaScript (javascript)

Restart processes to keep everything clean:

pm2 save
pm2 restart all

7) Where PM2 Stores Logs

By default, PM2 logs live here:

~/.pm2/logs/
Code language: JavaScript (javascript)

Quick check

ls -lah ~/.pm2/logs/
Code language: JavaScript (javascript)

8) Uninstall Log Rotation (If Needed)

pm2 uninstall pm2-logrotate

Debugging Slow Debian 13 on WSL, When a Single Ping Tells the Whole Story

Sometimes the simplest command reveals the truth hiding beneath the surface. My Debian 13 environment inside WSL had been unusually slow for days.

Not broken, not throwing warnings, but hanging on apt update long enough for me to lose patience.

“Reading package lists…” would freeze, repository connections took forever, and every small task felt like it was wading through molasses.

I suspected DNS, but I’ve learned not to blame DNS until I see the evidence myself. So I reached for the most basic diagnostic tool,

ping.

I started with Cloudflare’s 1.1.1.1 to see if the outbound path was clean.

PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: time=39.1 ms
64 bytes from 1.1.1.1: time=27.7 ms
64 bytes from 1.1.1.1: time=34.9 ms
Code language: JavaScript (javascript)

Stable, no packet loss, nothing suspicious. Then I pinged the Debian Fastly mirror.

PING debian.map.fastlydns.net (***) 56(84) bytes of data.
64 bytes from ***: time=19.1 ms
64 bytes from ***: time=16.4 ms
64 bytes from ***: time=486 ms
Code language: JavaScript (javascript)

The story appears in the third packet, almost half a second of delay. That single blip told me everything that apt had been screaming silently.

The network inside the office wasn’t clean. Something,congestion, traffic shaping, SSL inspection, policy based routing, was introducing inconsistent latency.

For regular browsing nobody would notice, but package managers certainly do.

At this point the problem wasn’t Debian and it wasn’t WSL. It was the inconsistent outbound path to public mirrors.

But inconsistency doesn’t mean helplessness.

There are ways to minimize the damage, and that’s when I started adjusting the environment rather than fighting the network.

The first fix was preventing WSL from overwriting /etc/resolv.conf every time it starts.

WSL loves injecting Windows DNS resolvers, and in many corporate networks, those DNS servers are either filtered or slow.

Inside Debian I created

sudo nano /etc/wsl.conf

And added

[network]
generateResolvConf = false
Code language: JavaScript (javascript)

After shutting down WSL from Windows:

wsl --shutdown

I created my own resolver file:

sudo rm -f /etc/resolv.conf
sudo nano /etc/resolv.conf

And set Cloudflare manually:

nameserver 1.1.1.1
nameserver 1.0.0.1
options edns0
Code language: CSS (css)

Then I locked it so nothing, neither WSL nor a package, could silently revert it

sudo chattr +i /etc/resolv.conf

With DNS fixed, I turned my attention to the next silent culprit, IPv6.

Corporate networks often broadcast IPv6 routes that don’t actually work.

apt tries IPv6 first, fails silently, and gives the illusion of “stuck on reading headers.”

So I forced apt to use IPv4.

echo 'Acquire::ForceIPv4 "true";' | sudo tee /etc/apt/apt.conf.d/99force-ipv4
Code language: PHP (php)

After that, everything felt lighter. apt update flowed more smoothly, package installation no longer paused, and general network activity inside WSL became more predictable.

It didn’t magically bypass corporate constraints, but it turned an unusable system into a functional, stable environment.

What I love about this kind of debugging is how honest the system is if you listen carefully.

A single half-second ping spike can explain hours of inconsistent performance.

A resolver file quietly replaced by WSL can sabotage an entire session. An unused IPv6 route can stall apt without ever throwing an error.

All the clues were there, I just had to slow down enough to read them.

If you ever face slow Debian or Ubuntu inside WSL, especially in an office network, start simple.

Ping external DNS.

Ping your mirrors.

ook for jitter.

Then fix DNS, force IPv4, and regain control of /etc/resolv.conf.

Most of the time, the solution isn’t about changing Linux, it’s about understanding the path it must travel to reach the outside world.

And sometimes, truly, one ping is all you need to see the full picture.

MariaDB vs PostgreSQL, Two Open-Source Patterns of Freedom

When people compare MariaDB and PostgreSQL, most focus on performance, indexing, or JSON support. But beneath those metrics lies a deeper question, one about freedom patterns and how open-source software evolves when power, profit, and community intersect.

This is not just a database comparison, it’s a story about two philosophies that shaped how open source survives in the age of cloud giants.

The Fork that Saved Openness

MariaDB was born out of rebellion. When Oracle bought Sun Microsystems, and with it, MySQL, the open-source world panicked.

What if Oracle slowly closed off MySQL, turning it into a corporate product?

Michael “Monty” Widenius, MySQL’s original creator, didn’t wait to find out. He forked the code and launched MariaDB, named after his daughter.

But this wasn’t just a technical fork, it was an ideological one.

MariaDB Foundation was created with a single purpose, to keep the database free forever, every commit, every new feature, had to align with that principle.

That philosophy speaks to me deeply. I choose MariaDB for most of my project is not because it’s faster or lighter, but because it embodies the idea that open source must protect itself.

It’s the same instinct that keeps communities free from corporate capture, a belief that openness isn’t just about code, it’s about governance and continuity.

The Quiet Strength of Continuity

PostgreSQL took an unusual path. It wasn’t born from a corporation but from the POSTGRES research project at UC Berkeley, and it matured under the PostgreSQL License, permissive in the same family spirit as BSD/MIT, built to invite adoption, forks, experiments, and strange new ideas without asking permission first.

Its governance is still profoundly community-shaped, a small core team, a long tail of contributors, and decisions hammered out through consensus rather than executive decree.

That gives PostgreSQL a quiet kind of legitimacy, earned, not marketed. But let’s not romanticize the consequence, permissive licensing also means anyone can take the core, wrap it in a proprietary control plane, and sell you “PostgreSQL” as a product while the real value, monitoring, automation, scaling knobs, guardrails, lives behind a curtain.

You see it everywhere, AWS RDS for PostgreSQL, Azure Database for PostgreSQL, Google Cloud SQL for PostgreSQL, the same familiar engine, but the steering wheel is locked in a vendor’s glovebox.

And you see a more explicit version of the same gravity in projects like TimescaleDB, where part of the stack stays permissively open, and another part shifts into a more restrictive, source-available reality depending on the feature set.

Perfectly legal, perfectly rational and exactly the kind of “open on the brochure, closed in the cockpit” pattern that makes my skin itch.

Licenses as Philosophy

Here’s where the difference stops being academic and becomes personal. MariaDB lives under GNU GPL v2, strong copyleft, not as a vibe, but as a weaponized boundary.

It doesn’t mean you must publish your code just because you run MariaDB. It means this, if you ship a modified MariaDB as a product, if you distribute the derivative, you don’t get to hoard the source like a dragon.

You either give back, or you don’t play.

PostgreSQL, on the other hand, is permissive by design. It lets you modify, embed, repackage, and yes, close your version if that’s what your business model demands.

That freedom is powerful, sometimes beautiful, but it also leaves a wide-open door for enclosure, rebranding, and “innovation” that never comes home.

So when people say, “It’s just a license,” I laugh.

MariaDB enforces openness through law, PostgreSQL relies on culture.

One is a contract you can take to court. The other is hope, and I’ve been in this industry long enough to know hope is not a strategy, especially when corporate gravity starts pulling.

Copyleft isn’t a limitation. It’s a seatbelt, You don’t wear it because you plan to crash, You wear it because eventually someone will.

Ecosystem and Commercial Gravity

MariaDB’s ecosystem stays closer to its roots, Galera Cluster, Aria, ColumnStore, there’s an insistence that the heart remains shareable, inspectable, forkable.

While there’s a commercial company in the picture, the MariaDB Foundation exists as a custodian of the server project, an upstream anchor that’s meant to keep “open” from becoming a marketing costume the moment a quarterly report gets nervous.

PostgreSQL thrives in a much wider commercial universe, and that’s both its triumph and its vulnerability. The ecosystem is full of excellent work, some fully open source, some open-core, some unapologetically proprietary, especially at the hosted layer where the money really lives.

You get compatible engines, managed platforms, proprietary acceleration, closed operational tooling, and branded “Postgres” experiences that are mostly about everything around Postgres.

The database becomes the seed crystal, the vendor layer becomes the cathedral.

MariaDB feels more deliberate. It keeps the core under GPL even when that choice makes monetization harder, slower, less seductive.

That philosophy resonates with how I build systems, I’d rather pay the price of discipline now than pay the ransom of lock-in later.

Openness first, business second, and if business can’t survive that order, maybe the business model is the problem.

Freedom vs Flexibility

MariaDB’s copyleft model offers a legal shield against enclosure. PostgreSQL’s permissive model offers freedom to innovate but leaves the door open for fragmentation.

To me, it comes down to what kind of freedom you value. PostgreSQL gives you freedom to close. MariaDB gives you freedom to stay open.

I personally prefer the second, because once your foundation is closed, rebuilding openness becomes almost impossible.

Choosing the Path

When I build systems that I want to last, that others can audit, fork, and improve, I want that permanence to be enforced, not assumed. GPL gives that.

PostgreSQL trusts the world to stay kind, MariaDB assumes the world will change and protects itself accordingly.

That’s why, when I architect systems using Node.js, MongoDB, and Debian, MariaDB feels like a natural fit, philosophically consistent with how I see open source, not as a license choice, but as a commitment to the next generation of developers.

Two Patterns, One Goal

Both MariaDB and PostgreSQL are pillars of open-source freedom. One preserves it through law, the other through trust, both work, both matter.

But in my journey, I choose MariaDB, not because it’s easier or faster, but because its very existence is a reminder, freedom in software doesn’t just survive by trust, it survives by design.

Make MongoDB “Master” Primary Again on Debian 12 (MongoDB 8)

A practical guide for a 3-node replica set

  • 1 preferred master, 2 replicas ( do not mention slave, use replica ..)

MongoDB replica sets do not have a permanent “master.” MongoDB uses the term PRIMARY, and the PRIMARY can move after elections (restart, network hiccup, maintenance).

If your PRIMARY has shifted to a replica node and you want it back on your preferred “master” server, the right approach is to set election priority and trigger a controlled election.

In this article, server names are masked like this, oh this using Debian 12, I do not have master replica yet on Debian 13 Trixie

  • mongo1.internal:27017 = preferred “master” node (you want this to be PRIMARY, because maybe the resource and performance is better)
  • mongo2.internal:27017 = replica
  • mongo3.internal:27017 = replica (currently PRIMARY)

This is safe and standard for MongoDB 8 running on Debian 12.

Why MongoDB PRIMARY Moves ?

MongoDB automatically elects a PRIMARY based on health, replication freshness, votes, and priority. If your preferred node restarts later than the others, or briefly becomes unavailable, another member can win the election and stay PRIMARY.

Your goal is not to “force forever,” but to make mongo1 the preferred winner whenever it’s healthy and caught up.

Step 0, Connect with mongosh

Connect to the node that is currently PRIMARY (or any node, but you’ll apply changes on the current PRIMARY).

Example,

mongosh "mongodb://mongo3.internal:27017/?replicaSet=rs0"
Code language: JavaScript (javascript)

If you use authentication/TLS, keep your existing connection flags as-is.

Step 1, Check Current Replica Set State

Run these two checks, they tell you exactly who is PRIMARY and whether all members are healthy and caught up.

Command 1,

rs.conf().members
Code language: CSS (css)

Command 2,

rs.status().members.map(m => ({name:m.name, state:m.stateStr, health:m.health, optimeDate:m.optimeDate}))
Code language: JavaScript (javascript)

Example output (masked),

[
  {
    name: 'mongo1.internal:27017',
    state: 'SECONDARY',
    health: 1,
    optimeDate: ISODate("2026-01-09T16:50:45.000Z")
  },
  {
    name: 'mongo2.internal:27017',
    state: 'SECONDARY',
    health: 1,
    optimeDate: ISODate("2026-01-09T16:50:45.000Z")
  },
  {
    name: 'mongo3.internal:27017',
    state: 'PRIMARY',
    health: 1,
    optimeDate: ISODate("2026-01-09T16:50:45.000Z")
  }
]
Code language: JavaScript (javascript)

If optimeDate is the same (or very close) across nodes, it’s a good sign: your preferred node is caught up and eligible to win an election.

Step 2, Make mongo1 the Preferred PRIMARY (Priority)

Run this on the current PRIMARY (in the example, that’s mongo3).

cfg = rs.conf()

cfg.members.forEach(m => {
  if (m.host === "mongo1.internal:27017") {
    m.priority = 2
  } else {
    m.priority = 1
  }
})

rs.reconfig(cfg)
Code language: JavaScript (javascript)

Verify priorities applied,

rs.conf().members.map(m => ({ host: m.host, priority: m.priority, votes: m.votes }))
Code language: JavaScript (javascript)

Example output

[
  { host: 'mongo1.internal:27017', priority: 2, votes: 1 },
  { host: 'mongo2.internal:27017', priority: 1, votes: 1 },
  { host: 'mongo3.internal:27017', priority: 1, votes: 1 }
]
Code language: JavaScript (javascript)

Step 3, Trigger a Controlled Election (Step Down Current PRIMARY)

Still on the current PRIMARY,

rs.stepDown(60)
Code language: CSS (css)

This forces the current PRIMARY to step down for 60 seconds. With higher priority and equal freshness, mongo1 should be elected PRIMARY.

Re-check status:

rs.status().members.map(m => ({name:m.name, state:m.stateStr, health:m.health, optimeDate:m.optimeDate}))
Code language: JavaScript (javascript)

Expected output:

[
  {
    name: 'mongo1.internal:27017',
    state: 'PRIMARY',
    health: 1,
    optimeDate: ISODate("2026-01-09T16:52:10.000Z")
  },
  {
    name: 'mongo2.internal:27017',
    state: 'SECONDARY',
    health: 1,
    optimeDate: ISODate("2026-01-09T16:52:10.000Z")
  },
  {
    name: 'mongo3.internal:27017',
    state: 'SECONDARY',
    health: 1,
    optimeDate: ISODate("2026-01-09T16:52:10.000Z")
  }
]
Code language: JavaScript (javascript)

If mongo1 Still Doesn’t Become PRIMARY

These are the common blockers in real life.

mongo1 is configured as “never primary”
Check rs.conf().members and confirm mongo1 does not have

  • priority: 0
  • hidden: true
  • votes: 0

No majority
A 3-member replica set needs 2 reachable voting members to elect a PRIMARY. If you’re missing a node, elections may not behave as expected.

Replication lag
If mongo1 is behind, MongoDB may not elect it even with higher priority. Compare optimeDate values again and confirm mongo1 is close to the PRIMARY.

DNS/hostname mismatch
Your m.host match must be exact. Copy it directly from rs.conf().members to avoid typos.

Production Notes

A PRIMARY switch causes a brief write interruption. Well-configured drivers will reconnect automatically if your connection string includes the replica set name and multiple hosts. If you have strict write timeouts, do this during low traffic.

Cheat Sheet (Minimal Commands)

Check

rs.conf().members
rs.status().members.map(m => ({name:m.name, state:m.stateStr, health:m.health, optimeDate:m.optimeDate}))
Code language: JavaScript (javascript)

Prefer mongo1 and move PRIMARY

cfg = rs.conf()
cfg.members.forEach(m => { m.priority = (m.host === "mongo1.internal:27017") ? 2 : 1 })
rs.reconfig(cfg)
rs.stepDown(60)
Code language: JavaScript (javascript)

That’s all folks