MySQL Max Allowed Packet

I recently conducted an upgrade, and for the life of me I couldn't figure out why the application wouldn't initialize.

I checked MySQL - it seemed to be running fine. I logged into the database, checked the Percona cluster status, it looked fine.

I checked RabbitMQ, and it also seemed to be running fine.

In checking the application logs, I saw an exception about a query and the packet size being too big, and I thought this was strange - mainly because of the huge size of the packet.

Sure enough, after calling support, I was informed that I needed to change the MySQL configuration in my.cnf and add a directive in the [mysqld] section.

max_allowed_packet=128M

In terms of what this value should 'really' be, I was told that this is a normal setting on most installations.

Who knew? It's unusual to be adding new parameters on the fly like this to a clustered database. 

But, sure enough, after restarting the database (well, the whole VM actually because I had done updates), it came up just fine.