UI Slowness, Nodes and Instances temporarily showing Offline or missing, or other UI related issues
Scenarios
- Loading the UI seems to be very slow
- Node status all seem to be delayed or alternating between different states randomly
- Instances don’t show up immediately or at all
Confirm an
average_queue_timeof > 5 when curling the controller’sapi/v1/statsendpoint
Common Causes
- Lots of VMS, Nodes, and not enough load balancing/instances of the controller, etcd, etc.
- ETCD is very sensitive to disk latency; not using SSDs for the etcd storage
- Session stickiness is not enabled for your load balancer or too low
Solutions:
- Check your load balancer session stickyness settings and increase them to several minutes
- Increase the space quota from the default 2GB: [https://etcd.io/docs/current/op-guide/maintenance/#space-quota]
- Rejoin your nodes with a higher
--heartbeatvalue (> 20s) - Upgrade the host’s disk to an SSD or faster disk
- Set
ANKA_NUM_WORKERSto more than the default 2 in the Controller configuration.
State changes can also be caused by network issues between the Node and the controller. Check the node’s
/var/log/veertu/anka_agent.ERRORlog to confirm you’re not seeing timeouts or connection errors.