Skip to main content

Troubleshooting

General Troubleshooting guide

Kubectl / helm commands fail​

If you receive an error as shown below when trying to invoke kubectl or helm commands, it is most likely caused by the KUBECONFIG variable not being set.

WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

The installation script attempts to create $HOME/.kube/config and sets this in the $HOME/.bashrc

You may need to log out and log back in, to create a new shell terminal session which will contain the correct KUBECONFIG variable, or run source ~/.bashrc to reload you shell config, or manually set it with export KUBECONFIG=~/.kube/config

NOTE: It is not advisable to run kubectl and helm commands as root (or sudo kubectl ... or sudo helm ....

The user-provided configs are sufficient for interacting with the k3s cluster and its resources.

Make sure that you $KUBECONFIG is set to a file that is accessible by your system user with the correct permissions.

You can view the functions responsible for setting kubeconfig here

$KUBECONFIG file has expired​

With the K3s installation, the kubeconfig can "expire" causing issues authenticating with the kubeapi.

eg response would be like this:

kubectl get secret chronicle-feed-eth-keys -n chronicle-feed -o jsonpath="{.data.ethFrom}" | base64 --decode
E1002 15:13:49.343237 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.345241 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.349349 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.351739 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.353627 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

This happens because we are interacting with the k3s cluster as a non-root user (best practice), and we are getting our kubeconfig file rancher at /etc/rancher/k3s/k3s.yaml. The client-certificate-data gets refreshed from time to time, in which case you will need to create a new kubeconfig file (normally once a year).

To fix this, run the following commands:

# as your validator user (eg demo-user)

rm $KUBECONFIG
sudo cp /etc/rancher/k3s/k3s.yaml $KUBECONFIG
sudo chown $USER $KUBECONFIG

You should be able to authenticate with the cluster and proceed with kubectl and helm commands.

Debug Bundle​

If you need further debugging, please retrieve the container logs, and some base info and provide it to the chronicle team for assistance:

tip

Make sure that you set FEED_NAME to match your feed in question, and run these commands from the correct user so that $HOME/$FEED_NAME/generated-values.yaml resolves to the correct file.

export FEED_NAME=<CHANGE_ME>
cd /tmp
kubectl logs deployment/ghost -n $FEED_NAME > ghost.log
kubectl logs deployment.apps/ghost-tor-daemon -n $FEED_NAME > tor-proxy.log
kubectl get pods,deployment,service,secrets,onion --all-namespaces > kbom.txt
helm list --all-namespaces > all-helm-releases.log
tar czf chronicle-debug.tar.gz ghost.log tor-proxy.log all-helm-releases.log kbom.txt $HOME/$FEED_NAME/generated-values.yaml

The above commands will get logs for each of the pods running, the state of the services, as well as the generated-values.yaml for your feed, and bundle it into a tarball chronicle-debug.tar.gz (Please change the values / paths accordingly)