Troubleshooting
General Troubleshooting guide
Kubectl / helm commands fail​
If you receive an error as shown below when trying to invoke kubectl
or helm
commands, it is most likely caused by the KUBECONFIG
variable not being set.
WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied
The installation script attempts to create $HOME/.kube/config
and sets this in the $HOME/.bashrc
You may need to log out and log back in, to create a new shell terminal session which will contain the correct KUBECONFIG variable, or run source ~/.bashrc
to reload you shell config, or manually set it with export KUBECONFIG=~/.kube/config
NOTE: It is not advisable to run kubectl and helm commands as root (or sudo kubectl ...
or sudo helm ...
.
The user-provided configs are sufficient for interacting with the k3s cluster and its resources.
Make sure that you $KUBECONFIG
is set to a file that is accessible by your system user with the correct permissions.
You can view the functions responsible for setting kubeconfig here
$KUBECONFIG file has expired​
With the K3s installation, the kubeconfig
can "expire" causing issues authenticating with the kubeapi.
eg response would be like this:
kubectl get secret chronicle-feed-eth-keys -n chronicle-feed -o jsonpath="{.data.ethFrom}" | base64 --decode
E1002 15:13:49.343237 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.345241 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.349349 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.351739 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E1002 15:13:49.353627 2375376 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)
This happens because we are interacting with the k3s cluster as a non-root user (best practice), and we are getting our
kubeconfig
file rancher at/etc/rancher/k3s/k3s.yaml
. Theclient-certificate-data
gets refreshed from time to time, in which case you will need to create a new kubeconfig file (normally once a year).
To fix this, run the following commands:
# as your validator user (eg demo-user)
rm $KUBECONFIG
sudo cp /etc/rancher/k3s/k3s.yaml $KUBECONFIG
sudo chown $USER $KUBECONFIG
You should be able to authenticate with the cluster and proceed with kubectl
and helm
commands.
Debug Bundle​
If you need further debugging, please retrieve the container logs, and some base info and provide it to the chronicle team for assistance:
Make sure that you set FEED_NAME
to match your feed in question, and run these commands from the correct user so that $HOME/$FEED_NAME/generated-values.yaml
resolves to the correct file.
export FEED_NAME=<CHANGE_ME>
cd /tmp
kubectl logs deployment/ghost -n $FEED_NAME > ghost.log
kubectl logs deployment.apps/ghost-tor-daemon -n $FEED_NAME > tor-proxy.log
kubectl get pods,deployment,service,secrets,onion --all-namespaces > kbom.txt
helm list --all-namespaces > all-helm-releases.log
tar czf chronicle-debug.tar.gz ghost.log tor-proxy.log all-helm-releases.log kbom.txt $HOME/$FEED_NAME/generated-values.yaml
The above commands will get logs for each of the pods running, the state of the services, as well as the generated-values.yaml for your feed, and bundle it into a tarball chronicle-debug.tar.gz
(Please change the values / paths accordingly)