Troubleshoot Worker

How to Resolve Unsupported GPU Issues on Windows and Linux?

If a user's supported GPU is listed as unsupported on the website, they should verify their NVIDIA driver configuration. Often, when a Docker container running nvidia-smi fails, the backend receives this information and marks the GPU as unsupported.

To check the configuration, running the following command should provide the correct output:

docker run --gpus all nvidia/cuda:11.0.3-base-ubuntu18.04 nvidia-smi

If I received a "device code authorization returned: Bad Request" error?

If this error is displayed following authorization:

Error: device code authorization returned: Bad Request
Error: Error authenticating: provided token has expired or invalid. Please re-authenticate using --no_cache=true flag.

This error might be caused by an issue with network connections. Visit https://auth0.io.solutions/activate. If the page does not open, it means Auth0 is not reachable for you.

How can I pause or reset a Worker if it has disconnection issues?

If your worker has been disconnected, here are a few steps to fix it:

1. Go to your running Worker Page and click on Pause for the Worker (located at the top-right corner).

2. Optionally, restart the device as needed.

3. After restarting the device, open your Worker page at IO.NET and copy the command from the Re-Run docker command block on the Worker Page, then run it Terminal.

   

4. If prompted, authorize the device using IO.ID. Remember, you have about 3 minutes to complete the authorization.

   

5. Confirm the removal of all previous containers in Docker.

   

6. Access your running Worker Page and note the Worker ID and Worker Name for future reference:

   • Open your browser and navigate to the Worker Page. Copy the Worker ID from the URL, which is the last value (e.g., https//worker.io.net/worker/devices/db4e9217-61d8-4869-bc49-05b9e1b51d85)

   • Copy the Worker Name from the title of the Worker Page.

     

   • Then enter this Worker name and Worker ID in turn into the Terminal when prompted by the Binary installer.

     

7. Wait for up to 10 minutes to see the progress. Your worker is now ready to use again.

How can suppliers change their accounts in IO Worker?

To change accounts, suppliers should include the --no_cache=true flag in the binary run command. This triggers a re-authentication process. Simply add --no_cache=true at the end of your main request when connecting the device with the new login.

./launch_binary_mac --disable_sleep_mode=true --no_cache=true

./io_net_launch_binary_windows.exe --disable_sleep_mode=true --no_cache=true

./io_net_launch_binary_linux --disable_sleep_mode=true --no_cache=true

📘

After one successful sign-in, the token will be saved in memory.

How can you run an existing worker UUID in the new authentication and authorization system?

As of May 2024, we have transitioned to a new authentication and authorization system. Follow these steps to run your existing worker UUID again.

1. Run the Command to Connect the Device

Open the Terminal on your system by navigating to the Start menu and using the search function. The process is similar regardless of your operating system.

Then run the command in the Terminal. For Windows, start by downloading and running the executable file.

• Download the binary for your operating system running command.
  For Windows you need to download the executable file. You can see how it's done here
  MacOSWindowsLinux

  curl -L https://github.com/ionet-official/io_launch_binaries/blob/main/io_net_launch_binary_mac -o io_net_launch_binary_mac
  

  https://github.com/ionet-official/io_launch_binaries/raw/main/io_net_launch_binary_windows.exe
  

  curl -L https://github.com/ionet-official/io_launch_binaries/blob/main/io_net_launch_binary_linux -o io_net_launch_binary_linux
  

• Use the following command to grant permissions to the new binary:
  MacOSLinux

  chmod +x io_net_launch_binary_mac
  

  chmod +x io_net_launch_binary_linux
  

• Launch the binary to connect your device to the platform.
  MacOSWindowsLinux

  ./io_net_launch_binary_mac --no_warnings=true 
  

  ./io_net_launch_binary_windows.exe --no_warnings=true 
  

  ./io_net_launch_binary_linux --no_warnings=true 
  

2: To authorize the device with IO.ID, follow one of the options below and verify your IO.ID account:

📘

Remember, you will have about 1-2 minutes to complete the authorization of the device. If it expires, run the code again.

You can do this in two ways:

1. Copy the Link from the Terminal: Paste it into your browser and confirm the action.
2. Copy the Code from the Terminal: Enter this code on the page https://auth0.io.solutions/activate to authorize the device.

After confirmation, the system will prompt you to log in.

3. Save the Token

The information below is found in the CLI. Keep this token accessible.

4. For all other machines use the saved token

This does not require manual reauthentication and can be used for all your worker nodes.

Why Do Containers Disappear from Docker?

If the container disappears from docker, then your worker was likely blocked. If this happens, you must recreate the worker from scratch.

Why Is My Worker Blocked?

Blocked status is indicative that our system detected GPU utilization that was not authorized by our internal checks. It's important that GPU availability is dedicated 100% to the task being volunteered for the health of the DePIN.

Blocked status can occur for a few different reasons, primarily:

• Excessive GPU Utilization: Activities such as playing games or using graphics-intensive applications. (You'll need to pause these activities before you start usage.)
• Mining Detection: Our team has implemented an update to detect mining devices and instances with high GPU usage, resulting in an automatic ban.
• Device and Hardware Switching: Our team has implemented a mechanism for blocking device if they were verified and later a different device with a different hardware was detected. If you need to use a new hardware, please create a new device

Common Issue: Container CPU Dropping to 0

A common issue that many users encounter is the CPU of the container dropping to 0.

This problem is often due to missing necessary software components. For instance, on Windows, you need to ensure CUDA and WSL2 are installed, while on macOS, you need to have Rosetta installed. Detailed instructions on how to install these software components can be found here.

If you still encounter this issue after installing all the necessary software components, try deleting the containers and images, then re-run the worker command and wait. You may need to repeat this process 3 or 4 times until they function normally. If the issue persists after these steps, it may indicate a system-level error.

Why Does My Worker Have an Unsupported Status?

This occurs when your GPU/CPU is not listed among the supported devices on the IO Network. You can check the list of supported devices here.

Why Does My Worker Disappear from My Dashboard?

This issue can stem from a few main causes:

1. Unsuccessful Connection: You didn't successfully connect the new worker. Here's an example of a successful worker connection:

   

2. Unsupported Hardware: If you successfully connected a new worker but the GPU/CPU you're using is not supported, you won't see your worker on the dashboard. You can check the list of supported devices here.

3. User Interface Issues: If your worker was running normally and suddenly disappears from the dashboard, it could be due to a problem with the user interface (UI). In such cases, try refreshing the website. If the worker still doesn't appear, please try again later or create a ticket for our support team to assist you.

Creating Multiple Workers from the Same Device

Users may inadvertently create a new worker each time the previous one fails instead of re-running the existing worker. To address this issue, you need to Terminate the old workers and learn how to restart Docker to continue running the existing worker. Follow the instructions here.

Bad CPU Type in Executable Error

If you come across an error message like "bad CPU type in executable," it's likely because you are trying to run software intended for an Intel processor on an Apple Silicon device. To resolve this issue, you'll need to install Rosetta 2, which enables support for Intel processors to operate within Docker on Apple Silicon devices.

To check if Rosetta is installed and active, enter the following command in the Terminal and press Enter:

/usr/sbin/sysctl sysctl.proc_translated

If the output is sysctl.proc_translated: 1, Rosetta is installed and active on your system.
If the output is either sysctl.proc_translated: 0 or there is no output, Rosetta is not installed or not active.

If Rosetta 2 is not installed, use this command to install it:

softwareupdate --install-rosetta --agree-to-license

Once Rosetta 2 installation is complete, rerun the execution command:

./io_net_launch_binary_mac

Regulate RAM Usage

Create a file called .wslconfig to restrict the resources used by WSL2 (Windows Subsystem for Linux). Follow the steps below to create it:

• Open File Explorer and navigate to your user's home directory (usually C:\Users<Username>).

• Create a new text file in your home directory and name it .wslconfig.

• Edit the .wslconfig File: Right-click on the newly created .wslconfig file and open it with a text editor such as Notepad.

• Add the following configuration parameters to limit memory (set values according to your preference):

  [wsl2]  
  memory=4GB # Limits the VM memory in WSL between 2 and 4 GB  
  processors=2 # Limits the number of processors to 2  
  swap=8GB # Sets the swap size to 8 GB
  

• Save the file and restart your computer. If the worker does not connect automatically, you may need to install it again.

Device Readiness Status

Your device status needs to be either Cluster Ready or Hired to be nominated for Block Rewards and be eligible for hiring. You can verify device status either on the device detail page or on the Workers tab in IO Explore. For more information, see the Get Started - IO Worker doc.

The four possible Readiness statuses are:

• Cluster Ready- Device meets PoW requirements and passed several Cluster Formation verifications.
• Hired- Device is currently hired by a cluster.
• Pending- The device has joined the network and is currently undergoing both the PoW and Cluster Readiness test. This process can take up to 12 hours of cumulative uptime after onboarding, but may complete sooner if your device passes our tests early. If your device remains in a Pending state after more than 12 hours of uptime, please contact us.
• Not Block Reward Ready- Device doesn't meet the criteria for block reward eligibility, either due to PoW failure or Cluster Formation verification failure or both.

Not Block Reward Ready offers one of three tooltips in the UI to provide troubleshooting tips.

• Please check your device's computational capacity- Your device's computational capacity is below the required threshold.
• Please check your device setup and computational capacity- Your device setup might not be configured correctly and its computational capacity is below the required threshold. Please refer to the worker setup guide.
• Please check your device setup- Your device setup might not be configured correctly. Please refer to the worker setup guide.

Computer Time Synchronization Issue

Make sure your computer's time is synchronized with the server. If it's not, the IO binary won't work properly. Click the Start Menu and open the Settings app.

Next, in the Settings app, go to Time & Language and select Date & Time. Then, under Additional settings, click the Sync now button to synchronize your computer's time with the server.

📘

Encountering problems? Feel free to open a support ticket by logging into your IO.Net account and submitting a ticket!