Diffusion models are a powerful class of AI models for generating images from text prompts. However, its true creative potential is unlocked through fine-tuning. Fine-tuning allows you to adapt the base model to generate images in a highly specific style, or to consistently depict a particular character or object. By training the model on a curated dataset—even a small one—you can create a specialized model that produces results aligned with your unique vision. This process makes AI art creation more accessible, enabling artists, developers, and enthusiasts to craft bespoke visual content without the need to train a massive model from the ground up.

However, the conventional fine-tuning process presents significant hurdles that can make it inaccessible for many. It often demands access to expensive, high-performance GPUs, which are frequently in short supply and costly to rent from centralized cloud providers. Beyond the hardware costs, setting up and maintaining the correct software environment—with specific drivers and library versions—can be a complex and time-consuming task. Furthermore, ensuring the reproducibility of the training process and the integrity of the final model can be challenging in a traditional setup. These challenges highlight the need for a more accessible and robust infrastructure for AI development.

Solving the Challenge with Crynux and Irys

Crynux is a decentralized AI network that empowers users to fine-tune AI models like Diffusion models by tapping into a global network of GPU providers. It operates on a permissionless and trustless architecture, where tasks are distributed among multiple nodes that execute the computation.

For developers, this means there is no need to set up and maintain complex software environments. You simply define your fine-tuning task and submit it to the network, where nodes handle all the underlying dependencies and execution. This opens up access to AI model training, allowing anyone to train their own models without needing powerful and expensive hardware, all while ensuring transparency and reliability through its decentralized design.

Irys provides the data layer for the decentralized web. For AI model fine-tuning, the integrity and perpetual availability of the dataset are paramount. Irys addresses this with its specialized datachain, a blockchain optimized for affordable and immutable data storage. By hosting the training dataset on Irys, developers ensure that their data is permanently stored, tamper-proof, and globally accessible.

When a fine-tuning task is initiated on Crynux, the Crynux nodes can seamlessly and reliably retrieve the dataset from Irys, guaranteeing a consistent and reproducible training process. This combination of decentralized computation from Crynux and decentralized data from Irys creates a robust, end-to-end solution for trustless AI model development.

Example: Creating a Naruto-Style Image Generation Model

In this blog post, we'll walk through a practical example of fine-tuning a Diffusion model to generate images in a specific artistic style. We will use a dataset of images from the popular anime series Naruto. By fine-tuning the model on this dataset, we will create a LoRA (Low-Rank Adaptation) model capable of producing new images that capture the unique visual essence of Naruto. This example will showcase how you can leverage Crynux and Irys to create your own specialized image generation models.

Step 1: Prepare the Dataset

Our first step is to prepare the training data. For this Naruto example, our dataset comes from the open Naruto dataset provided by Lambda Labs. This dataset is stored in Apache Arrow format (.arrow files), which contains a single table with two columns: a column of PIL Image objects in JPEG format and another column of string captions corresponding to the images.

First of all we need to upload the dataset to Irys. This action gives us a unique Irys data link, which points to our immutable dataset. We can achieve it using the Irys SDK:

For more details on managing your data with Irys, please refer to their official documentation:

Irys | Irys SDK

Irys SDK

Step 2: Run the Fine-Tuning Task

With our dataset ready on Irys, we can now tell the Crynux Network to start fine-tuning. The process is surprisingly simple. As you can see from the Python code below, you don't need to handle complex blockchain transactions or wallet signatures. It's all done through straightforward API calls, much like interacting with any modern web service.

import time

import httpx

client = httpx.Client(
    base_url="https://bridge.crynux.io",
    timeout=180,
)

# For public demonstration only, strict rate limit applied.
api_key = "q3hXHA_8O0LuGJ1_tou4_KamMlQqAo-aYwyAIDttdmI="

data = {
    "model_name": "crynux-ai/stable-diffusion-v1-5",
    "model_variant": "fp16",
    "dataset_url": "https://gateway.irys.xyz/DLsfGoiehnzpGNdsQ3XPEaVL9D9HNELGjL3yxsj67SUK",
    "validation_num_images": 4,
    "learning_rate": 0.0001,
    "batch_size": 1,
    "num_train_steps": 100,
    "max_train_steps": 200,
    "lr_scheduler": "cosine",
    "lr_warmup_steps": 0,
    "rank": 4,
    "init_lora_weights": "gaussian",
    "target_modules": ["to_k", "to_q", "to_v", "to_out.0"],
    "center_crop": True,
    "random_flip": True,
    "mixed_precision": "fp16",
    "seed": 42,
    "timeout": 1800,
}

header = {
    "Authorization": f"Bearer {api_key}",
}
resp = client.post(
    "/v1/images/models",
    json=data,
    headers=header,
    timeout=180,
)
resp.raise_for_status()
res = resp.json()
data = res["data"]
task_id = data["id"]
print(f"Task ID: {task_id}")

success = False
while True:
    resp = client.get(
        f"/v1/images/models/{task_id}/status",
    )
    resp.raise_for_status()
    res = resp.json()
    data = res["data"]
    status = data["status"]
    if status == "success":
        print(f"Task success")
        success = True
        break
    elif status == "failed":
        print(f"Task failed")
        success = False
        break
    elif status == "running":
        print(f"Task is still running")
    time.sleep(60)

if success:
    with client.stream(
        "GET",
        f"/v1/images/models/{task_id}/result",
        headers=header,
        timeout=180,
    ) as resp:
        resp.raise_for_status()
        with open("result.zip", "wb") as f:
            for data in resp.iter_bytes():
                f.write(data)

    print("Task completed")

In the code, we simply pass the Irys link to our dataset in the dataset_url parameter. Submitting the task sends it to the Crynux Network, where it is distributed to a group of decentralized nodes. Each node then independently downloads the dataset from Irys using the provided link and begins the fine-tuning computation.

Executing the script returns a unique Task ID for the job. The script then periodically checks the status of this task until it's successfully completed.

Once the fine-tuning is complete, the script automatically downloads the resulting LoRA model, a single zip file containing the learned Naruto style.

Step 3: Generate Images using the LoRA Model

Now for the fun part! With the trained LoRA model file, you can start generating your own Naruto-style images. You can load this file into a local tool like the Stable Diffusion WebUI, or even use the Crynux Network again to generate images through its API.

This completes our example of creating a fine-tuned LoRA model using Crynux and Irys.

Learn More

To dive deeper into the technical details of fine-tuning on Crynux, check out the official documentation:

How to Fine-tune a Stable Diffusion Model using Crynux Network

Beyond fine-tuning, the Crynux Network also supports running a wide range of popular Large Language Models (LLMs) through an OpenAI-compliant API, making it compatible with most mainstream libraries and frameworks.

How to Run LLM using Crynux Network

You can also use the Crynux Network API to generate images:

How to Generate Images using Crynux Network

Have a GPU sitting idle? You can also contribute to the network and earn token rewards by running a Crynux Node in a single click.

Start a Crynux Node

Join the Crynux community on Discord to get instant tech support and connect with developers and other users:

Crynux Discord

Follow Crynux on X for the latest news and updates:

https://x.com/crynuxio

Hey NEAR Community!

Get ready to level up your contribution to the decentralized web! We're thrilled to announce the release of Crynux Node v2.5.0, a massive upgrade to support NEARverse. If you're looking for a way to actively participate in the NEAR ecosystem and earn rewards, now is the perfect time to spin up a Crynux node!

What's Crynux and Why You Should Care:

Crynux is a distributed orchestration layer on edge devices. It means you can leverage the power of your own computer to provide AI services and get rewarded for it! But what makes Crynux truly special?

• Permissionless: Forget applications and waitlists! Anyone with a device can join the Crynux network right now. It's truly open and accessible to everyone in our NEAR family.
• Serverless: Unlike traditional AI services relying on massive cloud infrastructure, Crynux runs entirely on the collective power of community-owned devices. Your contribution directly fuels the network!
• Trustless: Crynux uses a trustless protocol to manage AI tasks without the need for traditional validators. Instead, it incentivizes node operators like you with proof of meaningful work.
• Economical efficiency: By tapping into the underutilized power of edge devices, Crynux can offer AI task execution at a fraction of the cost – sometimes as low as 5% compared to traditional cloud providers!
• Computational efficiency: Crynux can intelligently distribute tasks across hundreds of devices with elastic deployment.
• Robustness: Worried about accuracy? Crynux employs a robust verification system called VssML (Verifiable Secret Sampling) to ensure the integrity of results, harnessing the community's power while preventing fraud.
• Scalability: The beauty of Crynux lies in its ability to scale to potentially billions of nodes without incurring massive overhead. As our NEAR community grows, so does the potential of Crynux to offer affordable AI services.

Ready to Dive In? Setting Up Your Crynux Node is a Breeze!

Getting your Crynux Node v2.5.0 up and running is surprisingly simple. We've created a comprehensive guide to walk you through every step:

https://docs.crynux.ai/node-hosting/start-a-node

Whether you're running Windows, Linux, or Mac, you can start contributing with just a few clicks:

1. Download the Node App: Get the Crynux Node application for your operating system.
2. Create Your Wallet: Open the app and create a new wallet. Crucially, make sure to securely store your private and public keys!
3. Join the Crynux Discord: Head over to https://discord.gg/JPpN4RNdqA and register your wallet for test tokens.
4. Get Your Test Tokens: Once you've registered your wallet, you'll receive test tokens to get started.
5. Wait for Model Download: The node will automatically download the necessary AI models.

Join the Crynux Network and Empower NEAR!

By running a Crynux node, you're not just earning rewards – you're actively contributing to the growth and resilience of the NEAR ecosystem. Your node plays a vital role in powering decentralized AI computations and strengthening the network for everyone.

We wholeheartedly encourage every member of the NEAR community to explore this incredible opportunity and become a part of the Crynux revolution. Let's work together to build a more powerful, efficient, and truly decentralized future!

Have questions or need support?

Don't hesitate to reach out to the Crynux team through their community channels. We are always happy to help!

Join the Crynux #DeAI Discord Server!

Built by ex-Google engineers, Crynux is pioneering decentralized AI infrastructure from the ground up. | 1147 members

Discord

Follow Crynux on X for more updates and announcements:

https://x.com/crynuxai

Videos, music, images and text, generative multi-modality models are revolutionizing the way we create content. These advanced tools are not only incredibly useful but are also pioneering new avenues in creative content production, significantly reducing the workload involved.

For instance, consider a music generation model capable of composing chiptune pieces for video game backgrounds, the soundtracks could be dynamically created to align with the unfolding game narrative, offering players an immersive and tailored gaming experience.

This approach not only streamlines the soundtrack creation process but also democratizes game music production, significantly reducing labor costs and lowering the barriers to entry in the industry.

Specifically tailored models for the target styles, areas, industries, or applications can dramatically enhance the quality of the generated content. Unlike general-purpose large models, tailored models understand the nuances and intricacies of their designated domain, leading to outputs that are more refined, accurate, and contextually appropriate. This specialization not only elevates the content quality but also reduces the computational resources necessary for running these models, translating into significant cost savings.

Federated Fine-tuning for Maximizing Extensive Data Usage

The fidelity and efficiency of these models come from the fine-tuning/distilling from general purpose models, with large-scale data from extensive user collections.

The model quality is not static. They could be continuously improved through iterative fine-tuning process. This process ensures that the models evolve, becoming more sophisticated and capable of producing even more accurate and high-quality content over time.

Federated Learning (FL) plays an important role here. FL enables models to learn from decentralized data sources without actually moving the data, enhancing privacy and allowing the model to continuously improve from real-world use across different devices and users.

FL will enable a broader participation in the creative model fine-tuning, pooling a richer diversity of data, which in turn elevates the quality and effectiveness of the models.

Bittensor Incentivized Data Contribution

Thanks to the incentivization mechanism provided by Bittensor, the subnet offers a compelling proposition where community members are motivated to contribute to data collection and the fine-tuning of models.

By contributing the data, individuals can earn rewards, creating a sustainable ecosystem where every contribution is valued and rewarded. This method not only preserves the privacy and security of user data but also ensures a continuous stream of income for those actively participating in the system. As more members join and contribute, the diversity and volume of data for fine-tuning improve, leading to more accurate and effective models.

In essence, Bittensor's incentivization mechanism fosters a vibrant community-driven environment, where the collective efforts of participants drive the advancement of AI technology in a secure and profitable manner.

Lower the Barrier of Participation using Crynux Network

The need for costly, high-end AI computing cards for validators and miners to carry out model fine-tuning and validation tasks is no longer an issue, thanks to Crynux Network's decentralized AI service cloud.

Miners can now concentrate solely on supplying higher quality data, as miner nodes can be operated on standard laptops or even smartphones. The federated computing tasks will be executed remotely via the Crynux Network.

Similarly, validators can obtain model evaluation results effortlessly through a straightforward API call on regular servers, eliminating the need for GPU cards.

Enforced Transparent Revenue Distribution, End-2-End

Models developed on the subnet will be hosted on the Crynux Network, offering services across all applications. Validators and miners contributed to these models on the subnet will share ongoing rewards from the payments of the models' usage.

Crynux Network transforms the creative models into AI assets. Thanks to its decentralized model service cloud, all services related to the models—from inference to fine-tuning—are transparently processed on the blockchain. This ensures direct payments from users of the models to the holders of the model assets, eliminating intermediaries and manual intervention, technically securing the profit of the asset holders. Embrace a new era of trust and efficiency.

Additional AI-Fi applications may also spring up around the model and data assets, expanding their reach by linking these assets to a diverse array of AI-Fi applications and thus enlarging the current DeFi ecosystem.

The Genki-Dama Subnet

Genki-Dama, inspired by the iconic Dragon Ball technique, is a Bittensor subnet that utilizes decentralized data and harnesses decentralized computing resources. Built upon the incentive mechanism of Bittensor, and the computing network of Crynux, Genki-Dama empowers federated learning in a decentralized manner, shattering the limitation of centralized approaches.

The subnet aims to incentivize miners to contribute high quality data and train creative generative models with federated learning.

It includes two parts:

• Genki: federated learning SDK to utilize Bittensor incentive mechanism and Crynux decentralized computing resources
• Dama: open-sourced model checkpoints trained by Genki, we will focus on generative models for creative contents.

The first Dama is called Ruby, which is an Electronic Chiptune style music model that could be used to generate music for games. A demo video of fine-tuning such a model, and using it to generate music is given on X:

🎶 Unleash your creativity with Crynux! Generate your own music on distributed edge – the magic starts at 00:20. 🎧✨ #DeAI #MusicMagic pic.twitter.com/wysw2s47V1

— Crynux #DeAI (@crynuxai) September 12, 2024

Find more information about the Genki-Dama Subnet in the GitHub repo:

GitHub - crynux-ecosystem/genki-dama

Contribute to crynux-ecosystem/genki-dama development by creating an account on GitHub.

GitHubcrynux-ecosystem

And follow the tutorials inside the README file to start a validator/miner node.

🎉 Exciting News: HappyAIGen Is Now Live! 🚀 Play to Earn, and Have Fun!

HappyAIGen, our community-driven project, powered by AI tools from the Crynux Network, is officially live today! Dive into a world of fun activities while earning credits along the way.

Join our Discord server to kickstart your journey with Welcome Credits 👉https://discord.gg/5U2jV9rDSC

Remember, the earlier you join, the more credits you can earn!

🎮 Play AI Games

Engage in thrilling AI-powered games and unleash your creativity by generating images with our AI bot. Stay tuned as we continuously introduce more games to keep the excitement flowing.

Find all the exciting details inside the Crynux Discord server.

🌱 Help Our Community Thrive

Your support is crucial in expanding the Crynux ecosystem. Earn credits by inviting friends to our Discord server and spreading the word about Crynux through tweets.

Follow the instructions within our Discord server to start earning credits by inviting your friends.

⚛️ Start a Crynux Node

Become a pillar of the Crynux Network by starting a Crynux Node. As a node provider, you'll be rewarded for contributing AI computing power to the HappyAIGen project.

Starting a Crynux Node is hassle-free! Simply download the Crynux Node software and launch it with just one click. If you own an NVidia GPU or a Macbook/Mac Studio M1/M2/M3, you can start collecting credits automatically.

Get started with these easy-to-use apps:

Start a node on Windows

Start a node on Mac

Start a node using Docker

The REAL DeAI is the one you can join.

Join us in making decentralized AI a reality!

We are excited to announce the release of the Helium Network, which is the second testnet of the Crynux Network.

The Crynux Network evolves through the sequential release of various networks, each building upon its predecessor by introducing additional features and improvements. Initial releases are designated as testnets, serving as preliminary versions for testing purposes. Subsequent releases are mainnets, which incorporate upgrades to the previous networks through forking.

The networks are named after the elements in the Periodic Table, in which Helium is the second element. In the last testnet, the Hydrogen Network, a decentralized Stable Diffusion task execution engine is implemented, the applications could use the network to generate images, and everyone possessing an Nvidia graphic card could start a node to join the network, exchanging the computation power for tokens.

Building upon the foundation of the Hydrogen Network, the Helium Network introduces an array of enhancements. These include a variety of new application use cases and support for more node device types which are poised to significantly increase the number of nodes connecting to the network.

Running the GPT Text Generation Tasks

The long expected feature to run the GPT text generation tasks on the Crynux Network is finally out. AI Chatbot applications could have already been built on top of the Crynux Network. Choose from most of the LLM models on the Huggingface, such as LLaMa 2 and Gemma, specify the model ID and the prompt in a GPT task, and the task could be executed in the Crynux Network to get output texts.

A chatbot UI has been deployed to demonstrate the ability. To try the chatbot yourself, go to https://chat.crynux.ai:

After the user selects an open source LLM model, and submits the question in the WebUI, the question will be sent to the Crynux Network as a GPT task. The result will be returned to the user when the execution is finished.

To the application developers, it is easy to connect the application to the Crynux Network, enhancing the application with AI ability instantly. Just follow the getting started guide for the developers.

Mac as the Computation Node

Now the Crynux Node could be started on the Mac with Apple Silicon chips (m1, m2 and m3 series). All the owners of a supported Mac could now join the network to earn tokens by simply downloading the app and starting it with one click.

The Unified Memory Architecture of the Mac, which allows the shared use of system memory to both the CPU and GPU with high bandwidth, offers a significant advantage when working with large AI models. Although the execution might be slower, Mac's ability to utilize its extensive system memory enables it to successfully complete certain AI tasks involving large models—tasks that might fail on Nvidia cards due to their limited VRAM. This capability offers a cost-effective alternative in scenarios where execution speed is not the primary concern.

To start a node on your Mac, follow the node starting guide.

Flexible Task Pricing

The task price is now determined completely by the market. If there are more nodes to execute the same number of tasks, the average task price will drop. If there are more tasks to be executed by the same set of nodes, the task price will rise.

The task price is not set by the Crynux Network though, neither fixed nor dynamically. Instead, the price is set by the user, and the Crynux Network will choose the tasks with higher prices to execute first.

If the user needs his task to be completed faster, he could set a higher price on his task. However, if the price is a critical concern to the user, he could also finish the task at a lower price, and wait longer for the result.

The flexible task pricing mechanism allows for an effective and efficient allocation of tasks across the network. By allowing the market to dictate task prices, Crynux ensures that both nodes and users benefit from a dynamically balanced system. Users are incentivized to offer competitive prices for quicker task execution, while more nodes are attracted to join the network when the task prices go higher, providing more computation power to the Crynux Network.

Even more, the order of task execution is not simply determined by the price set by the user, but rather a task value estimated by taking the task execution duration into account. This approach effectively identifies tasks that provide optimal value—those that contribute a significant amount without demanding an excessive portion of resources, maximizing the incomes of all the nodes in the network.

The details about the task pricing strategy is described in the documentation,

Improvements on the Network Efficiency and Availability

Crynux Network is a loosely coupled decentralized network, where a stable connection to a node can never be expected. A node could be shutdown while still claimed online on the blockchain, or stop responding at any time during the task execution. Which, if not handled properly, will cause a lot of task failures that renders the network unusable to the applications.

A lot of efforts have been made to improve the network efficiency and availability, such as giving higher probability to select nodes that providing better services, kicking out nodes that constantly fails, and adjust the incentivization based on the node availability.

The details can be found in the documentation.

A complete list of all the features and improvements of the Helium Network can be found in the release node:

Helium Network | Crynux Network

[Jan 30, 2024] Decentralized GPT Task Execution Engine

Logo

The Helium Network is a big step towards a minimum yet complete network to be massively adopted. We are already close to a mainnet release that brings the exciting experience to the real-life applications and users.

Stay tuned!

We are thrilled to announce the release of the Hydrogen Network. A fully featured decentralized network that supports running the Stable Diffusion image generation tasks.

From the application's perspective, it is an inference API service on the cloud that could be used just like AWS. The application submits the task to the API, and gets the images in return.

The computation power comes from a decentralized network of the home computers that are coordinated by a consensus protocol running on the Blockchain. The individuals who have the spared computation power could connect their devices to the network, exchanging the computation power for tokens by starting a node, to run the inference tasks for the applications.

The Hydrogen Network is the very first testnet of Crynux. Although called a testnet, the featured consensus protocol is robust enough to allow everyone to join at this moment.

In case you can't wait, here is the getting started guide for the application developers: https://docs.crynux.ai/application-development/application-workflow. And this is the getting started guide for the node providers: https://docs.crynux.ai/node-hosting/join-the-network

The Image Generator

Let's start by introducing the Image Generator, the first application running on the Hydrogen Network.

The Image Generator provides a web interface (just like stable-diffusion-webui) for the users to generate images in the browser. Thanks to the Hydrogen Network, the application could be used on the devices that do not have a capable GPU integrated. If the browser exists, the Image Generator could be used.

The base models, such as SD1.5 and SDXL, can be selected in the user interface. The LoRA models can also be selected and combined with the base model. The OpenPose Controlnet is also supported. The users could even input a model download link from Civitai to be used as the LoRA model.

So far, the application looks exactly the same as a traditional web application, nothing special. Until the "Generate Image" button is clicked, and the modal displaying the task execution status is shown, we could find a little bit of difference.

From the modal, we could see that the task is sent to the Blockchain first, and then executed on the computation nodes, and finally the result images are verified and retrieved.

The Image Generator can be accessed directly online, give it a try yourself:

https://ig.crynux.ai

The Decentralized Computing Network Underneath

The Blockchain has maintained a list of all the available nodes in the network. When a task is sent to the Blockchain, the Blockchain randomly selects 3 nodes in the list, and notify them to execute the task.

The nodes will run the task locally. When the images are finally computed,  each of the nodes will disclose a similarity score on the Blockchain. The Blockchain compares the score of 3 nodes to find out if there are nodes who are escaping from the computation by submitting random scores, to save the local resources. If the result is validated to be correct, the images are returned to the application. And the nodes will get paid. The video below demonstrates this process:

Four windows are shown in the video:

• Top-left: the Image Generator application
• Top-right: the WebUI of the Crynux Node
• Bottom-left: the Blockchain explorer
• Bottom-right: the logs from the Docker container of the Crynux Node

The image generation started from the user's operation in the Image Generator(top-left window). The user selected a base model and a LoRA model, and then clicked the "Generate Image" button. A modal popup was displayed to show the running status to the user.

The task was sent to the Blockchain. When it was confirmed on the Blockchain, the event logs were shown in the block explorer(bottom-left window). When the task started executing, the task execution status changed from "idle" to "executing" on the WebUI of the Crynux Node (top-right window).

More logs of the Node was printed(bottom-right window) and more event logs appeared in the block explorer when the Crynux Nodes finished the local execution and started to execute the consensus protocol on the Blockchain.

Then the task was successfully finished, the images were shown in the Image Generator, and the status of the Crynux  Node became idle again, with 10 more tokens in the wallet.

As a decentralized task execution engine, the Hydrogen Network lays a solid foundation for the Crynux Network. Upon which more task types will be supported, such as the fine-tuning of the Stable Diffusion models, and GPT models, in a short future. And the decentralized model serving layer will be built next.

The most important thing in the decentralized network is the consensus protocol, which ensures that all the nodes are behaving honestly. Since beside the rules defined in the consensus protocol, there is nothing else to control what a node can do. They join and quit the network freely. Nobody knows who they are and where they are. They will try to exploit all the vulnerabilities in the consensus protocol to get more tokens at lower cost.

Find more about the consensus protocol in the document:

https://docs.crynux.ai/system-design/consensus-protocol

Start a Node

If you have purchased an NVIDIA RTX graphic card, whether for gaming or designing, why not letting it make some extra profit for you, when you're not using it.

The node can be easily started on both Windows and Linux machines. The node is limited to a Docker container, which can not affect the other parts of the system. An integrated WebUI is provided to visualize the node status and the system resource usage, which also provides an easy way to control the node:

The node can also be started in the headless mode, which is convenient for the server environment. After proper configuration, the node will automatically join the network when the container is started, and quit the network when the container is terminated.

Open Source and Well Documented

Crynux is committed to build an open and democratic ecosystem. We prioritize the community by being completely open to it. The projects are open sourced from day one. A lot of efforts have been put on the documentation of every aspect of the system.

All the source codes of the Hydrogen Network can be found on the GitHub:

https://github.com/crynux-ai

All the technical documents of the Hydrogen Network can be found on the documentation site:

https://docs.crynux.ai

If you want to learn more about the design of the Hydrogen Network, start from the network architecture:

https://docs.crynux.ai/system-design/network-architecture

And don't hesitate to contribute, you are always welcome to join the revolution.

The verification mechanism is critical to every decentralized network. Vulnerabilities in the mechanism will lead to the collapse of the entire network.

In the decentralized GPU sharing network, the individual GPU holders contribute their spared computing power to the network in exchange for the profit, and the users pay to use the computing power to run their AI tasks, such as the Stable Diffusion image generation.  If the GPU holders could send random computation results back without being discovered, the users will keep getting bad results for their AI tasks, and they will soon stop using the system, and the system will be abandoned by all eventually.

Note that we can not rely on the users to discover the cheating, since the user could also cheat by reporting error computation for every task to save his money. The verification could only be performed by the system, or more precisely, the consensus mechanism in the decentralized network.

The verification is much harder in the decentralized systems, comparing to the verification in the centralized systems. The verification rules must be open to the public since it must be executed by all the consensus participants in the network. Which gives the attackers the opportunity to fully examine all the details to find vulnerabilities. And the rules is fixed before the attack happens. There is no chance to manually examine the attacking behaviors and modify the rules accordingly before the real damage occurs.

Works have been done on various of schemas and the related technologies by quite a lot of researchers and companies. At Crynux, we have been working on the Zero Knowledge Proofs (ZKP) for quite a long time, to verify the actual execution of the AI tasks. The bottleneck so far, is still the speed of the ZKP generation for large neural networks, or a practical contribution measurement schema that survives the randomness and the attacks of the long-running gradient descending process.

But if we put some limits on the supported AI tasks, we could already implement a practical schema that works on large scale neural networks in a large scale decentralized network.

More specifically, the limit we put on the task is that the input must be open and the task must be deterministic. If the same task is executed by multiple nodes, they should all get the same result. For example, the training using private datasets from other nodes is not supported since the private data can not be revealed to others. But most of the tasks are supported other than those related to the private data. Stable Diffusion training task is supported, as well as the image generation task. LLM fine-tuning and inferencing tasks are also supported.

The idea is actually quite simple: for a given computation task, the network randomly selects 3 nodes to run the task simultaneously, and compares the results returned from the nodes. If one of the nodes has a different result than the other two nodes, the node is cheating. And the node should have certain amounts of tokens staked on the network before receiving tasks. If the node is found cheating, the staked tokens are slashed immediately.

The design consumes 3 times the computation power than what is required only to get the computation result. But since it is the spared computation resource we are talking about, it will still be cheaper than the cloud hosted GPUs. And we can still further cut it down to 2 times in most cases by a small optimization in the workflow.

For the attacker, it becomes a calculation of the probabilities of winning or loosing. The variable the attacker could control is the number of the (fake) nodes he possesses in the network. The attacker could start as many nodes as he wants, as long as he has enough tokens to stake. Possessing more nodes gives the attacker higher probability to win, since it will be more likely to have more than two nodes of himself been selected in a single task. And in this case, he can cheat the system by giving two identical fake results.

The probability of a successful attack (an attacker getting more than 2 nodes of himself selected in a task) could be calculated as:

\[  p = \frac{ C_d^2 * C_h^1 + C_d^3}{C_{d+h}^3} \]

Where \( h \) is the number of the honest nodes, and \( d \) is the number of the dishonest nodes the attacker possesses.

And the expectation of the income from cheating is given by:

\[ E = p * k - (1-p) * s \]

Where \( k \) is the price of the task, and \( s \) is the number of staked tokens.

By increasing the number of staked tokens \( s \), we could decrease the expectation \( E \) down to zero or even below. If \( E \) is below zero, there is no benefit to attack the system by starting more fake nodes. The attacking is highly likely to cause the attacker to loose money rather than earn.

Let's see some examples of the calculated numbers.

For easier understanding, assume the price for executing a task is $0.1, rather than a number in tokens. And let's say 10% of the nodes in the network are honest. Set the target \( E \) to be zero, the required amount of money to stake is then:

\[ s = \frac{p * k}{1-p} \]

Note that we might double the amount of the tokens required to stake in practice, to further limit the gambling attempts of the attackers.

Here's a table showing the required money to stake in different network sizes:

We could tell from the calculation that the probability of successful cheating is only related to the proportion of the honest nodes in the network. And at such a low proportion, the amount of money required to stake is surprisingly low.

Let's decrease the proportion of the honest nodes further more and set it to 1%, here's what happens:

The required amount of money to stake is still not too much. And note that we are assuming all the dishonest nodes belong to a single attacker. In the real world, if there're multiple attackers, and if they're not conspired to summit the same fake result, the attackers could be treated as the honest ones to each other. Which will further reduce the risk of being successfully attacked.

The staking based verification mechanism lays a solid foundation for the decentralized GPU sharing network, but there are still possibilities to attack the network from the engineering aspect.

For example, the attacker could monitor the Blockchain for computation result submission, and submit the same result as other nodes in the task. Another way of attack is to submit fake results only when the attacker finds that two of his nodes are selected in a same task. In other cases, he just aborts the task to avoid being slashed.

The engineering attacks could be solved with careful system design and implementation. But the attacks are still somewhat inevitable for a growing system. Several rounds of the testnet runs under controlled environment will help resolving the potential problems.

Stay tuned for the upcoming!

Stable Diffusion and AIGC has become a groundbreaking tool that has transformed the landscape of various industries, offering unprecedented benefits and opportunities.

Being an open source solution, the original Stable Diffusion models are freely accessible to the public. The open source toolkits around the models, such as the web user interface, are developing rapidly as well. More and more pretrained and LoRA models are published online for others to download and use directly. The ecosystem around Stable Diffusion is evolving fast.

Despite the growing on the size of the Stable Diffusion community, there are still obstacles that prevent the adoption of such AIGC tools to the vast majority.

Expensive GPU is required just to begin

The requirement to run the Stable Diffusion tools for an Nvidia graphic card, although not as high as for LLMs, is still mandatory. At least a consumer level RTX card that supports CUDA must be provided. Given that most of the laptops are equipped with the integrated graphic cards, it is not possible for them to use Stable Diffusion locally. Yes, you can of course run the image generation using CPUs, but you will eventually give it up after minutes and minutes of waiting.

The learning curve is really steep

Even with the help of the stable-diffusion-webui, it is still too easy to get lost in the huge amount of concepts and parameters for beginners. The pretrained model, the LoRA model and the ControlNet model should all be downloaded and placed at different places, the LoRA network plugin and ControlNet plugin must be installed and the corresponding keywords and weights must be added to the prompt.

After the installation, it is now time to step into the ocean of the hyper-parameters tuning. It is hard even for AI professionals to know the exact meaning of all the params and to find a best combination of them to produce good images.

Now that we have mastered all the basics about Stable Diffusion. Finally! Welcome to the area of the prompt engineering. The images we get highly depends on the prompt given, in an unpredictable way. There are quite some guidelines and best practices to learn on how to setup the prompt, it still feels like rolling a dice most of the time.

The quality of the open source models is low

Even at its best setup, the images generated from the open source Stable Diffusion models are hardly acceptable for serious use cases. It might seems promising at first attempts, but there is always something missing even after tens of trails, either the whole picture is not elegant enough, or some details are incorrect and you just cannot make it right.

The limitation of the commercial platforms

Several online solutions, such as Midjourney, emerged to tackle those problems. By training private in-house models and serving them through an online service, the images generated from those platforms are finally commercial ready. By utilizing the cloud hosted GPUs, the service could be used anywhere, as long as a web browser presents.

Everything comes at a price. The cost of the cloud hosted GPUs are surprisingly high these days. No thanks to the competition of the LLM training among large enterprises. And the GPU costs are reflected in the overall costs of using the platforms.

The private in-house model is the most valuable asset to the platform. So it must be kept safe and never open to anyone. This limits the possible use cases for the platforms. The users can only use the model as is, no further customization could be made.

For example, a platform trained a private anime style model that is very good at generating images for the anime characters. And a user wants to generate an image for his favorite anime character. If, by any chance, the model is trained with the data of the character included, the user could get what he wants happily.

However, if the model is trained without those data, the image cannot be generated correctly. At such circumstance, a LoRA fine-tuning could be easily performed on the model to give the user what he wants, only a few images of the character is required. But if the model is not open to the user, and the platform does not support fine-tuning (which is the case for most existing platforms), there is nothing else the user can do.

Cheap spared GPU resources from the decentralized network

Crynux incentivizes the GPU holders to contribute their spared computing resources to others. A decentralized network is formed to connect all the GPU holders and the users. The user sends the task, such as the image generation task and the LoRA fine-tuning task, to the network. And the task is automatically dispatched to some of the GPU holders' nodes according to the node availability. After remote execution, the result is returned to the user. No more local GPU is required on the user's laptop.

The cost is calculated on the granularity of tasks. No cost for anything more than the task consumed should be paid. And since this is the spared computing resource, the users could expect a much lower price compared to the cloud hosted solutions, or buying cards themselves.

From the users perspective, the complexity of the decentralized network is completely hiding behind the applications. The user experience of  a Crynux enabled application remains exactly the same as the cloud hosted ones.

Reshaped user experience from the deep integration of Crynux APIs

It is usually not the infrastructure's duty to optimize the user experience but rather the application's. However, when talking about Stable Diffusion (and other AI model services as well), how the infra is organized has a great impact on the capabilities of the downstream applications. And Crynux has been designed to give all the possibilities to the applications, allowing them to build workflows around Stable Diffusion models in anyway they could imagine.

Let's take the application Character Studio as an example, to see how exactly Crynux could be used to bring new possibilities to the Stable Diffusion ecosystem.

How to train a consistent image generator for the given character

Character Studio focuses on one thing: Train a (LoRA) model for a character, whether it is from the anime or a real person, and generate more images of the same character in different poses consistently.

Before Character Studio, unlike image generation, the training can only be performed by the professional, using tools like lora-scripts. For those who do not have programming or machine learning experiences but very good at learning, lora-scripts could also be used under just a few tutorials. But this is still too hard for everyone.

And for the professionals, the hardest part is "consistent". Usually one will find himself struggling on a very close image that has a key feature missing on the character. And usually the problem lies in the dataset and the annotations, which is hard to find out without hundreds of times of trail and error.

Character Studio makes it possible for everyone to train the model just by providing images and annotations as instructed. The best practices and know-hows of "consistent" is integrated into the app, transparently making the user feels like "it just works".

With the help of Crynux, Character Studio has no requirements on the hardware. It can be used either from the web browser, or as a desktop app on any laptops.

Training LoRA models upon a base model from Crynux marketplace

The training of a LoRA model requires an underlying pretrained base model. And the quality of the base model matters. If the character is from the anime, a base model that is pretrained using a lot of anime images should be used.

The high quality base models are all ready to use in the Crynux marketplace. And can be selected by the users directly in the Character Studio.

After the preparation of the dataset, select a base model, and the training could be started on the Crynux network with just one click.

Note that the base model is not downloaded to the local machine, but is served in the decentralized network. The training task is likely to be dispatched to a node that has the required base model cached for faster execution.

Generating Images using the combination of the LoRA and base models

The trained LoRA model is downloaded to the local laptop to be saved. And Character Studio supports generating images using a combination of the local saved LoRA model and a marketplace hosted base model.

Image generating using the LoRA models from the marketplace is also supported. Users could find the suitable LoRA models (and a pretrained base model) in the marketplace, and use them to generate images directly.

After the training of a model, the model could be hosted on the Crynux marketplace, to be used by others, for both image generation and fine-tuning.

One more thing: the image dataset could still make more benefits for the creator.

Although not implemented in the Character Studio (yet), but the annotated image dataset could make more benefits for the creator thanks to the private data protection technology provided by Crynux.

The creator could provide the dataset to others, to be used in their training tasks, for profit. The dataset is kept private during the whole process, the original data never leaves the owner's computer.

This is a new way for the data owners to make money from their private data, continuously, without the fear of data leakage. And the model developers get more data to produce high quality models.