Hugging Face Party @ PyTorch Conference

community

Activity Feed Request to join this org

AI & ML interests

None defined yet.

Recent Activity

woojun-jung authored a paper 18 days ago

Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models

xianbao authored a paper about 2 months ago

RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies

osanseviero authored a paper 3 months ago

EmbeddingGemma: Powerful and Lightweight Text Representations

View all activity

csabakecskemeti

posted an update 3 days ago

Post

2974

Just sharing a result of a homelab infrastructure experiment:

I've managed to setup a distributed inference infra at home using a DGX Spark (128GB unified gddr6) and a linux workstation with an RTX 6000 Pro (96GB gddr7) connected via 100Gbps RoCEv2. The model I've used (https://lnkd.in/gx6J7YuB) is about 140GB so could not fit either of the GPU. Full setup and tutorial soon on devquasar.com

Screen recording:
https://lnkd.in/gKM9H5GJ

3 replies

danielhanchen

posted an update 6 days ago

Post

3620

You can now run GLM-4.7, the new 355B parameter SOTA model on your local device (128GB RAM).✨

The model achieves SOTA performance on coding, agentic and chat benchmarks.

GGUF: unsloth/GLM-4.7-GGUF
Guide: https://docs.unsloth.ai/models/glm-4.7

2 replies

danielhanchen

posted an update 11 days ago

Post

2213

Google releases FunctionGemma, a new 270M parameter model that runs on just 0.5 GB RAM.✨

Built for tool-calling, run locally on your phone at 50+ tokens/s, or fine-tune with Unsloth & deploy to your phone.

GGUF: unsloth/functiongemma-270m-it-GGUF
Docs + Notebook: https://docs.unsloth.ai/models/functiongemma

2 replies

danielhanchen

posted an update 14 days ago

Post

5258

NVIDIA releases Nemotron 3 Nano, a new 30B hybrid reasoning model! 🔥

Has 1M context window & best in class performance for SWE-Bench, reasoning & chat. Run the MoE model locally with 24GB RAM.

GGUF: unsloth/Nemotron-3-Nano-30B-A3B-GGUF
💚 Step-by-step Guide: https://docs.unsloth.ai/models/nemotron-3

1 reply

danielhanchen

posted an update 18 days ago

Post

2007

Mistral's new SOTA coding models Devstral 2 can now be Run locally! (25GB RAM) 🐱
We fixed the chat template, so performance should be much better now!
24B: unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF
123B: unsloth/Devstral-2-123B-Instruct-2512-GGUF

🧡Step-by-step Guide: https://docs.unsloth.ai/models/devstral-2

csabakecskemeti

posted an update 25 days ago

Post

1251

FYI: Mistral.Ministral-3 dequantizer FP8->BF16

https://github.com/csabakecskemeti/ministral-3_dequantizer_fp8-bf16

(The instruct model weights are in FP8)

danielhanchen

posted an update 26 days ago

Post

3650

Mistral's new Ministral 3 models can now be Run & Fine-tuned locally! (16GB RAM)
Ministral 3 have vision support and the best-in-class performance for their sizes.
14B Instruct GGUF: unsloth/Ministral-3-14B-Instruct-2512-GGUF
14B Reasoning GGUF: unsloth/Ministral-3-14B-Reasoning-2512-GGUF

🐱 Step-by-step Guide: https://docs.unsloth.ai/new/ministral-3
All GGUFs, BnB, FP8 etc. variants uploads: https://huggingface.co/collections/unsloth/ministral-3

3 replies

csabakecskemeti

posted an update 26 days ago

Post

2070

Looking for some help to test an INT8 Deepseek 3.2:
SGLang supports Channel wise INT8 quants on CPUs with AMX instructions (Xeon 5 and above AFAIK)
https://lmsys.org/blog/2025-07-14-intel-xeon-optimization/

Currently uploading an INT8 version of Deepseek 3.2 Speciale:
DevQuasar/deepseek-ai.DeepSeek-V3.2-Speciale-Channel-INT8

I cannot test this I'm on AMD
"AssertionError: W8A8Int8LinearMethod on CPU requires that CPU has AMX support"
(I assumed it can fall back to some non optimized kernel but seems not)

If anyone with the required resources (Intel Xeon 5/6 + ~768-1TB ram) can help to test this that would be awesome.

If you have hints how to make this work on AMD Threadripper 7000 Pro series please guide me.

Thanks all!

8 replies

danielhanchen

posted an update about 1 month ago

Post

8455

Qwen3-Next can now be Run locally! (30GB RAM)
Instruct GGUF: unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF

The models come in Thinking and Instruct versions and utilize a new architecture, allowing it to have ~10x faster inference than Qwen32B.
💜 Step-by-step Guide: https://docs.unsloth.ai/models/qwen3-next

Thinking GGUF: unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF

csabakecskemeti

posted an update about 2 months ago

Post

303

Recently there are so much activity on token efficient formats, I've also build a package (inspired by toon).

Deep-TOON

My goal was to token efficiently handle json structures with complex embeddings.

So this is what I've built on the weekend. Feel free try:

https://pypi.org/project/deep-toon/0.1.0/

danielhanchen

posted an update about 2 months ago

Post

4364

You can now run Kimi K2 Thinking locally with our Dynamic 1-bit GGUFs: unsloth/Kimi-K2-Thinking-GGUF

We shrank the 1T model to 245GB (-62%) & retained ~85% of accuracy on Aider Polyglot. Run on >247GB RAM for fast inference.

We also collaborated with the Moonshot AI Kimi team on a system prompt fix! 🥰

Guide + fix details: https://docs.unsloth.ai/models/kimi-k2-thinking-how-to-run-locally