karlcordes.com

Running Llama.cpp Agents on AMD RX 9060 XT with Docker and gVisor

Mon, 11 May 2026 00:00:00 +0000

Hardware Setup

2x AMD RX 9060 XT GPUs
16GB VRAM per GPU
Total 32GB VRAM available for model loading

Software Stack

llama.cpp

Quantized model inference using GGUF format
Multi-GPU support for AMD RDNA architecture
CPU fallback for non-GPU workloads

Docker with gVisor

gVisor provides user-space container runtime for enhanced security
Isolated from host kernel for agent sandboxing
Suitable for running untrusted or experimental AI agents

Docker Configuration

Base Image

FROM gcr.io/go-containerregistry/docker:27.3.1-dind

GPU Passthrough

--gpus all

Memory Limits

--memory="32g"
--memory-swap="32g"

Volume Mounts

-v /path/to/models:/models:ro
-v /path/to/agents:/agents

Model Loading

Supported Quantizations

Q4_K_M (4-bit, good balance of speed/accuracy)
Q5_K_M (5-bit, improved accuracy)
Q8_0 (8-bit, near-precision)

Model Size Considerations

7B models: ~4-5GB at Q4_K_M
13B models: ~8-9GB at Q4_K_M
Can load multiple smaller models simultaneously

Agent Architecture

Container Structure

agents/
├── agent1/
│   ├── Dockerfile
│   └── main.py
├── agent2/
│   ├── Dockerfile
│   └── main.py
└── shared/
    └── models/

Communication

Inter-agent communication via shared volumes
Model sharing between agents to reduce memory footprint
Centralized logging and monitoring

Performance Considerations

VRAM Management

Monitor VRAM usage with nvidia-smi (for AMD: radeontop)
Implement model unloading for idle agents
Use quantization to fit larger models

Inference Speed

Batch processing for improved throughput
Context window management
Temperature and top-p tuning for response quality

Security Notes

gVisor provides kernel isolation but not network isolation
Additional network policies recommended
Regular container image scanning
Minimal base images reduce attack surface

Troubleshooting

Common Issues

VRAM fragmentation: Use smaller quantizations
Slow inference: Check CPU fallback is not being used
Container crashes: Verify GPU driver compatibility

AMD GPU Specific

Verify ROCm support in Docker image
Check dmesg for GPU errors
Monitor with rocm-smi

Future Improvements

Implement model caching between agents
Add persistent storage for conversation history
Integrate with existing LLM orchestration tools

Compile Emacs 29 with Tree-sitter

Thu, 15 Jun 2023 00:00:00 +0000

Compile emacs with tree-sitter master

Add the deb-src lines to /etc/apt/source.list

``` sudo apt update sudo apt install build-essential sudo (apt build-dep thing) emacs make ```

Checkout tree-sitter

``` make sudo make install ```

``` export LD_LIBRARY_PATH=/usr/local/lib/ make ```

References: https://www.masteringemacs.org/article/how-to-get-started-tree-sitter

Can't start emacs from dmenu. Add this to /etc/ld.so.conf.d/x86_64-linux-gnu.conf

```

/usr/local/lib/x86_64-linux-gnu /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu /usr/local/li ```

Then run ``` sudo ldconfig ```

run emacs and tree-sitter
figure out how to use tree-sitter..

ecowitt to Prometheus using JSON exporter

Tue, 20 Jul 2021 00:00:00 +0000

ecowitt to Prometheus using JSON exporter

Get ecowitt sensor data into InfluxDB

Sun, 15 Mar 2020 00:00:00 +0000

Get ecowitt sensor data into InfluxDB

How to flash coreboot on Lenovo X220

Fri, 16 Dec 2016 03:19:05 +1100

Intro

Laptops are regressing. 2016’s laptops are thinner and sleeker than last year’s - but this usually comes at the cost of user servicability.

Internal batteries are hard to swap out, RAM is soldered to the motherboard. If you want to upgrade any single component, you’re going to need to buy a new laptop.

The Lenovo X220 are great little machines from 2011 that still have plenty of life left in them. I picked up mine on Ebay for AUD $200 - 4GB of RAM, i5 CPU and 120 GB Samsung SSD. Excellent Linux support at a very reasonable price. I use one at work instead of the provided Macbook/Dell options, and it’s great.

Configure a ZFS mirror on Ubuntu 16.04

Sun, 01 May 2016 03:19:05 +1100

Very rough notes on setting up ZFS on Ubuntu 16.04 - with LXD

sudo apt-get install zfsutils-linux

dmesg | grep -i sata | grep 'link up'
[    6.208581] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
  [    6.896647] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
  [    7.216674] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)


k@sagan:~$ sudo zpool create apollo mirror /dev/sdb /dev/sdc
invalid vdev specification
use '-f' to override the following errors:
/dev/sdb does not contain an EFI label but it may contain partition
information in the MBR.
/dev/sdc does not contain an EFI label but it may contain partition
information in the MBR.
# wtf

k@sagan:~$ sudo zpool create -f apollo mirror /dev/sdb /dev/sdc
k@sagan:~$ sudo zpool status
  pool: apollo
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        apollo      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0

errors: No known data errors


k@sagan:~$ sudo zfs create apollo/projects
k@sagan:~$ ls
k@sagan:~$ sudo zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
apollo            372K  5.27T    96K  /apollo
apollo/projects    96K  5.27T    96K  /apollo/projects

Performance

http://blog.ashodnakashian.com/2013/12/18tb-home-nashtpc-with-zfs-on-linux-part-3/

Root Nook Glowlight plus and install Safari Queue

Tue, 16 Feb 2016 03:19:05 +1100

Update: May 23 2016

My Rooted nook locked up and refused to boot properly. I managed to restore it to factory defaults but the screen had permanent distortion.

I returned the Nook to B&N for a replacement. Which is running the stock software.

An E-ink Android device has been on my wish list for a while, but until yesterday I hadn’t seen any high quality examples. The Nook Glowlight Plus looks great, feels sturdy and runs Android, so I picked one up.

Creating RAID 1 MDADM on an existing Ubuntu install

Sat, 23 Aug 2014 00:00:00 +0000

Remove all the old partions on the disks

sudo parted

select /dev/sda
rm 1
rm 2
select /dev/sdb
sudo mkfs.ext4 /dev/sd[a-d]1

#Setup MDADM

sudo mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 /dev/sdb1

karlcordes@spacex:~$ sudo mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm: /dev/sda1 appears to contain an ext2fs file system
size=1953513472K  mtime=Thu Jan  1 10:00:00 1970
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device.  If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: /dev/sdb1 appears to contain an ext2fs file system
size=1953513472K  mtime=Thu Jan  1 10:00:00 1970
mdadm: size set to 1953382208K


karlcordes@spacex:~$ sudo mdadm --create --verbose /dev/md1 --level=mirror --raid-devices=2 /dev/sdc1 /dev/sdd1
mdadm: /dev/sdc1 appears to contain an ext2fs file system
size=1953513472K  mtime=Thu Jan  1 10:00:00 1970
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device.  If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: /dev/sdd1 appears to contain an ext2fs file system
size=1953513472K  mtime=Thu Jan  1 10:00:00 1970
mdadm: size set to 1953382208K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.


karlcordes@spacex:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdd1[1] sdc1[0]
1953382208 blocks super 1.2 [2/2] [UU]
[>....................]  resync =  0.4% (8139008/1953382208) finish=243.9min speed=132882K/sec

md0 : active raid1 sdb1[1] sda1[0]
1953382208 blocks super 1.2 [2/2] [UU]
[>....................]  resync =  0.4% (9560128/1953382208) finish=253.9min speed=127572K/sec


karlcordes@spacex:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdd1[1] sdc1[0]
1953382208 blocks super 1.2 [2/2] [UU]
[=>...................]  resync =  7.2% (140990656/1953382208) finish=222.1min speed=135966K/sec

md0 : active raid1 sdb1[1] sda1[0]
1953382208 blocks super 1.2 [2/2] [UU]
[=>...................]  resync =  6.9% (136385792/1953382208) finish=239.9min speed=126214K/sec

Save the configuration

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

"after you've created your array, it's important to save the configuration in the proper mdadm configuration file. In Ubuntu, this is file /etc/mdadm/mdadm.conf. In some other distributions, this is file /etc/mdadm.conf. Check your distribution's documentation, or look at man mdadm.conf, to see what applies to your distribution.
To save the configuration information:
Ubuntu:

https://raid.wiki.kernel.org/index.php/RAID_setup#Saving_your_RAID_configuration

Add to /etc/fstab

/dev/md0  /mnt/io       ext4  noatime,rw 0 0
/dev/md1  /mnt/europa   ext4  noatime,rw 0 0

CV

Mon, 01 Jan 0001 00:00:00 +0000

Karl Cordes is a software developer and systems administrator with experience at Atlassian, Yahoo, and more.

Skills

Infrastructure & DevOps: AWS, Docker, Kubernetes, Terraform, CI/CD pipelines, monitoring (Prometheus, Grafana)

Languages & Frameworks: Bash, Python, Go, Ruby, JavaScript, PHP, Perl, Flask, Rails, Sinatra

Databases: PostgreSQL, MySQL, MongoDB, Redis, Elasticsearch