jump to navigation

Hardware has arrived! October 4, 2010

Posted by Paul Whalen in ARM, Koji.
Tags: ,

Fedora ARM Koji hardware

Globalscale has finally come through with our remaining builders! Their all online now, ready and enabled in Koji. So we now have 20 GuruPlug Server Plus, 1 SheevaPlug and an OpenRd Ultimate in the ARM Koji build system.

The new GuruPlugs include a small fan to keep the plugs cool, and allow for all features to be used with out the plug randomly rebooting. This fan however comes with a price – the plugs are very loud. For our purposes, they do their job well, but for more typical uses the fan volume is enough to look at other solutions. Additional photo’s of our builders can be found here.

As we stand now we still need as much help with glibc as we can get. This package is significantly blocking us from building Fedora 13 and moving forward with Fedora Arm. If you think you can help us, please feel free to drop in on the IRC, you can find us in #fedora-arm on Freenode, or email. You can use the ARM Koji as a means to test glibc or any other package. For Fedora 13 we are looking at building glibc 2.12-3.


Fedora ARM Update September 16, 2010

Posted by Paul Whalen in ARM, Koji.
Tags: ,
add a comment

Apologies my posts have been sporadic at best. We’ve been trying hard to get Fedora 13 built prior to the release of F14 in November, but have hit a number of roadblocks along the way. I’m trying to commit to at least one blog post a week updating the status of the Fedora ARM, mentioning current problems slowing us down and to provide an idea of how long till we’re able to produce something people can test.
On the hardware side. We have been assured once again that our remaining builders will be arriving sometime next week. We’re expecting an additional 2 SheevaPlugs and 16 GuruPlug Server Plus. We have already received a BeagleBoard XM but have yet to get Fedora up and running, this should change in the next day or so and the XM will be added to our pool of builders.
Dennis Gilmore came to visit us here in Toronto on August 25th-27th. Dennis was here to assist us in planning the next steps for Fedora ARM in the immediate future, but to also discuss long terms plans including support for newer ARM processors.
We have completed our first successful run of ‘build-previous’. The ‘build-previous’ script queries the primary Koji server instance to find out the NVR of each package that shipped with Fedora 13, then downloads the previous version and submits it to Koji to build on ARM. During this first pass our queue sky-rocketed to roughly 1300 packages waiting to be built, and took roughly 9 days to complete. The result left us with more successful builds with a total of 7744 packages built, out of the roughly 16,000 in F13. So we’ve got a long way to go still.
We’ve made some minor changes to the hardware here. For testing purposes we have two of our GuruPlugs using eSATA drives rather then SD. The majority of our builders are using a microSD card for the root file system, with additional storage and swap space via NFS. The change to eSATA increased the performance significantly, but also increases our costs with each builder requiring their own external drive and enclosure.
Before our next execution of build previous we are attempting to build glibc on ARM, which requires some edits to the existing spec file and the inclusion of the ports tarball found in the more recent releases of glibc. At present our attempts have been unsuccessful. If anyone is able to assist us, your help would be greatly appreciated. We’re trying to build ‘glibc-2.12-3.src.rpm’ in particular. The ARM Koji is available for use by all with a FAS account and can be found at http://arm.koji.fedoraproject.org.

Update – Fedora Arm Koji August 19, 2010

Posted by Paul Whalen in ARM, Koji.
Tags: ,

We’ve been working at importing the remaining Fedora 13 packages into the ARM Koji, but have suffered a number of set backs including some issues with circular dependencies as mentioned in my previous blog. We have a total of six full time builders so the building is going very slow. Gcc alone took approximately 52 hours to complete. In order for us to build gcc on ARM we had to modify the spec file and make a few changes provided by the previous ARM Koji as well as a few of our own, these changes will be committed upstream to ensure gcc will build on ARM in the future.

Currently we have a modified version of the build-previous.py script written by Dennis Gilmore importing packages into the ARM Koji. We have just under 5000 completed builds, and a growing queue of roughly 700 packages waiting to built. The modification made to the script includes checking for dependencies of a given package to reduce build attempts for packages that will not complete on the first pass due to missing deps. Each subsequent pass should build more and more packages until F13 is completed. Noarch packages are also being directly imported rather then building again, in attempt to speed up this process. We’ve been waiting on the arrival of our additional 20 GuruPlug Server Plus builders which will also make this less painful in the future. Globalscale assures us they should be arriving within a few weeks. We shall see.

Our builders have been crashing for reasons unknown to us. Most likely thermal issues already identified with the GuruPlug, so we are going to also experiment with additional cooling. The ‘server farm’ will also be moved into a supply closet that is very cool so hopefully this also helps to reduce the temperatures while we punish these little computers with our mass rpm assaults.

We have arranged to have Dennis Gilmore visit us here in Toronto next week to assist us moving forward with the ARM Koji (thank you very very much for your continued help, Dennis ).

We’re also hoping to track down a script that checks the repository for broken dependencies while we build, however we haven’t been successful at the time of this posting. If anyone can assist with this it would be greatly appreciated.

Fedora ARM Build System – First Koji breakage August 3, 2010

Posted by Paul Whalen in ARM, Koji.
Tags: ,
add a comment

Experienced my first breakage of Koji this past weekend due to a build of db4-4.8.24-1.fc13. It created dependency issues with rpm, perl, pam and cyrus-sasl. As a result mock was unable to initiate the build root, thus bringing the ARM Koji to a stand still. To correct this issue I attempted to rebuild the packages manually using mock, using newer versions of the packages, however there were still dependency issues with db4-4.7. This seemed to be an endless loop until compat-db was brought to my attention. This binary provides compatibility with the older versions of db4 and breaks dependency loop, allowing us to move forward and continue building packages.

Currently our list of packages that require patching has been actioned and Bugzilla tickets are soon to follow. Some of the packages have dependency issues that need to be resolved first to ensure they build on ARM, while others were patched without issue on all platforms. Again this list will grow substantially as more builds are attempted, so those looking to help out, your contributions would be greatly appreciated.

Fedora ARM Build System Status Update July 9, 2010

Posted by Paul Whalen in ARM, Koji.
Tags: ,

Koji Builders

OLPC announced earlier this week that Fedora 11 would be the operating system in their XO-1 release, and their intention to use Fedora in subsequent releases in the future.
In light of this announcement I wanted to provide an update on the status of the Koji Build System for the ARM architecture. In terms of builders we currently have a total of 5 GuruPlug Server Plus, 1 SheevaPlug and an OpenRD connected locally as well as 5 additional remote builders being provided by the folks at One Laptop Per Child and Marvell. The OpenRD and Sheeva are plugged into a Gigabit Switch and the GuruPlugs are connected to a 10/100 switch (due to an overheating issue with the hardware). We have ordered more builders however they will not be arriving for another month.
We have imported approximately 3000 ARM built packages provided by the previous Koji instance and are in the process of building the remaining packages for Fedora 13. We have compiled a list of 44 packages that were patched for Fedora 12 but not commited to upstream. The first step will be to patch the Fedora 13 version of the package, make sure that the source will build successfully on ARM, then verify it still builds on the primary architectures with the patch applied. Once completed a Bugzilla ticket will be filed to notify the package maintainer, asking them to accept the changes made and submit to CVS. Due to time constraints we will wait for 3 days to ensure there is no objection from the package maintainer and providing them time to submit to CVS themselves, if no response or action takes place it will be submitted to CVS.

Due to an outage on the Fedora Koji this weekend, those Fedora users who would like to test packages, please feel free to use ours! Currently the ARM Koji is functional and Fedora users can use their FAS credentials to sign in and submit packages to be built for ARM. The web interface can be found here

Importing SRPM’s and RPM’s into Koji June 25, 2010

Posted by Paul Whalen in Uncategorized.
add a comment

In order to get our instance of Koji up and running we need to import packages already built on ARM. The first step is to look at the packages that are in Fedora 13, that were also in Fedora 12. This can be done by looking at the package list in Fedora 13 found here, and including only rpms ending in ‘fc12.src.rpm’ and ‘fc11.src.rpm’. Once that list is compiled, it was then checked against the list of existing packages built on ARM, making sure the version number also matched. The list of packages ended up being 3024/9157.
Both the source and binary rpms were available on our local repo so importing them into Koji was a simple process. A short script was created to go through the list and import both the source and binary rpm packages.
while read line
echo 'importing' $line
koji import /var/www/html/yum/base/12/arm/$line*.rpm /var/www/html/yum/source/12/arm/$line*.rpm
done < package.lst

Once all the packages are imported, to tag them all in Koji enter the following command:
koji list-untagged | xargs -n 1 koji call tagBuildBypass dist-f13

This tags all untagged packages as dist-f13.

Adding Swap over NFS for Koji Builders June 18, 2010

Posted by Paul Whalen in ARM, Koji.
add a comment

In our efforts to set up a Koji Build farm for Fedora we have decided to use the GuruPlug Server Plus. The GuruPlug has very limited memory with only 512 MB, previously the build farm used boards that had 2 GB of ram. In order to get around this limitation we have decided to add swap space over NFS. Unfortunately the kernel doesn’t support swap over NFS, for good reason due to the unreliability of NFS. In our situation these risks don’t apply as we are already using NFS for additional space during the build process, if the share becomes unavailable the system will be impacted regardless. In order to trick our system and use swap over NFS we created an ext3 file system named swap0.ext3 on an existing NFS share.
First create the file and format using ext3:
dd if=/dev/zero of=swap0.ext3 bs=1024 count=7864320
mkfs -t ext3 swap0.ext3

This creates a file of approximately 8.1 GB, to change this amount simply edit the count value to reflect the desired amount. This is calculated by multiplying 1024*x, where x is the MB.
Mount the ext3 file system by adding it to ‘/etc/fstab’ (path should be changed, should be executed as root):
echo '/arm2/swap0.ext3 /swap0 ext3 defaults,noauto 0 0' >> /etc/fstab
mkdir /swap0
mount -o loop /swap0

Once mounted move to that directory and create the actual swap file using:
dd if=/dev/zero of=swap0 bs=1024 count=7860000
mkswap swap0
swapon swap0

As we have added the swap space to the fstab including the noauto option it will not be automatically be mounted at boot, instead we have added the mount to ‘/etc/rc.d/rc.local’. This was done to ensure if the share was unavailable during the boot, the system wouldn’t stall waiting for the share.
mount -o loop /swap0
swapon /swap0/swap0

By giving the builders a swap file the createrepo task being run by Koji was completed successfully. With no swap the task would fail. The actual amount of swap space being used was limited to approximately 500 MB so its possible a smaller swap file would suffice, however moving forward other tasks may require the additional resources. This solution is far from ideal, and is quite messy but produced the desired outcome.

GuruPlug Server Plus Thermal Issues – Design Modifications Set for June May 31, 2010

Posted by Paul Whalen in ARM, Koji.
add a comment

In my previous blog post I talked about installing Fedora 12 on the micro-SD for use with the GuruPlug Server Plus. Unfortunately the plug reboots when connected to Gigabit Ethernet due to a known thermal issue. Some have come to creative solutions to prevent this, by voiding their warranty, opening up the plug and attaching a massive heat sink to it. Others have made some attempts through active cooling and attaching a fan.
I have opted to contact Globalscale and request a replacement. During previous contact with them I was told this was indeed a known thermal issue and Globalscale have revised their design and will be shipping the new and improved GuruPlug Server Plus in June. This bodes well for us as we have placed an order for 20 additional GuruPlug’s for the planned Koji Server Farm supporting ARM for Fedora, that order is planned to ship in June and include the new design.
In the meantime I have connected the GuruPlug to a fast Ethernet switch, which prevents the rebooting issue.

More information can be found in this forum.

Installing Fedora 12 on a Guruplug May 13, 2010

Posted by Paul Whalen in ARM.

The first GuruPlug arrived recently for our planned ARM Koji farrm for the Fedora Project. The plug ships with Debian, so the first step was to install Fedora 12. With 512 MB NAND we decided to install the Fedora root file system provided here onto a 4 GB class 6 micro SD card. Initially we are going to use the Kernel provided on the GuruPlug and then upgrade at a later date. This will also leave the Debian file system in place in case it is needed later. The micro SD card was previously used in our OpenRD client and therefore has some existing partitioning which I left, so if you plan on following this as a guide make sure you substitute the major and minor device numbers so that your Uboot commands work for your partitioning.
First untar the provided rootfs tarball onto the SD card (pls change mount point of the SD card to match yours):
** The SD card should be formatted as an ext2/3 filesystem, and below commands executed as root**
[root@newzealand ~]# tar -jxf rootfs-f12.tar.bz2
[root@newzealand ~]# cp -r rootfs-f12/* /media/flash/

Insert the micro SD into your GuruPlug, connect the JTAG breakout box (hopefully you have one) provided to the GuruPlug and connect the USB to another PC running Linux (hopefully Fedora!). You will need to have the package Minicom installed, if you do already please skip the first step:
[root@newzealand ~]# yum install minicom
[root@newzealand ~]# minicom -s

Running the last command as root will allow you to configure minicom to use the USB you have connected to the GuruPlug as a serial connection. Go to the option ‘Serial port setup’ and hit enter. Edit the fields as shown below, substituting the USB connection if needed to match yours.
| A - Serial Device : /dev/ttyUSB0
| B - Lockfile Location : /var/lock
| C - Callin Program :
| D - Callout Program :
| E - Bps/Par/Bits : 115200 8N1
| F - Hardware Flow Control : No
| G - Software Flow Control : No

Once completed run the following command to connect to the serial connection on the GuruPlug:
[root@newzealand ~]# minicom

Power on the GuruPlug and you should see the boot up sequence, allow it to boot to Debian so you can find out the major and minor device numbers for the micro SD card that has the Fedora rootfs on it. Run the following command when you have logged into Debian as root:
sheevaplug-debian:/dev# ls -l /dev/sd*
brw-rw---- 1 root floppy 8, 0 2009-08-08 09:30 /dev/sda
brw-rw---- 1 root floppy 8, 16 2009-08-08 09:30 /dev/sdb
brw-rw---- 1 root floppy 8, 17 2009-08-08 09:30 /dev/sdb1
brw-rw---- 1 root floppy 8, 18 2009-08-08 09:30 /dev/sdb2

This shows that the major and minor device numbers for ‘/dev/sdb2’ which I put the rootfs on, are 8 and 18 respectively. That number needs to be converted to hex, for entry in Uboot – 8,18 in decimal converts to 8,12 in Hex, for entry in Uboot this will be entered as ‘0812’. Again, yours will be different, so be sure to substitute the correct values.
Reboot the GuruPlug and this time hit any key before the system boots into Debian. Once at the Marvell prompt type:
Marvell>> nand read.e 0x6400000 0x100000 0x400000
Marvell>> setenv bootargs console=ttyS0,115200 root=0812 rootdelay=10
Marvell>> bootm 0x6400000

This should boot the GuruPlug into Fedora 12, the default password for root is ‘fedoraarm’. I had some issues booting the plug initially as it was not recognizing the micro SD card and finding the rootfs, once I added the rootdelay of 10 this worked, however lesser numbers did not. You will also of course have issues if the major and minor numbers are incorrect, so please make sure this is correct and converted to Hex. If you are happy with the these settings and would like the GuruPlug to boot using the Fedora file system you will need to reboot, again hitting any key to get to the Marvell prompt, this time you will need to edit and remove existing boot parameters, below is the default output of the ‘printenv’ command:
Marvell>> printenv
bootcmd=setenv ethact egiga0; ${x_bootcmd_ethernet}; setenv ethact egiga1; ${x_;
x_bootcmd_usb=usb start
x_bootcmd_kernel=nand read.e 0x6400000 0x100000 0x400000
x_bootargs_root=ubi.mtd=2 root=ubi0:rootfs rootfstype=ubifs

To change the location of the rootfs, enter commands below, changing the values again to represent your configuration:
Marvell>> editenv x_bootargs_root
edit: ubi.mtd=2 root=0812 rootdelay=10
Marvell>> saveenv
Saving Environment to NAND...
Erasing Nand...
Erasing at 0x40000 -- 100% complete.
Writing to Nand... done
Marvell>> reset

Hopefully you see ‘Welcome to Fedora’ after the boot. If you would like to change the plug back to the default install of Debian, then you simply have to edit the boot parameters and change back to their original values, saving once completed.

Koji Daemon – Builder – Enabled, but not Ready April 14, 2010

Posted by Paul Whalen in ARM, Koji.
add a comment

After setting up the Koji Hub the next step was to get some of the builders on-line. Running the command:
[root@fedora-arm ~]#yum install koji-builder

Then editing the ‘/etc/kojid/kojid.conf’ to reflect our set up on Hong Kong:
; The URL for the xmlrpc server
; the username has to be the same as what you used with add-host
; in this example follow as below
user = arm-001-007

The user name section was not present, so the lines were taken from the example provided at Fedora.
Also in the the ”/etc/kojid/kojid.conf uncomment the following lines and edit according to the the name of the certificate created for the specific builder:
;configuration for SSL authentication
;client certificate
cert = /etc/kojid/arm-001-007.pem
;certificate of the CA that issued the client certificate
ca = /etc/kojid/koji_ca_cert.crt
;certificate of the CA that issued the HTTP server certificate
serverca = /etc/kojid/koji_ca_cert.crt

The certificates mentioned must be scp’d from the directory on Hong Kong ‘/etc/pki/koji’.
The first few machines will also be added to the create repo channel to handle repo creation as required by Kojira. Builders are added to the ‘default’ channel upon creation. Channels are used as a method to control which builders perform certain tasks.
[kojiadmin@hongkong arm]$ koji add-host-to-channel arm-001-007 createrepo

Before starting kojid on the builder make sure it has been added to the Koji database or there will be a permission issue. The command to add a builder to Koji is
[kojiadmin@hongkong arm]$ koji add-host arm-001-007 arm

If in error kojid was started prior to the addition in the db simply remove the created session from the sessions table, then the user from the user table in postgresql.
The kojid daemon was then started with:
/sbin/service kojid start

At this point the created builder was enabled but not ready. After checking the hub for configurations issues none could be found. Looking at the logs on the builder the error reported was insufficient disk space. The disk had 3.2G available but for some reason Koji was asking for another 8G. Shutting down the vm, the following commands were used to resize, adding an additional 10G.
dd if=/dev/zero bs=1M count=10000 >> rootfs-arm-f12-007
e2fsck -f rootfs-arm-f12-007
resize2fs -p rootfs-arm-f12-007

Once completed the vm was restarted and after checking was enabled and ready.