kernelpanic/iglunix-autobuild
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Init lazy-autobuild

Browse source
This commit is contained in:
aheirman 2021-02-04 17:58:42 +01:00
commit eca12bc30e
19 changed files with 739 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.gitignore vendored Normal file
View file

@ -0,0 +1,4 @@
root/
build/
docker_move_tar_load
docker_delete_abyss

11
Dockerfile_update Normal file
View file

@ -0,0 +1,11 @@
FROM abyssos/abyss:dev
COPY dont_use_here/init_custom_container /init_custom_container
COPY dont_use_here/bootup_script /bootup_script
COPY dont_use_here/distccd_cmd_list /distccd_cmd_list
COPY dont_use_here/compile /compile
CMD ["sh", "init_custom_container"]

63
README.md Normal file
View file

@ -0,0 +1,63 @@
Welcome, this is lazy-autobuild.
Thu GNU General Public License v3 licesed (mostly) automatic build system for Lazybox.
Lazybox for the uninitiated is a inux distribution designed to not use _GNU_, sadly this is quit hard. GNU make is still available inside this environment.
# Usage
## Fase 1: Docker
This will create an llvm/musl toolchain.
I've heard that you guys don't like waiting for code to be compiled.
A docker swarm to the recue, completely over engineered.
Make your main pc/laptop the manager of the swarm using:
`docker swarm init --advertise-addr [The IP-address of your device you want to use local/global]`
It will return a command that allows your other pcs to join, use it.
On the main node run: `docker network create -d overlay my_docker_net`
The problem is that we can't chain commands when starting a docker service create, so we'll have to modify the base abyssos/abyss:dev docker container. We'll need to dowload packages for the installation, we'll add these to the our modified package.
This method might also reduce internet usage.
To initialize run ```docker_init_clean```, this will create lazy_abyss.tar, this will have to be manually moved to all docker nodes and then loaded using: ``` docker load -i lazy_abyss.tar```
To start the distributed compiler `docker_start_farm` can be used. Prior to execution modify `docker_start_farm` to suit the number of jobs you need to run.
Run ```docker ps -a``` to find the container id of your work nodes
and execute a shell on a node of your chosing using. This can be done as follows:
`docker exec -it [container id] /bin/sh`
Modify DISTCC_HOSTS in `compile` in one container to point to all the docker distcc volunteers. Afterwards run `compile`.
When your compiles are finished, `docker_rm_farm` can be used to remove the compile farm.
It could be useful to leave the compile farm around for the compilation indide the chroot. This is not yet supported.
## Fase 2: Chroot
This part is significantly less over engineerd (luckily).
There are 4 commands the user should be aware of, they are writen in chronological order.
* chroot_prepare_lazy
This creates the chroot and cleans the most important locations (/lib, /bin, /sbin, /usr, /lazybox, /etc).
* chroot_fetch
This command will fetch the sources needed to get curl working inside the chroot. And some files for to be used inside the chroot environmen. This command will automatically be execute by `chroot_prepare_lazy`. It it __NOT NEEDED__ to be manually exected. But it can be.
* chroot_lazy
This will chroot into the lazybox environment.
* inside_chroot
This is the script that has to be executed inside the chroot to compile all the required apckages in the correct order.
This will also generate a
## Fase 3: Boot
Use virt-manager or qemu to boot the `lazybox.img`.
Sources:
* https://www.youtube.com/watch?v=nGSNULpHHZc
* https://docs.docker.com/network/overlay/
* https://docs.docker.com/registry/

100
bugs/om4-build-twice Normal file
View file

@ -0,0 +1,100 @@
Going to build: om4
stat: 'out': No such file or directory
checking for C compiler... clang
checking for -w compiler flag... yes
checking for OS... Linux
checking for __dead... no
checking for __dead2... no
checking for __attribute__((__noreturn__))... yes
checking for ohash_init... no
checking for pledge... no
checking for reallocarray... yes
checking for strlcpy... yes
checking for strtonum... no
creating Makefile... done
--- parser.c ---
--- eval.o ---
--- expr.o ---
--- look.o ---
--- main.o ---
--- misc.o ---
--- gnum4.o ---
--- trace.o ---
--- parser.h ---
--- ohash.o ---
--- reallocarray.o ---
--- strlcpy.o ---
--- strtonum.o ---
--- parser.c ---
yacc -d parser.y && mv y.tab.c parser.c && mv y.tab.h parser.h
--- eval.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c eval.c
--- expr.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c expr.c
--- look.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c look.c
--- main.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c main.c
--- misc.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c misc.c
--- gnum4.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c gnum4.c
--- trace.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c trace.c
--- parser.h ---
yacc -d parser.y && mv y.tab.c parser.c && mv y.tab.h parser.h
--- ohash.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c ohash.c
--- reallocarray.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c reallocarray.c
--- strlcpy.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c strlcpy.c
--- strtonum.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c strtonum.c
--- tokenizer.o ---
lex tokenizer.l
lex go brr
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c -o tokenizer.o lex.yy.c
--- parser.c ---
mv: bad 'y.tab.c': No such file or directory
*** [parser.c] Error code 1
make: stopped in /lazybox/pkgs/om4/src/om4-6.7
--- tokenizer.o ---
rm -f lex.yy.c
1 error
make: stopped in /lazybox/pkgs/om4/src/om4-6.7
install: can't stat 'om4': No such file or directory
doc
Premature exit
# cd lazybox/pkgs/om4/
# ../../laz
lazy.sh lazypkg.sh
# ../../lazypkg.sh
checking for C compiler... clang
checking for -w compiler flag... yes
checking for OS... Linux
checking for __dead... no
checking for __dead2... no
checking for __attribute__((__noreturn__))... yes
checking for ohash_init... no
checking for pledge... no
checking for reallocarray... yes
checking for strlcpy... yes
checking for strtonum... no
creating Makefile... done
--- parser.o ---
clang -DEXTENDED -I. -w -D_GNU_SOURCE -c parser.c
--- om4 ---
clang -o om4 eval.o expr.o look.o main.o misc.o gnum4.o trace.o tokenizer.o parser.o ohash.o reallocarray.o strlcpy.o strtonum.o -lm
doc
# ls out/om
om4-doc.6.7.tar.xz om4-doc/ om4.6.7.tar.xz om4/
# ls out/om4/usr/bin/
m4 om4
# ls out/om4/usr/bin/om4
out/om4/usr/bin/om4
# out/om4/usr/bin/om4
^Cm4: interrupted.
#

60
chroot_fetch Executable file
View file

@ -0,0 +1,60 @@
#!/bin/bash
#-----
# Packages
#-----
echo "------------------"
echo "Primary packages"
echo "------------------"
packages=(mksh bmake gmake busybox toybox libressl cmake curl rsync linux flex byacc om4 zlib expat samurai libffi python ca-certificates)
root=$(pwd)/root/
for pkg in ${packages[@]};do
echo "$pkg"
source ${root}/lazybox/pkgs/${pkg}/build.sh
cd ${root}/lazybox/pkgs/${pkg}/
#Only needed if you want to clean
#sudo rm -rf src
if test -d "src"; then
echo "Already got the source for ${pkg}?! No need to download.."
else
rm src 2> /dev/null
#src could be a file (file != dir)
mkdir src
cd src
fetch
fi
done
#-----
# tmp Packages
#-----
echo "------------------"
echo "Temporary packages"
echo "------------------"
mkdir ${root}/lazybox/tmp_pkgs/
#echo "ibara baseutils"
#cd ${root}/lazybox/tmp_pkgs/
#git clone https://github.com/ibara/baseutils.git
echo "------------------------------------"
echo " shell scripts/single files"
echo "------------------------------------"
cp ${root}/../dont_use_here/inside_chroot ${root}
cp ${root}/../dont_use_here/linux_header_only_patch ${root}
cp ${root}/../dont_use_here/inittab ${root}/etc/inittab
echo "---------------------"
echo "finished chroot_fetch"
echo "---------------------"

27
chroot_lazy Executable file
View file

@ -0,0 +1,27 @@
#!/bin/bash
cd ./root
echo "Going to chroot, woot!"
mkdir proc
mkdir sys
mkdir dev
mkdir tmp
if true
then
sudo mount --bind /tmp tmp/
sudo mount -t proc /proc proc/
sudo mount -t sysfs /sys sys/
sudo mount --rbind /dev dev/
fi
sudo chroot ./ /bin/sh
sudo umount root/tmp/
sudo umount root/proc/
sudo umount root/sys/
sudo mount --make-rprivate root/dev/
sudo umount -l root/dev/

130
chroot_prepare_lazy Executable file
View file

@ -0,0 +1,130 @@
#! /bin/bash
if [ "$#" -ne 1 ]; then
echo "You must enter exactly the container id of the docker in which the compilation took place. (the client not the volunteers)"
exit -1
fi
#Unmount the mounts
sudo umount root/tmp/
sudo umount root/proc/
sudo umount root/sys/
sudo mount --make-rprivate root/dev/
sudo umount -l root/dev/
#Delete everything!
#umout has problems :(
# Just delete a couple of this ot clean it..
#sudo rm -r -f ./root
if true; then
sudo rm -rf ./root/lib
sudo rm -rf ./root/bin
sudo rm -rf ./root/sbin
sudo rm -rf ./root/usr
sudo rm -rf ./root/lazybox
sudo rm -rf ./root/etc
fi
mkdir root
mkdir root/bin
mkdir root/sbin
mkdir root/usr
mkdir root/usr/sbin
mkdir root/etc
docker cp $1:/lazybox/pkgs/llvm/out/llvm.11.0.1.tar.xz ./root/llvm.tar.xz
docker cp $1:/lazybox/pkgs/musl/out/musl-dev.1.2.2.tar.xz ./root/musl-dev.tar.xz
docker cp $1:/lazybox/pkgs/musl/out/musl.1.2.2.tar.xz ./root/musl.tar.xz
docker cp $1:/lazybox/pkgs/bmake/out/bmake.20210110.tar.xz ./root/bmake.tar.xz
cd ./root
ROOT=$(pwd)
tar -xf musl.tar.xz -C ./
tar -xf musl-dev.tar.xz -C ./
tar -xf llvm.tar.xz -C ./
tar -xf bmake.tar.xz -C ./
#---
wget -nc -q https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-x86_64
chmod +x ./busybox-x86_64
mv busybox-x86_64 ./bin/busybox
wget -nc -q http://landley.net/toybox/bin/toybox-x86_64
chmod +x ./toybox-x86_64
mv toybox-x86_64 ./bin/toybox
echo ""
#---
git clone https://github.com/Ella-0/lazybox.git
#----
# Mount some things, we'll need it later
mkdir proc
mkdir sys
mkdir dev
mkdir tmp
if false
then
sudo mount --bind /tmp tmp/
sudo mount -t proc /proc proc/
sudo mount -t sysfs /sys sys/
sudo mount --rbind /dev dev/
fi
echo "mounted tmp, proc, sysfs, rbind"
# We'll need to compile mksh
# But we don't have tools to get the sources from inside the chroot yet..
cd ..
echo "Starting chroot_fetch"
./chroot_fetch
echo "Need to compile a lex file in om4, because of a circular dependency with flex"
#For the other attempt at fixing this circular dependency see inside_chroot
cd ${ROOT}/lazybox/pkgs/om4/src/om4-*/
lex ./tokenizer.l
sudo chroot ${ROOT} /bin/busybox --install -s
echo "installed busybox"
printf "export PATH=/bin:/usr/bin:/usr/sbin:\nexport JOBS=16\n" > ${ROOT}/etc/lazypkg.conf
#--------------------------------------
echo "---------------------------------"
echo "Starting networking configuration"
echo "---------------------------------"
RANDOM=$$
HOSTNAME="lazybox-$RANDOM"
echo "Hostname: ${HOSTNAME}"
mkdir ${ROOT}/etc
echo "$HOSTNAME" > ${ROOT}/etc/hostname
sudo cp /etc/resolv.conf ${ROOT}/etc/resolv.conf
echo "Copied the host resolv.conf into the target"
echo "Feel free to override the generated network settings prior to chrooting into the system"
echo "Finished prepare_chroot_lazy"

41
docker_init_clean Executable file
View file

@ -0,0 +1,41 @@
#! /bin/sh
#Stop & remove all the old
docker service rm my_compile_farm || true
docker image rm local:lazy_abyss || true
rm image_name || true
rm lazy_abyss.tar || true
echo "Cleaned and ready for the first/next attempt!"
#Create new modified container
docker build . --iidfile image_name --file Dockerfile_update
image_name="`cat image_name`" #This is an intermidiary image
echo "docker run starting, this may take a while"
docker run --name updated_abyss ${image_name}
echo "docker run finished"
if false
then
docker run -d -p 5000:5000 --name my_registry registry:2
echo "Created my_registry"
docker commit updated_abyss localhost:5000/lazy_abyss
echo "Created localhost:5000:lazy_abyss"
docker push localhost:5000/lazy_abyss
echo "pushed localhost:5000:lazy_abyss"
else
docker commit updated_abyss local:lazy_abyss
docker save -o ./lazy_abyss.tar local:lazy_abyss
echo "Move this tar to all nodes, then \"docker load -i lazy_abyss.tar\""
fi
#docker image rm ${image_name}
#echo "Removed intermidiary ${image_name}"

24
docker_port_forward Executable file
View file

@ -0,0 +1,24 @@
#! /bin/sh
# TCP port 2377 for cluster management communications
# TCP and UDP port 7946 for communication among nodes
# UDP port 4789 for overlay network traffic
#
# See https://docs.docker.com/network/overlay/
sudo iptables -I INPUT 1 -p tcp --dport 2377 -j ACCEPT
sudo iptables -I OUTPUT 1 -p tcp --dport 2377 -j ACCEPT
sudo iptables -I INPUT 1 -p tcp --dport 7946 -j ACCEPT
sudo iptables -I OUTPUT 1 -p tcp --dport 7946 -j ACCEPT
sudo iptables -I INPUT 1 -p udp --dport 7946 -j ACCEPT
sudo iptables -I OUTPUT 1 -p udp --dport 7946 -j ACCEPT
sudo iptables -I INPUT 1 -p udp --dport 4789 -j ACCEPT
sudo iptables -I OUTPUT 1 -p udp --dport 4789 -j ACCEPT

5
docker_remove_all_containers Executable file
View file

@ -0,0 +1,5 @@
#! /bin/sh
docker container rm -f $(docker container ls -aq)

5
docker_rm_farm Executable file
View file

@ -0,0 +1,5 @@
#! /bin/sh
docker service rm my_compile_farm

20
docker_start_farm Executable file
View file

@ -0,0 +1,20 @@
#! /bin/sh
if false
then
For use with a dcoker registry
docker service create --name my_compile_farm \
--network my_docker_overlay_net \
--replicas 2\
localhost:5000/lazy_abyss /bootup_script
else
docker service create --name my_compile_farm \
--network my_docker_overlay_net \
--replicas 2\
local:lazy_abyss /bootup_script
fi

16
dont_use_here/bootup_script Executable file
View file

@ -0,0 +1,16 @@
#! /bin/sh
trap "exit" SIGINT SIGTERM
#No need to keep that file around.
rm init_custom_container
export DISTCC_CMDLIST="/distccd_cmd_list"
distccd --daemon --allow 10.0.1.0 --stats -j20 --job-lifetime 10000 -N 0
#distccd --daemon --allow 10.0.1.79 --stats -j20 --job-lifetime 10000 -N 0 --listen 10.0.1.80
#distccd --daemon --allow 10.0.1.56 --stats --listen 10.0.1.57 --no-fork --no-detach --log-stderr
# Your compiles may take at most a day...
# TODO: fix?
sleep 1d

36
dont_use_here/compile Executable file
View file

@ -0,0 +1,36 @@
#! /bin/sh
export PATH="/usr/lib/distcc/bin:$PATH"
export DISTCC_HOSTS="localhost 10.0.1.57/10"
export DISTCC_CMDLIST="/distccd_cmd_list"
git clone https://github.com/Ella-0/lazybox.git
#---
#cd /
#git clone https://github.com/landley/toybox.git
#Can't build toybox
#Need ln -r for llvm :(
wget http://landley.net/toybox/bin/toybox-x86_64
chmod +x ./toybox-x86_64
mv toybox-x86_64 /bin/toybox
rm /bin/ln
/bin/toybox ln -s /bin/toybox /bin/ln
#---
cd lazybox/pkgs/llvm
../../lazypkg.sh
cd ../musl
../../lazypkg.sh
cd ../mksh
../../lazypkg.sh

12
dont_use_here/distccd_cmd_list Normal file
View file

@ -0,0 +1,12 @@
/usr/bin/clang
/usr/bin/clang++
clang
clang++
/usr/bin/cc
/usr/bin/c89
/usr/bin/c99
/usr/bin/c++
cc
c89
c99
c++

9
dont_use_here/init_custom_container Executable file
View file

@ -0,0 +1,9 @@
#! /bin/sh
echo Intitiating container upgrade
apk update &&
apk upgrade &&
apk add distcc samurai cmake python3

16
dont_use_here/inittab Normal file
View file

@ -0,0 +1,16 @@
# Default runlevel. The runlevels used by RHS are:
# 0 - halt (Do NOT set initdefault to this)
# 1 - Single user mode
# 2 - Multiuser, without NFS (The same as 3, if you do not have networking)
# 3 - Full multiuser mode
# 4 - unused
# 5 - X11
# 6 - reboot (Do NOT set initdefault to this)
#
l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
#l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6

143
dont_use_here/inside_chroot Executable file
View file

@ -0,0 +1,143 @@
#!/bin/sh
export PATH=$PATH:/bin:/usr/bin:/usr/sbin
compile_packages (){
#for pkg in ${packages[@]};do
n_packages=$(echo $packages | tr ' ' '\n')
for pkg in $n_packages; do
echo "Going to build: $pkg"
cd /lazybox/pkgs/${pkg}/
../../lazypkg.sh
tar -xf out/${pkg}.*.tar.xz -C /
tar -xf out/${pkg}-dev.*.tar.xz -C / 2> /dev/null
tar -xf out/${pkg}-doc.*.tar.xz -C / 2> /dev/null
done
}
#need to replace ln from busybox by toybox
rm /bin/ln
/bin/toybox ln -s /bin/toybox /bin/ln
echo "replaced ln from busybox by toybox"
ln -s /usr/bin/ld.lld /usr/bin/ld
echo "Linked ld.lld (from llvm) to ld"
ln -s /bin/busybox /usr/bin/fdisk
echo "Added fdisk from busybox"
mkdir /etc
touch /etc/passwd
echo "root:x:0:0::/root:/bin/sh" > /etc/passwd
touch /etc/group
echo "root:x:0:root" > /etc/group
#----
# Building spooky
#----
#---------
# Stage 1
#---------
#Apply the only headers patch for linux, we can't build it yet...
patch /lazybox/pkgs/linux/build.sh /linux_header_only_patch
#Stub lex
printf "#!/bin/sh\necho lex go brr\n" > /usr/bin/lex
chmod +x /usr/bin/lex
#Yes om4 has to be build twice
packages="mksh bmake gmake byacc om4 om4"
compile_packages
#---------
# Stage 2
#---------
rm /usr/bin/lex
packages="flex zlib expat"
compile_packages
#Samurai
cd /lazybox/pkgs/samurai
cd src/samurai-*/
clang *.c -o samu
cp /lazybox/pkgs/samurai/out/samurai/bin/samu /usr/bin
cd /lazybox/pkgs/samurai
../../lazypkg.sh
rm /usr/bin/samu
tar -xf out/samurai.*.tar.xz -C /
packages="rsync linux"
compile_packages
#---------
# Stage 3
#---------
# Libre ssl, a slight problem
# lible ssl needs cmake, but cmake needs libre ssl. :(
cd /lazybox/pkgs/libressl
cd src/libressl-*/
./configure --prefix=/usr && make -j12 && make install
cd ../..
cd /lazybox/pkgs/cmake
cd src/cmake-*/
./bootstrap \
--prefix=/usr \
--mandir=/usr/share/man \
--datadir=/usr/share/$pkgname \
--docdir=/usr/share/doc/$pkgname \
--generator=Ninja \
--no-system-libs
cd ../..
#NOTE: we recompile mksh, because now we have curl and it can download the man page!
#We remove the linux headers only patch
patch -R /lazybox/pkgs/linux/build.sh /linux_header_only_patch
packages="cmake libressl busybox toybox libffi python ca-certificates curl zlib expat kati gettext-tiny git netbsd-curses kakoune mksh linux lazybox llvm musl tiny-linux-bootloader"
compile_packages
echo "finished"
exit
#--------------
#--------------
# GARBAGE
#--------------
#--------------
# om4 and flex have a circular dependency
# NOTE: THIS DOES NOT WORK
# The lex file in om4 is compiled on the host :(
# Solution: see prepare_chroot_lazy
# PLAN:
# build https://github.com/ibara/baseutils lex
# build om4
# b uild flex
# rm baseutils lex
cd /lazybox/tmp_pkgs/baseutils
make -C libopenbsd
make -C lex
cp lex /usr/bin
# Build m4
# bulid flex
#make sure to overwrite /usr/bin/lex

17
dont_use_here/linux_header_only_patch Normal file
View file

@ -0,0 +1,17 @@
--- build.sh
+++ build.sh
@@ -16,10 +16,10 @@
build() {
cd $pkgname-$pkgver
#gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ YACC=yacc LLVM_IAS=1 defconfig
- gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ LEX=lex YACC=yacc LLVM_IAS=1 oldconfig
- gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ LEX=lex YACC=yacc LLVM_IAS=1 menuconfig
- cp .config ../../x86_64.config.new
- gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ LEX=lex YACC=yacc LLVM_IAS=1
+ #gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ LEX=lex YACC=yacc LLVM_IAS=1 oldconfig
+ #gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ LEX=lex YACC=yacc LLVM_IAS=1 menuconfig
+ #cp .config ../../x86_64.config.new
+ #gmake CC=cc CXX=c++ HOSTCC=cc HOSTCXX=c++ LEX=lex YACC=yacc LLVM_IAS=1
}
package() {