diff options
Diffstat (limited to '080_blog/00015_Admin')
8 files changed, 1276 insertions, 0 deletions
diff --git a/080_blog/00015_Admin/00005_Miscellanous/index.md b/080_blog/00015_Admin/00005_Miscellanous/index.md new file mode 100644 index 0000000..d247e7e --- /dev/null +++ b/080_blog/00015_Admin/00005_Miscellanous/index.md @@ -0,0 +1,866 @@ +Miscellanous Notes +================== + +A collection of short notes on things I wish to remember. Many topics are heavily outdated. + +Backing up and Restoring Docker Volumes +--------------------------------------- + + July 3, 2017 + +One of my first questions, when starting my docker experience was +how to backup and restore docker volumes. + +### On the source host + +First of all identify your volumes: + + $ docker ps + $ docker inspect my_container_name | grep -C 1 -i Source + +Now you can backup the directory to a tar on your host machine: + + $ docker run –rm –volumes-from my_container_name -v $(pwd):/backup ubuntu tar cvf /backup/backup_db.tar /var/lib/mysql + +### On the target host + +In order to securely copy and restore the volume on the target host one can run: + + $ scp source_host:~/backup_db.tar . + $ docker run –rm –volumes-frommy_container_name -v $(pwd):/backup ubuntu bash -c “cd /var/lib/mysql && tar xvf /backup/backup_db.tar –strip 1″ + +A Growing Collection of Linux Command Line One-Liners +------------------------------------------------------ + +Please believe me... this collection was really supposed to grow over time... + +inside a direcotry show disk usage of all hidden files and directories and sort by size: + + $ du $(ls .* -d | tail -n +3) -hs |sort -h + +inside a direcotry show disk usage of all files and directories (also hidden) and sort by size. +Exclude ./DATA file. + + $ du . -a -d 1 -h –exclude=’./DATA’ | sort -h + +Tar all files in current directory, excluding ./DATA and ./.cache + + $ tar –exclude=’./.cache’ –exclude=’./DATA’ -cvf home_miguel_20180216.tar . + +Find files in ./ARCHIVE NOT belonging to a specific user: miguel + + $ find ARCHIVE/ \! -user miguel + +set folder/ permissions to Read/Browse only for owner recursively + + $ sudo chmod -R u=r,g=,o= folder/ + $ chmod -R u=rX,g=,o= folder/ + +find all mails from Boban when in the maildir full of mailboxes and print only short headers without bodies: + + $ grepmail -H -B -Y ‘(^TO:|^From:)’ Boban * + +not really a one-lier but will print 256 colors in a bash: + + for i in {0..255} ; do + printf "\x1b[48;5;%sm%3d\e[0m " "$i" "$i" + if (( i == 15 )) || (( i > 15 )) && (( (i-15) % 6 == 0 )); then + printf "\n"; + fi + done + +Some Tools of Choice +-------------------- + +### Systems + +* joomla +* redaxo +* typo3 +* wordpress +* mediawiki + +* oscommerce +* opencart + +* owncloud +* alfresco +* mantis + +* piwik / matomo +* loganalyzer +* goaccess +* nagios / icinga + +### Servers + +* apache +* nginx + +* mariadb / mysql +* postfix +* postgis + +* geoserver + +<!-- +## Tools / Servers + openlayers + git / gitweb / stagit / gitolite + xmpp + mumble + docker / registry + kvm + kubernetes + postfix/dovecot / rouncube/postfixadmin + symfony + bootstrap + webGL + websockets + ajax + node/angular + jmeter + ceph +--> + +Compilation Notes +----------------- + +### build your own webkit + + March 14, 2018 + +Let’s compile a release with debug info and install to /usr/local + +~~~~~~ {.bash} +wget https://webkitgtk.org/releases/webkitgtk-2.20.0.tar.xz +tar -xvf webkitgtk-2.20.0.tar.xz +cd webkitgtk-2.20.0 +# install all the libs that will be reported missing in the next step. +# I could not find the woff2 stuff in debian so skipped it... +cmake -DPORT=GTK -DCMAKE_BUILD_TYPE=RelWithDebInfo -DUSE_WOFF2=NO -GNinja +# this takes about 30minutes on my i7-4790K .. zzzzz..zzz +ninja +sudo ninja install +~~~~~~~~~~~ + + pkg-config + + pkg-config uses our new build now: + + [1] https://trac.webkit.org/wiki/BuildingGtk + [2] https://webkitgtk.org/ + +### Build and Install GCC + + April 27, 2015 + +#### Preparations +Read the prequisites at +[https://gcc.gnu.org/install/prerequisites.html](https://gcc.gnu.org/install/prerequisites.html) +and get (most recent versions at time of this writing) the following: + +* gcc (5.2.0) +* binutils (2.25.1) + +#### Configure and Build + +* unpack binutils-x.y.z +* create a new directory binutils-x.y.z-build and inside it run the following commands: + +~~~~~~ {.bash} + $ ../binutils-x.y.z/configure --disable-nls --with-sysroot --enable-targets=all + $ make -j4 + $ make install +~~~~~~~~~~~~ + +* unpack gcc-x.y.z and run the contrib/download_prerequisites script inside. +* create a new directory: gcc-x.y.z-build and inside it run: + +~~~~~~ {.bash} + $ ../gcc-x.y.z/configure --disable-nls --enable-languages=c,c++ --enable-threads + $ make -j4 + $ make install +~~~~~~~~~~~~ + +#### Reference + [1] binutils and gcc README files. + [2] https://gcc.gnu.org/install/ + [3] http://wiki.osdev.org/Building_GCC + [4] http://stackoverflow.com/questions/1726042/recipe-for-compiling-binutils-gcc-together + +monad transformers in action +---------------------------- + January 1,2018 + + *Main Control.Monad.Writer Control.Monad.State> runState (runWriterT (get >>= \a -> tell ["foo"] >> put (a*a) >> tell ["bar"] >> tell [show a])) 5 + +emscripten +---------- + October 1,2017 + + sdl2 port + https://github.com/juj/emsdk + android-ndk-r15c + + * accomplish build sys with: sdl2+input+audio+opengl+SDL_net+SDL_thread @ linux, win, osx, ios, android, win-phone, steam, rasp + +LUKS container over sshfs +------------------------- + July 5, 2017 + +* enable user\_allow\_other in /etc/fuse.conf +* sshfs -o allow\_root user@server:/BACKUPS/ ~/mnt/ +* dd if=/dev/urandom of=~/mnt/megaloman bs=1M count=512 +* sudo cryptsetup -y luksFormat ~/mnt/megaloman +* sudo cryptsetup luksOpen ~/mnt/megaloman vol1 +* sudo mkfs.ext4 /dev/mapper/vol1 +* sudo mount /dev/mapper/vol1 /mnt +* df -h | grep vol1 +* sudo umount /mnt +* sudo cryptsetup luksClose vol1 +* fusermount -u ~/mnt + +NOTE: backup your data example use : rsync -a –info=progress2 source dist + +some areas of interest +---------------------- + December 20, 2014 + +Some General Topics I am reasearching right now (or plan to do it). Or used to reasearch... + +* Neuronal Networks +* Deep Learning +* Random Forrests (and other techniques based on Decision Trees) +* Simmulated Annealing +* (Linear) Integer Programming + + +Inter Process Communication +--------------------------- + March 14, 2018 + +We can attach nicely to same memory segment from 2 different processes: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.c .numberLines} +// ipc via shared mem +// attach to shared memory; +key_t my_ftok = ftok("~/surf-webext-dom-shared-mem",'a'); + +int mem_seg=shmget(my_ftok,1024*1024,IPC_CREAT|0660); +if(mem_seg==-1) +{ +g_print("shmget failed: %s\n",strerror(errno)); +} + +shared_buf=shmat(mem_seg,NULL,0); +if(shared_buf==(void*)-1) +{ +g_print("shmat failed: %s\n",strerror(errno)); +} +g_print("attached to shared memory.\n"); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +SSL Certificate CSR +------------------- + + February 20, 2015 + + for apache ssl mod + + first check your old csr if you like: + $ openssl req -text -noout -verify -in CSR.csr + + 1. generate key + $ openssl genrsa -des3 -out www.yourdomain-example.com.key 2048 + + 2. create CSR + $ openssl req -new -key www.yourdomain-example.com.key -out www.yourdomain-example.com.csr + + refs: + [1] https://search.thawte.com/support/ssl-digital-certificates/index?page=content&id=AR1108&actp=LIST + +Competitive Coding +------------------ + March 17, 2015 + +A small selection of websites, which I visit from time to time, addressing competitive coding. + +* topcoder.com +* codeforces.com +* codility.com +* codersclan.ne +* projecteuler.net +* www.codingame.com (need to check this) +* www.codechef.com (need to check this) +* http://psyho.gg/overview-of-programming-contests/ (and read here) +* https://www.hackerrank.com/ +* http://code.google.com/codejam +* https://www.kaggle.com/ – predicitve modelling +* odesk.com / elance + +Comp Science Literature +----------------------- + March 17, 2015 +This is a list of some books, focusing on topics around computer science, which I recently read or am currently reading or want to read :P + +* Meyers, Effective C++ (Addison-Wesley) C++98 only? +* Meyers, More Effective C++ (Addison-Wesley) C++98 only? +* Meyers, Effective STL. C++98 only? +* Sutter, Exceptional C++ +* Sutter, More Exceptional C++ +* Karlson, Beyond the C++ Standard Library: An Introduction to Boost 1st Edition +* Maybe Something on Multiprocessing (?) +* Donald Knuth. The Art Of Computer Programming +* Algorithms 3rd Edition (by Cormen, Leiserson, Rivest, Stein) +* Jonathan Bartlett. Programming from the Ground Up +* C++ in a Nutshell (O’REILLEY, by Lischner) +* C++ Primer (5th Edition) by Lippman, Lajoie, Moo +* The C++ Programming Language 4th Edition +* Anthony Williams, C++ Concurrency in Action: Practical Multihreading +* Dive into Python (2 and 3) by Mark Pilgrim +* Learn You a Haskell for Great Good by Marian Lipovaca +* Real World Haskell +* Version Control with Git (O’REILLEY, by Loeliger & McCullough) (next: 9(10) ?) +* The Linux Command Line, by William Shotts (http://linuxcommand.org/tlcl.php) +* Debian, The Administrators Handbook (by Hertzog and Mas) +* Absolute FreeBSD by Michael W. Lucas +* Modern Operating Systems, Third Edition (by Andrew S. Tannebaum) +* Linux Device Drivers (O’REILLEY, by Corbert, Rubini, Kroah-Hartman) +* Linux Kernel in a Nutshell (O’REILLEY, Greg Kroah-Hartman) + +* Concrete Math (2nd) +* UPENN cis194 + +* STOC '83 Proceedings of the fifteenth annual ACM symposium on Theory of computing / Primitives for the manipulation of general subdivisions and the computation of Voronoi diagrams + +* http://haskellbook.com/ +* Wadler paper "Monads for Functional Programming" +* simon marlow - book (parallel) +* data 6 - course (Github) +* sedgewick & waynes algorithms +* books from appendix, from programming from ground up? + +VNC +--- + April 2, 2015 + +### Server: + $ apt-get install tightvncserver + $ xtightvncserver -geometry 1900×1100 + $ netstat -tap | grep vnc # remember port (e.g.5901) + +### Client: + $ ssh -L 5901:localhost:5901 you@remote # tunnel + $ xtightvncviewer localhost::5901 # in new terminal + +### Kill Server: + $ vncserver -kill :1 # (or :2 etc..) + +Owncloud Notes +-------------- + +### Mounting Ownclouds DavFS + + April 15, 2015 + + $ apt-get install davfs2 + + * since owncloud seems to have problems with locks, edit /etc/davfs2/davfs2.conf and set use_locks to 0 in order to allow creating files (as well as probably writing) + + $ mount -t davfs https://secure.sf.com/owncloud/remote.php/webdav /mnt/ + +### rescan/rebuild ownlocud files database + + June 22, 2016 + + inside ownlcoud directory run: + sudo -u www-owncloud php console.php files:scan –all + + +LXC +--- + April 28, 2015 + + maybe worth a try: + + apt-get install lxc; lxc create ctname -t download — -d debian -r jessie + -a amd64; lxc-start -d -n ctname, lxc-attach ctname + + similar tools: chroot/ debootsrap + +Add vim Lang +------------ + May 14, 2015 + + $ sudo apt-get install myspell-pl + $ cd /ush/share/hunspell + $ vim + :mkspell pl pl_PL + +What every programmer should know about +--------------------------------------- + November 7, 2015 + + Undefined behaviour: http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html + Memory: http://lwn.net/Articles/250967/ + Floating point arithmetic: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html + +IDE Comparison +-------------- + + December 7, 2015 + +While I love working with vim (ctags, taglist, etc..) and the command line, it might be worth to have a look at the following IDE’s for Java and/or C++: + +* IntelliJ IDEA +* Eclipse +* Eclipse CDT +* Codeblocks +* Netbeans +* Codelite +* KDevelop + +Setup Postfix +------------- + +We want to handle different domains on a single postfix server and have virtual users along real unix users. For the virtual users we use uid/gid 5000. + +Relevant config files are: + + /etc/postfix/master.cf + /etc/postfix/main.cf + /etc/postfix/smtpd_sender_login_maps (tells which emails belong to which login) + /etc/postfix/vmailbox (sepcifies our VIRTUAL mailboxes and the names of the spoolfiles) + /etc/postfix/virtual (aliases for our local unix users) + /etc/dovecot/users (specifies the logins, passwords, home directories,..) + /etc/dovecot/conf.d/10-mail (set inbox and mailbox dirs) + +Be careful about the permissions of the different folders, so uid/gid 5000, postfix and dovecot can access as relevant. + +A very nice tutorial on setting up postfixadmin can be found here: + + https://lelutin.ca/posts/installing_postfix_-_clamav_-_spamassassin_-_dovecot_-_postfixadmin_on_debian_squeeze/ + +Windows Specific Notes +---------------------- + +### disable hiberfile.sys + + September 7, 2017 + + run cmd.exe as administrator: + powercfg.exe -h off + +### GTA4 on NVIDIA-GTX 980 + + April 17, 2016 + + Add following flags : -nomemrestrict -norestriction + Otherwise video memory is not identified correctly. + +### Civilization 4 BTS Autosaves + + September 23, 2016 + + In the file: C:\Users\miguel\Documents\My Games\Beyond the Sword\CivilizationIV + + Adjust: + + ; The maximum number of autosaves kept in the directory before being deleted. + MaxAutoSaves = 100 + + ; Specify the number of turns between autoSaves. 0 means no autosave. + AutoSaveInterval = 1 + +Fool’s Wiki Knowledge +===================== + + February 24, 2016 + +Migrated from our former Mediawiki: + +Welcome to the Foolo-Pedia! Foolo-Pedia is a growing collection of interesting, important, ridiculous or useless information gathered and refined by the SoftwareFools team for all Friends of Fools. We decided to mainly keep this Wiki in english to be compatible with all our costumers and friends. Some destinct pages maybe in polish, german or klingon. + +Articles which might be of universal interest, divided by area and topic. + +## Minilinux + + software fools minimal linux + + 1. usb stick + 2. create one big partition (2gb) + 3. mkefs.ext4 /dev/sdx1 + 4. mount /dev/sdx1 /mnt + 5. compile kernel + 6. cp bzImage /mnt + 7. boot + + 8. + grub> search /bzImage + grub> linux /bzImage root=?? + + PROBLEMS WITH kernel config! + +## Kernel Building + + cd linux-source-[xxx] + make mrproper + cp someconfig .config -i + make oldconfig + make menuconfig + make localmodconfig + make localyesconfig + make + (OPT) make modules + su + make install + (OPT) make modules_install + (OPT) update-intiramfs -c -k [kernel-postfix] + + (REMOVE UNWANTED FILES FROM BOOT!) + update-grub + grub-install /dev/sda + reboot + +## GDB + + http://www.cprogramming.com/debuggers.html + + use gcc or clang with : -ggdb -O0 + + h - help + + r - run + c - continue + n - next + s - step + + b - break <line> + p - print <var> + + wa - watch <var> + + bt + frame <number> + list + info locals + + set <var> = <value> + set substitute-path FROM TO + + call <func> + +## clang + + http://llvm.org/releases/3.7.0/docs/CMake.html + http://clang.llvm.org/get_started.html + put MAKEFLAGS="-j8" or similar in front of your CMake invocations. + + extracted llvm to ~/temp/clang/llvm-3.7.0.src/ + extracted clang to ~/temp/clang/llvm-3.7.0.src/tools/clang/ + extracted compiler-rt to ~/temp/clang/llvm-3.7.0.src/projects/compiler-rt + extracted libcxx to ~/temp/clang/llvm-3.7.0.src/projects/libcxx + + $ cd ~/temp/clang/llvm-3.7.0-build + $ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/miguel/opt/llvm-3.7.0 ../llvm-3.7.0.src + $ cmake --build . + $ cmake --build . --target install + +## XMPP + +### Idea + +Today I figured out, once again, that there are many open-source and free (as in freedom) alternatives to all those chat clients and their proprietary protocols, which we dislike and want to avoid. Taking it seriously I decided to set up a XMPP Server on our Softwarefools Server (http://www.softwarefools.com) to finally enable secure and comfortable communication between me, myself and my fellow peers. + +### Starting Point + +* We have a Debian Virtual Private Server (VPS) Running wheezy (stable) +* I just found that tutorial: http://wiki.linuxwall.info/doku.php/en:ressources:articles:ejabberd + +### I. get package + + # apt-get install ejabberd + +### II. adapt config + +In /etc/ejabberd/ejabberd.cfg adapt the hosts line as that: (if you wonder this is Erlang) + + %% Hostname + {hosts, ["softwarefools.com"]}. + +### III. setup certificate + + # mkdir /etc/ejabberd/certs + +For now we skip the rest of that step and proceed with the self-signed certificate provided by Debian. (/etc/ejabberd/ejabberd.pem) + +### IV. restart server + + # /etc/init.d/ejabberd restart + +### V. create user + + # ejabberdctl register miguel softwarefools.com 'pass123' + # ejabberdctl register niki softwarefools.com 'pass321' + +### VI. open ports + + # iptables -I INPUT -p tcp --dport 5222 -j ACCEPT + # iptables -I INPUT -p tcp --dport 5269 -j ACCEPT + +TODO: in fact we need more here! + +### VII. connect with client + +* Get Pidgin http://www.pidgin.im/ +* Connect & Enjoy  + +### Further Work +* certificate +* dns configuration – should provide xmpp info (? + + +## Benchmark + +### Tools for Stress Testing your RIG + + May 12, 2017 + + Some nice tools to stress-test your computer: + + CPU: Prime95 + GPU: FurMark + RAM: MemTest86+ + HDD/SSD: S.M.A.R.T + + Ref: http://www.pcworld.com/article/2028882/keep-it-stable-stupid-how-to-stress-test-your-pc-hardware.html + +### Networking + + 1. Connected my [laptop] and [desktop] via 5meter cat5e cable -> 1000mbit full duplex link + 2. created ~1.5G ramdisks with tmpfs on each. + 3. created a 1 giga file with data from /dev/urandom on [desktop] + 4. copied file with scp from [desktop] to [laptop] + 5. copied file with scp from [laptop] to [desktop] + 6. repeated 4 & 5 10 times : average speed ~70+MB/s from desk & 80+MB/s to desk, link stays up all the time. ping around 0.2-0.3ms + 7. repeated 1-6 with my other 3meter ethernet cable. + 8. tried both direction simultanously which gave a total data throughput even over 100MB/s + +### Hard Disk + +To get some information about the disk run: + +Request identification info directly from the drive, which is displayed in a new expanded format with considerably more detail than with the older -i option. + + $ hdparm -I /dev/sda + +Timings + + $ hdparm -tT /dev/sda + +Clear Cache + + $ echo 3 > /proc/sys/vm/drop_caches + +I am using this block-size and count which results in ~2GB: + + $ dd [...] bs=1048576 count=2048 + +READ: + + $ dd if=test of=/dev/null + $ dd if=/dev/sdX of=/dev/null + +WRTIE: + + $ dd if=/dev/zero of=test + $ dd if=/dev/zero of=/dev/sdX + +smartmontools: + + seatage: + Raw_Read_Error_Rate + Seek_Error_Rate + + example: + % python + >>> 200009354607 & 0xFFFFFFFF + 2440858991 <---- total number + >>> (200009354607 & 0xFFFF00000000) >> 32 + 46 <--- number of errors + +## Fixing HDD after Free Falling + +So you also dropped your hard-disk and some of your secotrs got damaged, +probably by the head hitting the platter? The following short text outlines, +how I approached fixing my disk after it hit the floor. Of course I would +not rely on this disk for mission critical data anymore, but it +might serve as a scratch/temporarily/redundant space. + +Backup all your data in advance because the disk will almost certainly +require reformatting and repartitoning after follwing this procedure! + +Some Tools/Programms I used: + +* smartctl +* hdparm +* grep + +### smartctl + +Try from fastest to slowest to find faulty sector: replace LBAi\_START,LBA\_END with meaningful values. + + smartctl -t [long|conveyence|short|select,LBA_START-LBA_END] -C /dev/sda + +The following did not work for me. Resulted in ‘interrupted by host’ for some reason. +Probably the disks goes to sleep or something. Issueing some read operations every minute might fix this(?) + + -C to run test in foreground mode! + +### hdparm + +You can bi-search via –read-sector if complete ranges are affected (As in my case) and then fix them in one step: + + for i in {36312851..36312886}; do hdparm --write-sector $i /dev/sda; done + +### Example + +Exemplary session fixing one sector: + +**WARNING: this will destroy your data !** + + 1) # smartctl -a /dev/sda | egrep 'Pend|Real|Offline_Unc' + + 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 4 + 197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 47 + 198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 47 + + 2) # smartctl -t short /dev/sda + + 3) # smartctl -l selftest /dev/sda + + [...] 1 Short offline Completed: read failure 90% 10632 152076520 + + 4) # hdparm --read-sector 152076520 /dev/sda # verification + + [...] reading sector 152076520: FAILED: Input/output error + + 5) # hdparm --write-sector 152076520 /dev/sda + + 6) # smartctl -a /dev/sda | egrep 'Pend|Real|Offline_Unc' + + 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 4 + 197 Current_Pending_Sector 0x0012 098 098 000 Old_age Always - 46 + 198 Offline_Uncorrectable 0x0010 098 098 000 Old_age Offline - 46 + +## Raid and LVM + + + Also refer to: http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/foreward.html + + Here I will write down some of my considarations about setting up a RAID + and LVM. + + We want to get a performance boost and minimize the ssd wear-out. + Also we will not put there any mission-critical files, because the content + of the ssd will not be mirrored. beside regular backups of course. + We also need to take the limited diskspace into account. + while our hdd’s are 2*2TB we only have a single 250MB sdd. + + So we will be looking at files that are: + + * not updated very often + * read often + * not critical if they get lost. + * not too huge + + So I consider for my usecase + * /bin + * /sbin + * /boot + * /lib{,32,64} + + * /usr (without /usr/local) + + That directories will reside on our raid1 since they contain important files: + * /home/miguel + * /root + * /opt + * /usr/local + + Finally we will mount the following things in ram for superfast and temporary space: + + * /tmp + * /var/log (unless you care about logs) + + What about that directories? + consider speed vs. security vs. drive wear-out + + * /etc + * /var + + Never mount swap on your ssd! + + == extending logical volume and fs == + * https://wiki.archlinux.org/index.php/LVM + + # lvextend -L +20G VolGroup00/lvolhome + + resize fs (ext2, ext3 or ext4) + for ext4 this can be done even without unmounting the device + + # resize2fs /dev// + +## Some RAID Problems on my old Desktop + + https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray + https://plone.lucidsolutions.co.nz/linux/io/ssd-on-nvidia-sata-port-generates-error-eh-in-swncq-mode-and-failed-command-read-fpdma-queued/view + https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array + + Kernel options: sata_nv.swncq=0 + + [ 8625.896029] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen + [ 8625.896035] ata5.00: failed command: WRITE DMA EXT + [ 8625.896040] ata5.00: cmd 35/00:08:50:06:b0/00:00:13:00:00/e0 tag 0 dma 4096 out + res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) + [ 8625.896043] ata5.00: status: { DRDY } + [ 8625.896048] ata5: hard resetting link + [ 8625.896050] ata5: nv: skipping hardreset on occupied port + [ 8626.364038] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) + [ 8631.372026] ata5.00: qc timeout (cmd 0x27) + [ 8631.372031] ata5.00: failed to read native max address (err_mask=0x4) + [ 8631.372033] ata5.00: HPA support seems broken, skipping HPA handling + [ 8631.372035] ata5.00: revalidation failed (errno=-5) + [ 8631.372042] ata5: hard resetting link + [ 8631.372044] ata5: nv: skipping hardreset on occupied port + [ 8631.840045] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) + [ 8631.864140] ata5.00: configured for UDMA/133 + [ 8631.864146] ata5.00: device reported invalid CHS sector 0 + [ 8631.864154] ata5: EH complete + [ 8662.888029] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen + [ 8662.888036] ata5.00: failed command: WRITE DMA EXT + [ 8662.888041] ata5.00: cmd 35/00:08:50:06:b0/00:00:13:00:00/e0 tag 0 dma 4096 out + res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) + [ 8662.888043] ata5.00: status: { DRDY } + [ 8662.888049] ata5: hard resetting link + [ 8662.888051] ata5: nv: skipping hardreset on occupied port + [ 8663.356038] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) + [ 8663.396135] ata5.00: configured for UDMA/133 + [ 8663.396141] ata5.00: device reported invalid CHS sector 0 + [ 8663.396151] ata5: EH complete + [ 8693.864031] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen + [ 8693.864038] ata5.00: failed command: WRITE DMA EXT + [ 8693.864043] ata5.00: cmd 35/00:08:50:06:b0/00:00:13:00:00/e0 tag 0 dma 4096 out + res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) + [ 8693.864045] ata5.00: status: { DRDY } + [ 8693.864051] ata5: hard resetting link + [ 8693.864052] ata5: nv: skipping hardreset on occupied port + [ 8694.332036] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) + [ 8694.356134] ata5.00: configured for UDMA/133 + [ 8694.356141] ata5.00: device reported invalid CHS sector 0 + [ 8694.356150] ata5: EH complete + [ 8724.840030] ata5: limiting SATA link speed to 1.5 Gbps + + + diff --git a/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00105_GNU-Screen-Multiplexer-Keyboard-Shortcuts/index.md b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00105_GNU-Screen-Multiplexer-Keyboard-Shortcuts/index.md new file mode 100644 index 0000000..80b24b1 --- /dev/null +++ b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00105_GNU-Screen-Multiplexer-Keyboard-Shortcuts/index.md @@ -0,0 +1,43 @@ +GNU Screen - Keyboard Shortcuts +============================================= + +Invocation +---------- + $ screen [-S session_name] # simply start new session + $ screen -ls # lists sessions + $ screen -r session_name # resume detached session + $ screen -d -R session_name # resume session (also detach or create) + $ screen -d -RR #resume first session + +Shortcuts +--------- + ctrl-a c NEW WINDOW + ctrl-a n / ctrl-a p NEXT/PREVIOUS + ctrl-a d DETACH + ctrl-a D D DETACH and LOGOUT + + ctrl-a S SPLIT HORIZ. + ctrl-a | SPLIT VERTICAL + ctrl-a tab NEXT REGION + ctrl-x REMOVE REGION + ctrl-a q REMOVE ALL OTHER REGIONS + ctrl-a F FIT WINDOW + + ctrl-a c CLEAR SCREEN + ctrl-a k KILL WINDOW + ctrl-a \ QUIT + + ctrl-a esc COPY/SCROLLBACK MODE + +Layout Commands +--------------- + ctrl-a : #enter command mode + :layout new [title] + :layout remove [n|title] + :layout next + :layout prev + :layout show #list layouts + +Reference +--------- + [1] $ man screen diff --git a/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00110_Midnight-Commander-Cheatsheet/index.md b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00110_Midnight-Commander-Cheatsheet/index.md new file mode 100644 index 0000000..eedc66a --- /dev/null +++ b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/00110_Midnight-Commander-Cheatsheet/index.md @@ -0,0 +1,61 @@ +###>>>KWD midnight commander cheatsheet, mc cheatsheet, mc shortcuts, midnight commander shortcuts +###>>>DSC Cheatsheet for Midnight Commander summarizing the most useful shortcuts. +Midnight Commander Cheatsheet +============================= + +A short cheat-sheet summarizing midnight commander's most useful shortcut keys. + +Basic Navigation +---------------- + tab switch active panel + up/down select previous/next rown + return open selected directory + alt-o open selected directory on other panel + +View +---- + alt-. toggle hidden files + ctrl-o toggle console + alt-t toggle listing mode + alt-i sync with other panel + ctrl-u swap panels + +Miscellaneous +------------- + shift-f6 rename, filling in the current filename + insert/ctrl-t/+/\/* selections + esc-tab auto complete + ctrl-enter / alt-enter copy currently selected filename to console + Ctrl + Shift + Enter copy full path + + alt-shift-h show dir history + alt-y/alt-u navigate history + + alt+?/alt-s/ctrl-s searches + + ctrl-space calculte size + ctrl - x c chmod + ctrl - x o chown + +Troubleshooting / Extras +------------------------ +Adding this to your _.bashrc_ will let _mc_ stay in the current directory after exiting: + + alias mc='. /usr/libexec/mc/mc-wrapper.sh' + +I run _midgnight commander_ inside _tmux_ and the shifted function keys did not work as +expected. Use midnight commanders _Learn Keys_ dialog to fix this. + +Set default viewer for pdf: + + xdg-mime default evince.desktop application/pdf + +Set default editor: + + alias mc='EDITOR=vim' + +Reference +--------- + +* http://klimer.eu/2015/05/01/use-midnight-commander-like-a-pro/ +* man mc diff --git a/080_blog/00015_Admin/00010_Various-Cheat-Sheets/index.md b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/index.md new file mode 100644 index 0000000..ee779fa --- /dev/null +++ b/080_blog/00015_Admin/00010_Various-Cheat-Sheets/index.md @@ -0,0 +1,3 @@ +Cheat Sheets +============ +A growing collection of some of my cheat sheets. diff --git a/080_blog/00015_Admin/00070_Password-Management-on-the-Command-Line/index.md b/080_blog/00015_Admin/00070_Password-Management-on-the-Command-Line/index.md new file mode 100644 index 0000000..1b45b02 --- /dev/null +++ b/080_blog/00015_Admin/00070_Password-Management-on-the-Command-Line/index.md @@ -0,0 +1,89 @@ +# Passoword Managemet from the Command Line + + February 8, 2018 + +Today we will look at some simple, portable yet effective ways to manage +your passwords from the command line. You will need nothing more than +**apg** and one of **gnugpg** or **openssl** along with your favorite +text editor. + +~~~~~~~~~~ {.bash} +apt install apg gnupg openssl +~~~~~~~~~~ + +## Generate your Password + +Before you can manage your passwords, you will obviously first have to +generate them. Since your brain might be a very poor random number +generator you can use `/dev/random` here. + +A few examples for generating random passwords with **apg** follow: + +~~~~~~~~~~ {.bash} +# generate a few random passwords with default settings using /dev/random +apg -c /dev/random + +# set password length to 20-30 characters and generate 10 passwords +apg -m20 -x30 -n10 -c /dev/random + +Example output: + gootCoHuecJarItOojBouFrag + OignisholWulfisOdPearshed + fekfedsornUgbacyoimyab + ... + +# Other useful flags: +#- a0 pronouncable +# -a1 random + +# If you use -a1 you can specify the symbolset with -M +# You can combine multiple -M options as in: -MCnS + +# -MC / -Mc must/can use small leters set +# -MC / -Mc must/can use capital symbol set +# -MN / -Mn must/can use numeral symbol set +# -MS / -Ms must/can use special symbol set + + +# Finally we can exclude specific characters from the symbol set with -E +apg -a1 -m10 -MN -E 02345678 -c /dev/random + +Example output: + 9119191199 + 9919119919 + 1199999911 + ... +~~~~~~~~~~ + +## Managing your Password Safe + +Just put the passwords in a plaintext file (named mypasswords in the examples below), +along with related data and encrypt them symmetrically via **gnupg**. +Decrypt them as needed. You will be prompted for a passphrase in each case. + +Note that **gnupg** might cache your password for a few minutes, +so don’t worry if you can decrypt them without beeing prompted. + +~~~~~~~~~~ {.bash} +#encrypyt. Don't forget to delete the source file +gpg -c mypasswords + +#decrypt and write to STDOUT +gpg -d mypasswords.gpg +~~~~~~~~~~ + +You might prefer openssl, which some claim to be even more portable + +~~~~~~~~~~ {.bash} +#encrypyt. Don't forget to delete the source file +openssl aes-256-cbc -salt -in mypasswords > mypasswords.aes + +#decrypt and write to STDOUT +openssl aes-256-cbc -d -in mypasswords.aes +~~~~~~~~~~ + +Adding, Editing or Deleting a password constitutes simply of the three steps: + +* decrypt your password file and save it in a safe place +* edit the passwordfile as needed with your favorite text editor +* encrypt the password flie back again diff --git a/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md b/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md new file mode 100644 index 0000000..f69989b --- /dev/null +++ b/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/index.md @@ -0,0 +1,212 @@ +Miguel’s KVM Adventures +======================= + +February 7, 2018 + +Some notes about my explorations of the World of KVM virtualization. + +## Abstract +In my never ending pursuit of abstraction and encapsulation I recently started integrating all of my services into docker containers and deploying them inside virtual kvm guests. This article presents my continues effort to summarize the findings of this ongoing journey, which implies that the following material might be subject to change anytime without notice. This collection of notes and sentimental thoughts comes without any warranty or implication of fitness for any purpose. You have been warned! Now feel free to make use of it.  + +## Libvirt +This are the most common virsh commands I use to manage the kvm guests, where _domain_ is simply the name of the targeted guest and _FILE_ the name of a XML file. Remember that libvirt supports other virtualization infrastructure as well (Xen, VMware, QEMU). Most of the options are self-explanatory. With ‘virsh create’ starting a transient domain, that will disappear after shutdown, and the define/start combo resulting in a persistent domain that will even survive host restarts + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.bash} + virsh create _FILE_ # create domain from xml file + virsh destroy _domain_ # forcefully remove domain + + virsh define _FILE_ # define domain from xml file + virsh undefine _domain_ # undefine domain + + virsh suspend _domain_ # stop all scheduling + virsh resume _domain_ # start scheduling + + virsh start _domain_ # power on domain + virsh shutdown _domain_ # send corresponding ACPI signal to guest + + virsh edit _domain_ # edit xml config in place + + virsh autostart _domain_ # set autostart flag + virsh autostart _domain_ --disable # unset autostart flag + virsh list [--all] [--autostart] # list defined/active/autostart domains +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +## Disk Image +Most of the guest systems will require some sort of storage. Creating a fresh qcow2 image, to back our virtual disk, is as simple as running: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.bash} + qemu-img create -f qcow2 milky.img 200G +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Do not worry, the image will only take a fraction of the declared space, and will not grow larger than necessary, due to trimming, which will be explained later. + + +## Domain Definition +Domains, this is how libvirt calls our ‘guests’, can be defined in XML formatted files. This is my minimalistic defintion of the domain ‘milkman’ carrying 8GB RAM and 4 CPUs: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.xml .numberLines} + <domain type='kvm'> + + <name>milkman</name> + <uuid>504d80ee-1427-11e8-9861-0708f4830f96</uuid> + + <memory unit='KiB'>8388608</memory> + <currentMemory unit='KiB'>8388608</currentMemory> + <vcpu>4</vcpu> + + <os> + <type>hvm</type> + <boot dev='hd'/> + </os> + + <features> + <acpi/> + </features> + + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + + <devices> + + <emulator>/usr/bin/kvm</emulator> + + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2' discard='unmap' /> + <source file='/home/miguel/KVM/images/milky.img'/> + <target dev='sda' bus='scsi'/> + </disk> + + <interface type='bridge'> + <source bridge='virbr1'/> + <model type='virtio'/> + </interface> + + <controller type='scsi' index='0' model='virtio-scsi' /> + + <graphics type='vnc' port='55555' autoport='no' listen='::1' /> + + </devices> + + </domain> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Beside the obvious RAM size and CPU count w specify the underlying qcow2 image, to be used for our emulated hard disk. We also want to specify discard=’unmap’ and make use of a virtio-scsi controller, both to allow trimming. Trimming will be covered in more detail later. + +Our virtual machine relies on a virtual bridge virbr1. It is very important to use type=’virtio’ here. The defaults resulted in extremely poor network performance, at least in some of my particular use cases. The setup of the bridge with accompanying parameters is described in the next section about networking. + +At the very last we tell the vnc-server to listen on ::1 at port 5555. This values can be also adjusted during run-time as explained later on. + +In order to install an operating system we can add a virtual cd-rom along with an iso-image by augmenting the devices section in our XML defintion with the following lines: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.xml .numberLines} + <disk type='file' device='cdrom'> + <driver name='qemu' type='raw'/> + <source file='/home/miguel/KVM/isos/debian-9.3.0-amd64-netinst.iso'/> + <target dev='hdc' bus='ide'/> + <readonly/> + </disk> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Make sure to adapt the boot order in the os section by adding an appropriate line, so you end up with this: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.xml .numberLines} + <os> + <type>hvm</type> + <boot dev='cdrom'/> + <boot dev='hd'/> + </os> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +## Networking +Since my primary interface to the virtual machines is SSH, reliable network connectivity is one of the primary foci. IPv4 addresses became scarse so we will not waste any for the host systems virbr1 or eth0. The following diagram illustrates my IPv4 setup of a simple arp proxy utilizing ipv4 forwarding. The guests use their public ipv4 addreses and the ips of the hosts gateway. + +{.img-fluid} + +There is no need to save address space in case of IPv6 since we have a complete /64 IPv6 subnet at our disposal. While only a few guests are accessible by their IPv4 public addresses directly, we have virtually an infinite number of IPv6 addresses. Sidenote: One single /64 IPv6 subnet consists of 2^64 different addresses, which is over four billion times more than there are IPv4 addresses in the whole world! I use just the lower /65 half of our /64 subnet for the guests while the IPv6 address of the hosts NIC lies in the upper half. + +My IPv6 setup in /etc/network/interface goes along this lines: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.numberLines} + #/etc/network/interfaces + + iface eth0 inet6 static + address2a01:6a8:122:5622:8000::88/128 + gateway fe80::1 + + iface virbr1 inet6 static + pre-up brctl addbr virbr1 + address 2a01:6a8:122:5622::3/65 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All we need to do is activate IPv6 forwarding on the host to let our guests communicate with the world outside. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.bash} + sysctl -w net.ipv6.conf.all.forwarding=1 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +And this is how the IPv6 config of a particular guest looks like: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.numberLines} + iface ens3 inet6 static + address 2a01:6a8:122:5622::13/65 + gateway 2a01:6a8:122:5622::3 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Things could be improved further by running a DHCP server, like dnsmasq, to assign the guest addresses, but for now I want to keep it simple. + +## VNC + +While ssh is perfectly sufficient for most of the time, you sometimes might need to have a look at the frame-buffer console. You can start/stop listening on a specific port or interface with: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.bash} + sudo virsh qemu-monitor-command <guest_name> --hmp change vnc <listen_ip>:<port> + sudo virsh qemu-monitor-command <guest_name> --hmp change vnc none +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Interestingly the port is offset by 5900 meaning that e.g. :87 will let the vnc-server listen on port 5987! +Check it with netstat -tulpn to be sure. + +## Backup Running KVM + +One of the beautiful things about using virtual machines is the level of control we have over them. We can for instance backup our running machines with almost no downtime using the following approach: + +dump config to xml file +save kvm state (RAM etc.) and stop the guest. +create an overlay on the underlying qcow2 disk image. +restore the kvm on the overlay. +backup the original disk image. +commit deltas from overlay to the image. +switch to the image with merged changes and delete deltas. + +A downtime will be experienced only between the save and restore steps, while the most time consuming part of the process, backing up the disk, can be delayed. The XML, RAM state and HDD snapshot contain all the data required to re-spawn an identical consistent copy of our virtual machine, as at the time of the backup. NOTE: the clock might cause problems if not adjusted, if some applications rely on it. Ntp can take care of that. A fast and dirty implementation of this technique, for my particular setup, can be found on our gitweb [2]. A more complete but complex solution is Daniel Berteaud’s perl script [3], which I frankly did _not_ test myself. + +## Docker Containers +A common use case is to run docker inside the virtual guests, which makes it an integral part of my ‘KVM Adventures’. I prefer to remap docker’s root user to a non-privileged user of my host, as well as utilize syslog instead of the default json-file driver. This is reflected by the following config: + +/etc/docker/daemon.json: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.json .numberLines} + { + "userns-remap": "miguel", + "log-driver": "syslog" + } +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Optionally you can tell rsyslog to log deamon.* entries into a separate file and adjust logrotation as outlined here [5]. + +## Miscellaneous + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.bash} + virt-host-validate # validate host virtualization setup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +## References + + [1] https://libvirt.org/formatdomain.html + [2] https://gitweb.softwarefools.com/?p=miguel/kvm_tools.git + [3] http://repo.firewall-services.com/misc/virt/virt-backup.pl + [4] https://www.linux-kvm.org/page/Tuning_KVM + [5] https://www.wolfe.id.au/2015/05/03/syslog-logging-driver-for-docker/ + diff --git a/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/ipv4.png b/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/ipv4.png Binary files differnew file mode 100644 index 0000000..3dac4ed --- /dev/null +++ b/080_blog/00015_Admin/00090_Miguels-KVM-Adventures/ipv4.png diff --git a/080_blog/00015_Admin/index.md b/080_blog/00015_Admin/index.md new file mode 100644 index 0000000..fe5bd15 --- /dev/null +++ b/080_blog/00015_Admin/index.md @@ -0,0 +1,2 @@ +Admin Stuff +=========== |
